Geoevent Services not producing any in/out results

3862
6
Jump to solution
03-16-2017 08:43 AM
ThomasFurcron
Occasional Contributor

I have a fresh install of Arcgis Server 10.4.1 and geoevent extension on a new windows server 2012 R2 machine.

I have set up my input and output and services, when I send test data based on this link

https://community.esri.com/community/gis/enterprise-gis/geoevent/blog/2015/12/19/how-to-test-geoeven...    Morakot Pilouk

I am getting the inputs but nothing coming out in the geoevent services in/out counts.....

I have tried running what RJ Sunderman suggested here Diagnostic utility for checking the status of the RabbitMQ platform service but I am getting errors there as well.  

Running diagnostics ...

Exception in thread "main" com.esri.arcgis.discovery.json.JSONException: JSONObject["token"] not found.
  at com.esri.arcgis.discovery.json.JSONObject.get(JSONObject.java:372)
  at com.esri.arcgis.discovery.json.JSONObject.getString(JSONObject.java:499)
  at com.esri.arcgis.diagnostics.RabbitMQDiag.runDiagnostics(RabbitMQDiag.java:150)
  at com.esri.arcgis.diagnostics.RabbitMQDiag.main(RabbitMQDiag.java:461)


Geoevent Log

Logger:

net.jodah.lyra.internal.ConnectionHandler

Message:

Failed to create connection cxn-42 java.net.ConnectException: Connection refused: connect
         at java.net.DualStackPlainSocketImpl.connect0(Native Method)[:1.8.0_73]
         at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:79)[:1.8.0_73]
         at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)[:1.8.0_73]
         at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)[:1.8.0_73]
         at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)[:1.8.0_73]
         at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)[:1.8.0_73]
         at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)[:1.8.0_73]
         at java.net.Socket.connect(Socket.java:589)[:1.8.0_73]
         at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)[:1.8.0_73]
         at com.rabbitmq.client.impl.FrameHandlerFactory.create(FrameHandlerFactory.java:32)
             [107:com.esri.ges.framework.messaging.jms-messaging:10.4.1]
         at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:588)
             [107:com.esri.ges.framework.messaging.jms-messaging:10.4.1]
         at net.jodah.lyra.internal.ConnectionHandler$3.call(ConnectionHandler.java:233)
             [107:com.esri.ges.framework.messaging.jms-messaging:10.4.1]
         at net.jodah.lyra.internal.ConnectionHandler$3.call(ConnectionHandler.java:227)
             [107:com.esri.ges.framework.messaging.jms-messaging:10.4.1]
         at net.jodah.lyra.internal.RetryableResource.callWithRetries(RetryableResource.java:50)
             [107:com.esri.ges.framework.messaging.jms-messaging:10.4.1]
         at net.jodah.lyra.internal.ConnectionHandler.createConnection(ConnectionHandler.java:227)
             [107:com.esri.ges.framework.messaging.jms-messaging:10.4.1]
         at net.jodah.lyra.internal.ConnectionHandler.createConnection(ConnectionHandler.java:115)
             [107:com.esri.ges.framework.messaging.jms-messaging:10.4.1]
         at net.jodah.lyra.Connections.create(Connections.java:68)
             [107:com.esri.ges.framework.messaging.jms-messaging:10.4.1]
         at com.esri.ges.messaging.jms.amqp.AmqpMessagingFactory$ConnectionEstablisher.run(AmqpMessagingFactory.java:873)
             [107:com.esri.ges.framework.messaging.jms-messaging:10.4.1]
         at java.lang.Thread.run(Thread.java:745)[:1.8.0_73]

Any thoughts or help would be greatly appreciated!!!

Thanks, 

Tom 

0 Kudos
1 Solution

Accepted Solutions
ThomasFurcron
Occasional Contributor

I went back and forth with ESRI support on this for the last 2 weeks.  What has finally worked with get us working was a re-install of ArcGIS Server and Geoevent.  

I believe the issue was related to the anti-virus software being on and scanning while the ArcGIS Server software was installed.  We first tried removing the anti-virus software to stop any auto-scans, but that did not do the trick.  So with the anti-virus software removed, I reinstalled ArcGIS server and GeoEvent and everything is working fine.  We have also reinstalled the anti-virus software, but have set it to only do manual scans, and have not had any issues since.  

I hope this helps others having similar issues out there.  

Tom 

View solution in original post

6 Replies
RJSunderman
Esri Regular Contributor

Tom -

I am seeing the same com.esri.arcgis.discovery.json.JSONException thrown by the Check RabbitMQ diagnostic utility. I added a comment to the thread you referenced (Re: Diagnostic utility for checking the status of the RabbitMQ platform service ).  I have to assume that the utility is not going to work with releases beyond 10.3.1 for which it was originally developed.

Here are a some trouble shooting steps you might try with your 10.4.1 deployment. I would encourage you to submit this to Esri Technical Support as an incident and ask them to contribute to this thread with steps taken if you are able to work with them to resolve the issue.

  • Stop the ArcGIS Server windows service
    • At 10.4.1 a service dependency should also stop the GeoEvent Extension windows service.
      If not, please select and explicitly stop the GeoEvent windows service.
  • Check the list of running processes on your Windows platform. The following processes, launched
    as part of the rabbitmq/erlang framework, part of the ArcGIS Server product's framework (...\ArcGIS\Server\framework\runtime) should no longer be running; they should have stopped when
    you stopped the ArcGIS Server windows service:
    • Process Name: epmd.exe
      ExecutablePath: …\rabbitmq\erlang\ERTS-5~1.4\bin\epmd.exe
    • Process Name: erl.exe
      ExecutablePath: …\rabbitmq\erlang\bin\erl.exe
    • Process Name: win32sysinfo.exe
      ExecutablePath: …\rabbitmq\erlang\lib\os_mon-2.2.14\priv\bin\win32sysinfo.exe
    • Process Name: inet_gethost.exe
      ExecutablePath: …\rabbitmq\erlang\erts-5.10.4\bin\inet_gethost.exe
  • With GeoEvent stopped, delete its runtime files from its product directory folder:
    (...\ArcGIS\Server\GeoEvent\data)
  • Start the ArcGIS Server windows service
    • At this point you can check to verify that you can launch the ArcGIS Server manager web app
      to check the health of your Server site.
    • You can also follow the steps from my 22-May-2015 comment to access the undocumented
      administrative endpoints for the RabbitMQ platform service. Administrative endpoints for the
      platform services were exposed through the ArcGIS Server Administrator Directory as part of
      the 10.5 product release.
  • Start the GeoEvent windows service and check the logs for a message with text similar to:
    com.esri.ges.framework.messaging.jms-messaging
    GeoEvent successfully connected to ArcGIS Server Platform Service RabbitMQ's message broker
  • Test to see if you can receive event records on an inbound connector and if the 'In' / 'Out' event counts for a GeoEvent Service incorporating that inbound connector also increment as event data is ingested.

Hope this information helps -

RJ

TL2
by
Occasional Contributor III

RJ, do we delete everything within the (...\ArcGIS\Server\GeoEvent\data) folder?  

I am experiencing this in both our HA and non-HA environments which run completely independent of each other.

0 Kudos
ThomasFurcron
Occasional Contributor

I went back and forth with ESRI support on this for the last 2 weeks.  What has finally worked with get us working was a re-install of ArcGIS Server and Geoevent.  

I believe the issue was related to the anti-virus software being on and scanning while the ArcGIS Server software was installed.  We first tried removing the anti-virus software to stop any auto-scans, but that did not do the trick.  So with the anti-virus software removed, I reinstalled ArcGIS server and GeoEvent and everything is working fine.  We have also reinstalled the anti-virus software, but have set it to only do manual scans, and have not had any issues since.  

I hope this helps others having similar issues out there.  

Tom 

ThomasFurcron
Occasional Contributor

Also received this from ESRI Support on the issue.  

http://support.esri.com/technical-article/000012517
Also can whitelist the Arcgisserver folder.  Than IT can scan the machine with those exceptions.

0 Kudos
NathanKemphues
New Contributor III

We also experienced this behavior after installing a patch for 10.3.1 on a machine that was running McAfee Ant-Virus. To resolve it, in addition to re-installing 10.3.1 without the patch and without McAfee running, we also had to delete the GeoEvent configuration, which in 10.3.1 resides at C:\ProgramData\Esri\GeoEvent.

0 Kudos
ChadFoley
New Contributor

I was getting the same error and fix it by going to https://servername:6443/arcgis/admin/system/platformservices and selecting Message Bus, then clicking 'Stop' - it auto-restarted itself and data started pouring into my GeoEvent Services. RabbitMQ seemed to be stuck in 'stopping' mode for some reason. This discussion helped me troubleshoot though we are on 10.5.1 - https://community.esri.com/message/521869?commentID=521869#comment  

0 Kudos