AnsweredAssumed Answered

[SOLVED] Error 8273: geoprocessing services crashing on SOM start and not restarting

Question asked by csparpaglione on Jan 18, 2013
Latest reply on Sep 14, 2015 by nzmalc
Hi,

I've been running ArcGIS Server 10.1 with SP1 on a Windows Server 2012 64-bit machine for months without any problems in publishing map services and geoprocessing services. I have an ArcMap 32-bit installation on the same machine of the server, which I use to view maps and publish services on the server.

Yesterday, this crazy situation happened to me: I tried to publish a new map service through ArcMap, but it was stuck and said "Not responding" for minutes, so I killed it. Tried again and again: no fortune. Then I examined the server log file, which in my machine is at the path "C:\arcgisserver\logs\<machine_name>\server" and I found lines like the following ones:

< ... type="SEVERE" code="8273" source="Server" process="3192" thread="1" methodName="" ... user="" elapsed="">Services containing process crashed. Utilities/PrintingTools.GPServer instance has crashed.</Msg>  < ... type="SEVERE" code="8273" source="Server" process="3192" thread="1" methodName="" ... user="" elapsed="">Services containing process crashed. System/PublishingTools.GPServer instance has crashed.</Msg>



The log file contained one crash-report line for each of the geoprocessing services running on the server.

I therefore tried to investigate: I opened the ArcGIS Manager web application and tried to restart the PublishingTools service (which comes bundled with the server): it took a lot of time, then the service  correctly stopped but in the end this message popped up:

System.PublishingTools.GPServer: java.rmi.UnmarshalException: Error unmarshaling return header; nested exception is: java.net.SocketException: Connection reset


I tried to restart all of the other geoprocessing (either System's and my custom ones), and for each of them I got similar messages printed on screen.
I then tried to restart all of the other services (map services, geometry services) and everything went fine!
So it seems to be the just an error on the geoprocessing services.

I further investigated by setting the server log's verbosity to DEBUG (that is: print everything!) and this is the full stack-trace of the exception:

<Msg time="..." type="SEVERE" code="8273" source="Server" process="4664" thread="1" methodName="" machine="..." user="" elapsed="">Services containing process crashed. System/PublishingTools.GPServer instance has crashed.</Msg> <Msg time="..." type="DEBUG" code="9999" source="Server" process="4664" thread="1" methodName="" machine="..." user="" elapsed="">java.lang.Exception: System/PublishingTools.GPServer instance has crashed.     at com.esri.arcgis.discovery.ejb.util.EJBBase.initService(EJBBase.java:363)     at com.esri.arcgis.discovery.ejb.util.EJBBase.init(EJBBase.java:218)     at com.esri.arcgis.discovery.ejb.impl.GPServerSyncBean.init(GPServerSyncBean.java:47)     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)     at java.lang.reflect.Method.invoke(Method.java:597)     at org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:162)     at org.apache.openejb.core.interceptor.ReflectionInvocationContext$LifecycleInvocation.invoke(ReflectionInvocationContext.java:194)     at org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:144)     at org.apache.openejb.monitoring.StatsInterceptor.record(StatsInterceptor.java:164)     at org.apache.openejb.monitoring.StatsInterceptor.PostConstruct(StatsInterceptor.java:101)     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)     at java.lang.reflect.Method.invoke(Method.java:597)     at org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:162)     at org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:144)     at org.apache.openejb.core.interceptor.InterceptorStack.invoke(InterceptorStack.java:122)     at org.apache.openejb.core.stateless.StatelessInstanceManager.ceateInstance(StatelessInstanceManager.java:237)     at org.apache.openejb.core.stateless.StatelessInstanceManager.createInstance(StatelessInstanceManager.java:448)     at org.apache.openejb.core.stateless.StatelessInstanceManager.access$100(StatelessInstanceManager.java:72)     at org.apache.openejb.core.stateless.StatelessInstanceManager$BackgroundThread.run(StatelessInstanceManager.java:507) Caused by: java.rmi.UnmarshalException: Error unmarshaling return header; nested exception is:      java.net.SocketException: Connection reset     at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:209)     at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:142)     at sun.rmi.server.ActivatableRef.invoke(ActivatableRef.java:124)     at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:178)     at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:132)     at $Proxy43.createService(Unknown Source)     at com.esri.arcgis.discovery.ejb.util.EJBBase.initService(EJBBase.java:340)     ... 22 more Caused by: java.net.SocketException: Connection reset     at java.net.SocketInputStream.read(SocketInputStream.java:168)     at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)     at java.io.BufferedInputStream.read(BufferedInputStream.java:237)     at java.io.DataInputStream.readByte(DataInputStream.java:248)     at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:195)     ... 28 more </Msg>



As additional information:

  • All firewall systems on my machine are fully deactivated

  • No recent modification has been made to the networking in my LAN

  • I installed a few Windows updates last week

Can anyone help me in solving the issue?


Another question: could this problem be tied to a "dirty" Python installation? I'm not sure that GP services rely on Python (I'm getting a Java exception!)

Outcomes