Connection to callback URL '[serivces url]' failed

[norie]

I've been getting this error:

Connection to callback URL 'http://[SERVER NAME]:6080/arcgis/services' failed. For more information, see the ArcGIS Server help topic �?Ports used by ArcGIS Server�?. You can access this topic in the table of contents by navigating to Administering ArcGIS for Server > Securing your ArcGIS Server site > Configuring a secure environment for ArcGIS Server.

I am managing the ArcGIS Server directly from the server itself. The server is hosting ArcGIS Server with GEP. The server had no problems last week, so not really sure what has happened since then.

The error initially started appearing during service publishing this week. Failing with "Failed to Execute (UploadServiceDefinition)" and "Failed to Execute (Publish Service Definition)" in ArcMap and the Connection to callback failure message on the server logs. Along with this Exception/StackTrace below.

Since this instance wasn't hosting anything of value, I mistakenly decided to reinstall. Now I get the Connection to callback url failure message during the site creation.

Any ideas on how to fix this?

com.esri.arcgis.discovery.servicelib.AGSException: AutomationException: 0xc00cee3a - at com.esri.arcgis.discovery.catalog.RemoteCatalog.handleRequest(RemoteCatalog.java:71) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322) at sun.rmi.transport.Transport$1.run(Transport.java:177) at sun.rmi.transport.Transport$1.run(Transport.java:174) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:173) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:273) at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:251) at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:160) at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:194) at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:148) at com.sun.proxy.$Proxy47.handleRequest(Unknown Source) at com.esri.arcgis.discovery.ejb.impl.ServiceCatalogBean.handleRequest(ServiceCatalogBean.java:105) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:162) at org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:144) at org.apache.openejb.monitoring.StatsInterceptor.record(StatsInterceptor.java:164) at org.apache.openejb.monitoring.StatsInterceptor.invoke(StatsInterceptor.java:92) at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:162) at org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:144) at org.apache.openejb.core.interceptor.InterceptorStack.invoke(InterceptorStack.java:122) at org.apache.openejb.core.stateless.StatelessContainer._invoke(StatelessContainer.java:221) at org.apache.openejb.core.stateless.StatelessContainer.invoke(StatelessContainer.java:174) at org.apache.openejb.core.stateless.StatelessContainer.invoke(StatelessContainer.java:136) at org.apache.openejb.server.ejbd.EjbRequestHandler.doEjbObject_BUSINESS_METHOD(EjbRequestHandler.java:238) at org.apache.openejb.server.ejbd.EjbRequestHandler.processRequest(EjbRequestHandler.java:129) at org.apache.openejb.server.ejbd.EjbDaemon.processEjbRequest(EjbDaemon.java:196) at org.apache.openejb.server.ejbd.EjbDaemon.service(EjbDaemon.java:149) at org.apache.openejb.server.ejbd.EjbServer.service(EjbServer.java:71) at org.apache.openejb.server.ejbd.KeepAliveServer$Session.service(KeepAliveServer.java:213) at org.apache.openejb.server.ejbd.KeepAliveServer.service(KeepAliveServer.java:233) at org.apache.openejb.server.ejbd.EjbServer.service(EjbServer.java:66) at org.apache.openejb.server.ServicePool$2.run(ServicePool.java:91) at org.apache.openejb.server.ServicePool$3.run(ServicePool.java:120) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) Caused by: java.lang.Exception: AutomationException: 0xc00cee3a - at com.esri.arcgis.discovery.catalog.Catalog.handleRequest(Catalog.java:147) at com.esri.arcgis.discovery.catalog.RemoteCatalog.handleRequest(RemoteCatalog.java:67) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322) at sun.rmi.transport.Transport$1.run(Transport.java:177) at sun.rmi.transport.Transport$1.run(Transport.java:174) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:173) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667) ... 3 more Caused by: AutomationException: 0x0 - null at com.esri.arcgis.system.Message.readXML(Unknown Source) at com.esri.arcgis.discovery.catalog.Catalog$a_.a(Catalog$a_.java:376) at com.esri.arcgis.discovery.catalog.Catalog$a_.a(Catalog$a_.java:351) at com.esri.arcgis.discovery.catalog.Catalog$a_.run(Catalog$a_.java:288)

[StephanieSnider]

We've created organizational trusted certificiates using the server name as well as the server name with the domain.  Both are throwing back the same errors.  The web browser doesn't have a problem with the certificate.  I can access manager and rest services fine.  However, ArcGIS Server will not start the Printing Tools service and gives these errors.  We are using the same setup method in production (where we are getting the errors) that we did in development and test.  The trusted cert without domain worked fine in dev and test.  I suspect its a blocked port on the DMZ where our production server is located.  I sent the list of required ports to our IT dept, but I haven't confirmed with them that they actually did open those ports - and I don't have access to that server.  I'll keep trying.  Let me know if you hear anything good back from ESRI tech support.

[StephanieSnider]

Just to followup.  For my situation, we created a new site and restored the directories.  This fixed the error messages.  ESRI said that something must have been corrupted during the initial site creation.

[BenRay]

I am also getting this error on an Ubuntu 64 bit 12.04 machine..(I know this is not on the approved OS list). However, I am getting this error during site creation.

Connection to callback URL 'http://servername:6080/arcgis/services' failed. For more information, see the ArcGIS Server help topic �?Ports used by ArcGIS Server�?. You can access this topic in the table of contents by navigating to Administering ArcGIS for Server > Securing your ArcGIS Server site > Configuring a secure environment for ArcGIS Server.

It is on ArcGIS 10.2 Server for Linux.

Any fix on this yet?

[KennethOGuinn]

Hello,

When a callback URL error is being thrown, it is because the ArcGIS Server itself cannot connect back to itself on the URL specified in the error.

This can be caused by a number of things, but the best method I have found for troubleshooting this particular error is to run your browser as the user that is running your ArcGIS Server service. Then browse to the URL in the error (Linux or Windows.) The reason why this is helpful is this allows you to see what the ArcGIS Server would see when connecting to the ArcGIS Server URL specified in the error message.

For example I have seen two things cause this error in the last month, though there could be many more reasons in the same way that many things can network connections to fail for specific users. Both cases were diagnosed using and resolved using the above method. So far I have yet to see this error caused by a bad install or corrupt data, instead it's been environmental.

1) The proxy settings for the service account running the ArcGIS Server Service did not have the same proxy settings as other users on the domain applied to it. (Normal users were setup to have exceptions for local addresses, but the service account was not.
As a result, the proxy was interfering with the ArcGIS Server reaching its own address while other users could browse the manager page normally. So when attempting to start/stop a service, the callback URL error would be thrown as the server attempts to connect to its HTTP/HTTPS listener.
To change this we ran Internet Explorer as the service account. In Windows Shift+Right click Internet Explorer to bring up the pop up menu to allow us to select "Run as different user." (In Linux you could  "su - " to the account running your service, and run your preferred browser.)
Since this case was Windows, we Opened up: Tools > Internet Options > LAN Settings > Advanced, then set our proxy settings to match the settings of the other users. In this case adding an exception for the URL of the ArcGIS Server.

2) The account running the ArcGIS Server Windows Service could not download the Certificate Revocation list for the certificate being used in ArcGIS Server.
So the effect was users could browse the manager page via https and could check that the CRL was fine and the certificate had not been revoked. However, when the account that ArcGIS Server was running under tried to check the CRL, it was not allowed to download the CRL as a result of Group Policy. When  ArcGIS Server is setup to use HTTPS, if it cannot verify a Certificate Revocation list (assuming the certificate has one,) it assumes to error on the side of caution and not make the connection to the server with an unverifiable certificate.

Two additional tests to help resolve this issue:

1) Rerun the Configure ArcGIS Server account to have the ArcGIS Server run under your domain account, this way you know AGS has the same settings as you in terms of what you can browse.

2) If you are getting Callback URL errors that reference 6443. You can tell if it's HTTPS related by switching the HTTPS settings for ArcGIS Server back to HTTP only.  If the problem goes away, it is something related to your HTTPS/Certificates.

Additionally, issues with HTTPS/SSL  will also tend to have to be followed by (WinInet error code = xxxxx) If you google the WinInet error code, it often will tell you what is the problem .

I hope this information helps!

[LakshmananVenkatesan]

Hi, I am facing same issue in ArcGIS server 10.3.1 . Any additional clues or directions?

[KennethOGuinn]

Hi S R,

Did you try switching the account running the ArcGIS Service to your domain account? If your user account can access the server via 6443, if you run ArcGIS Server under those same credentials it should also be able to reach the callback URL.

I'd double check with your IT folks to make sure this is ok to do as a test, but I suspect you'll find it fixes it. (Conversely, if you were to log in as the account running the ArcGIS Server Service, I suspect you will see that if you browse to http://server.domain.com:6080/arcgis/manager that it will not open the page.)

Hope this helps,

Ken O.

[LakshmananVenkatesan]

Ken, I have created new thread and provided my results and observation https://community.esri.com/message/542623#542623 Summary: a) if ArcGIS server service running with local admin account, services are working fine in HTTP and HTTPS mode. b) if ArcGIS server service running with Service account (the account specifically created for running services like ArcGIS server, FME and others softwares) does not work in HTTPS mode. The call back error is thrown. I need to run ArcGIS server using SERVICE account because this account has privilege to connect to internet, which is must for running my Geoevent processor which connects to third party data. It looks like Service account not able to make a call back in HTTPS mode for some reason. How to find what was exact reason for the call back issue on HTTPS mode for some accounts? I tried deleting site and recreated ; recreated new self signed certificate.

[KennethOGuinn]

Hi S R,

Probably the easiest way to see what is happening is to run a web browser as the service account, and then open up https://server.domain.com:6443/arcgis/manager.

This will let you see what the AGS sees when it tries to reach the callback URL.

It could be a few different things, but it does sound like the service account is differently configured from the regular users.

Below is a list of things I've seen cause the issue, though there are probably many other things that could cause a problem:

- Service account does not have proxy configured, but regular account does. (You can check this by running Internet Explorer as the service account, then Checking the Internet options for that Service Account.)

- Service account does not have the Domain CA that sighed the server cert in it's trusted CAs. (you can check this in certmgr.msc if it's in Windows.)

- The trust settings have an excpetion for http://server.domain.com but not https://server.domain.com

Hope this helps!

Ken O.

[LakshmananVenkatesan]

Ken,

I will check all these and get back to you. I need to take a help of IT team for this to test.

[LakshmananVenkatesan]

Ken, Would you mind explaining Point 3 and 4 little in detail. a) how to check domain CA in trusted CA? b) Do you mean Trust settings in IE?. how to check?. Generally many of IE settings are controlled by IT admin folks.