Help with intermittent(WARNING/SEVERE) logs on ArcGIS server manager.

[Yogesh_Chavan]

We have a multimachine ArcGIS Enterprise deployment and since last week we are seeing the below waning and SEVERE logs on ArcGIS Server Manager.

All the services seems to work fine, datastore machine is working fine as well, and there is no communication conflict between machines hosting server, portal and datastores.

Did someone encountered such logs before? What could be triggering this?

[A_Wyn_Jones]

Hi @Yogesh_Chavan,

It looks like there may be a permissions error on ArcGIS Datastore which is preventing ArcGIS Server to read the logs of Datastore.

• Are all components running from the same ArcGIS Service account?
• Can you please verify the permissions of c:\arcgisdatastore 
• Please validate the Datastore through ArcGIS Server admin endpoint:https://enterprise.arcgis.com/en/portal/latest/administer/windows/data-store-function-reference.htm#....

[Yogesh_Chavan]

Hello @A_Wyn_Jones,

Thank you for sharing your insights.

Kindly find the response below:

• Are all components running from the same ArcGIS Service account?

Yes

• Can you please verify the permissions of c:\arcgisdatastore 

Service account have full control over this

• Please validate the Datastore through ArcGIS Server admin endpoint:https://enterprise.arcgis.com/en/portal/latest/administer/windows/data-store-function-reference.htm#....

Not sure what is going wrong. We actually started observing this since last 2-3 weeks. 

[A_Wyn_Jones]

@Yogesh_Chavan It looks like your Datastore is offline.

On the datastore machine, can you visit https://localhost:2443/arcgis/datastore and see a webpage?

It's also worth checking that you can get to https://Datastoremachine.domain:2443/arcgis/datastore from the ArcGIS Server machine - this will check if any networking has changed.

This issue would only be noticeable if you're viewing Hosted content in your Portal - if the majority of your layers are hosted on a separate Database like SQL or Postgres then you wouldn't necessarily notice. 

[Yogesh_Chavan]

Thank you @A_Wyn_Jones for you helpful insights.

To our findings we tried following your suggestions and observed that the server machine fails to connect to https://Datastoremachine.domain:2443/arcgis/datastore.

This works well in datastore machine for both localhost and domain URL.

There is one additional info that I think is quite suspicious. We are using NLB URLs configured on top of our web adaptor URLs for accessing server and portal, and those were configured before installing and registering datastore machines with the server.

Can this could be a possible trigger for this issue? Or should we focus on why this domain URL fails to communicate from the server machine?

Adding some more info from the describedatastore.bat utility we observed that the machine with relational datastore's owning system URL is web adaptor URL whereas the other machine with tile cache datastore shows Port URL.

We have 2 datastores installed on separate machines, and the issue with errors/logs is in relational.

[Yogesh_Chavan]

Hello @A_Wyn_Jones,

Were you able to review the above details? Just in case I missed out on some critical info earlier.

Also, would it be a good idea to un-register the relational datastore from the server and re-register using the Port or NLB URL and not the Web Adaptor URL?

Reading through the docs makes us clear that hosted services would not work? Does a re-registered datastore makes the services operate normally?

Thank you so much. Feel free to review when you have some off hours:)

[A_Wyn_Jones · Accepted solution]

@Yogesh_Chavan it's likely that the Server to Datastore machine communication is blocked by port configuration. 

Have a look at this port diagram https://enterprise.arcgis.com/en/system-requirements/latest/windows/pdf/ports-enterprise-deploy-dgm....

Note that port 2443 is required both inbound/outbound between Server and Datastore - loading the web page https://Datastoremachine.domain:2443/arcgis/datastore from the Server machine is a good way to test this.

When registering Datastore, I'd suggest using ArcGISServermachine.domain to register with ArcGIS Server machine - you shouldn't need to configure through the load balanced endpoints unless there's a specific configuration requirement e.g. highly available stack.

To check that Datastore can communicate with your Server - web browse to https://ArcGISServermachine.domain:6443/arcgis on the Datastore machine.

[Yogesh_Chavan]

Thank you @A_Wyn_Jones for your insights.