ArcGIS Web Adaptor 11.1 App Pool freezes

[Scott_Tansley]

Hi.   

I've recently upgraded a client to ArcGIS 11.1, and we're having random problems with the new web adaptors.  I'm looking to see if anyone has observed the same issues.

So the clients ArcGIS Enterprise was deployed at 10.8.1.  There's an IIS Web Server in the DMZ.  A single host with the rest of the base deployment, which is only used for Hosted Feature Services.  WA's exist for portal and hosting.  There is a third machine with a general-purpose ArcGIS Server, federated and primarily serving Map Image Layers.  There is a Web Adaptor called server.

There have never been any repeated outage issues.  The environment was upgraded to 10.9.1 last year.  Once again no issues.

They were upgraded to 11.1 two weeks ago.  Immediately, we found that the machine was running out of memory, we noted the advice given in the new dependencies, and increased the RAM from 4 to 8GB.  It sits at around 5-6GB with no issues, and we have not seen any spike above 6GB to date.

After adding RAM it all seemed to settle for a few days.  But now, every couple of days the IIS application pool for the 'server' web adaptor will just stall.  IIS logs show 200/304 responses for everything up to the freeze/stall and 500 for everything.  There is nothing untoward in the requeste.

ArcGIS Server is still available on 6443 and can be accessed.  It just isn't receiving requests from IIS.  With Info logging turned on, it shows the last good 200 request from the WA.  Then nothing, no errors, no issues.  It's just as if it's sat there waiting for a request and not receiving it.

There have been no firewall or environmental changes recently, the only change is the upgrade to 11.1 and the addition of memory.

On the web server there is nothing in event viewer, system/admin/security or IIS application logs.

I'm blind.  It's just as if the App Pool WA says I've had enough.

The only way to bring the application back online is to restart IIS.  On the AppPool you can stop it.  But it will not start unless IIS is restarted. 

I'm currently blind.  We've external ping monitoring in place so we know when the healthCheck API fails, but there's nothing else we can do but monitor and restart at this point.

[LukeSavage]

It would be interesting to understand hardware/resource usage as well as network throughput.  Procmon can trail this to see when it breaks.  Does anyone have ArcGIS Monitor to trace these instances?

[DavidColey]

We have a distributed site, as I have mentioned earlier in this post  For our hosting site, the web adaptor is named 'server' and the ESRI.ArcGIS.WebAdaptor.dll is  dated March 1, 2023. 

For our federate site, the web adaptor is named 'agsfed' and the ESRI.ArcGIS.WebAdaptor.dll is dated August 10, 2023.  For the portal WA, the ESRI.ArcGIS.WebAdaptor.dll is dated August 10, 2023.

We do not run ANY map and or image and / or server extensions from the hosting site - it purely serves hosted feature layers from portal.

All of our map and image and (network, validation, versioning, geolocator) etc services are run through the agsfed WA, per our system design.

All 3 WA's, server, agsfed, portal all show the patch as applied through 'installed updates' as accessed through control panel>view installed updates.

For IIS, I don't know what multiple bindings means.  We only have 80 and 443 showing as bindings on the Default Web Site.  There are no bindings on anything else.

To me, given what @eryildiz just reported, this suggests that ESRI did not expect anyone to be running traditional types of services from the hosting site.  If that's the case, that would seem like an oversite for sure.

That being said,  ESRI did not expect anyone to have the letters a-d-m-i-n in the name of their federated web adaptor site at 11.0 - like I do 'agsfed' - yet that did happen and I could not run a webgisdr until I upgraded to 11.1.

So - now I understand why maybe we 'got lucky'.  We are not running anything other than hosted feature layers from the hosting site.  All traditional services are accessed from the federated site. And although we have many (250 combined maybe?) traditional map, image, feature access, versioning, geolocator, validation, networking services running through the federated site, that app pool did not freeze before or after patch updates. 

I should note that wherever possible, all services on agsfed are using the 'shared instance' dynamic mapping service.  I have that set to 16 instances, appropriate for an 8 core 32gb server.

hope this helps some too - 

David

[eryildiz]

I just remembered that I forgot something, and I assume that it is also something necessary. The patch does not apply if you have multiple arcgis application pools. As you know, Esri has changed the application pool on 11.1. We were not sure which one, so we just removed both of them. Uninstalled and reinstalled web adaptor again. And applied the patch after that. This still didn't work until we removed one of the bindings , however this is also something that needs to be taken into consideration. 

When installing the patch, it doesn't give an error even if it doesn't install properly. You actually think it's installed. Just keep that in mind.

[DavidColey]

I'm sorry but this doesn't make sense to me: "The patch does not apply if you have multiple arcgis application pools."  Yes, at 11.1 each WA now runs in it's own application pool.  So you if you have any kind of Enterprise deploy, you are going to have at least 2 WAs - one for portal, one for the hosting server site.  We have 3 - portal, federated site, hosting site plus the original ArcIGISWebApaptorAppPool v4.0

 

But as I just noted above, the patch updated 2 of the WA's - according to the dll dates - although the patch ran 3 times - once per 11.1 v6 web adaptor.  The patch summary notes say nothing about stopping the IIS to run the updates. 

Is esri now recommending that the patch be re-applied while the IIS is stopped so that the ArcGIS-111-WAI-RU2-Patch.msp patch will update web adaptor pool for 'server'? Such that the ESRI.ArcGIS.WebAdaptor.dll for the 'server' will show as updated?  Because 'View Installed Updates' certainly shows that the updates have been applied.   @KaitlynPurcell ? @JonEmch ?

[SebastienPetit]

Having the same issue with only one of our ArcGIS Server (11.1 fully patched).

I will check if webadaptor patch 2 was correctly installed (I thought it was)

EDIT (seeing dll dates, it seems it is not applied correctly even if findpatch sees it).

Tried to install it with IIS stopped but has been hanging for 15 minutes before i killed the process

And try the workaround suggested by Luke

[KaitlynPurcell]

All, thank you to those who logged a case with Tech Support, and allowed us to look further into reports of continued instability even after the ArcGIS Web Adaptor (IIS) 11.1 Reliability Update 2 Patch has been installed. The investigation is still active. Based on several findings, and also indicated by the most recent posts on this thread, we are finding that the patch is not actually being fully applied. The patch install is not displaying errors but an inspection of the affected files is showing an earlier and thus incorrect time date stamp. In some cases, the patch is being correctly applied to one but not all instances of the ArcGIS Web Adaptor (IIS). The suspicion is that the files to be patched were in use and thus locked, preventing the patch from completing successfully. The file lock could occur because the ArcGIS Web Adaptor Configuration page has been opened (before all instances were patched), or if your ArcGIS Enterprise deployment is receiving requests during the patch installation.

To verify the ArcGIS Web Adaptor (IIS) 11.1 Reliability Update 2 Patch is fully installed, navigate to inetpub/wwwroot/(context) folder where (context) is the name of the ArcGIS Web Adaptor instance. Check that the ESRI.ArcGIS.WebAdaptor.dll has a date stamp of 8/10/2023. Repeat this check for every ArcGIS Web Adaptor (IIS) on the machine. An ArcGIS Enterprise deployment installed by the ArcGIS Enterprise Builder will create two instances of the ArcGIS Web Adaptor (IIS), one is named server and the other is named portal. If there is uncertainty about how many ArcGIS Web Adaptor (IIS) instances are installed on the machine, go to the Windows Control Panel > Programs > Programs and Features to see the list of installed ArcGIS Web Adaptors with their respective instance names.

If you find that the ESRI.ArcGIS.WebAdaptor.dll does not have a date stamp of 8/10/2023, try again to install the ArcGIS Web Adaptor (IIS) 11.1 Reliability Update 2 Patch. After applying the patch again, repeat the check of the date stamp on the ESRI.ArcGIS.WebAdaptor.dll for all instances. If a re-running of the patch is not successful, please then log an incident with Tech Support. 

Regards,

Kaitlyn

[RyanUthoff]

Thank you for the update! I just received this information from the Esri support analyst I'm working with today. We did confirm that the Web Adaptor patch appeared to be installed when viewing the installed updates in Control Panel. However, the time stamps of the .dll files did not appear to be correct.

Support told me to stop the IIS application pool of the Portal and Server web adaptor (like others have mentioned here) and then install the patch again which I did just a few minutes ago, and the .dll files have the correct date now.

[SebastienPetit]

Exactly the same.

The only way to get it installed correctly was to stop the application pool.

I tried without and no success.

I stopped them and done.

[MarceloMarques]

FYI: I encountered this issue on my ArcGIS Enterprise machines, even when I applied the webadaptor patch a 2nd time, the workaround was to stop the windows services for the datastore, arcgis server and portal then apply the webadaptor patch again.

[LukeSavage]

.net core requires changes in configuration for resources unlike .net framework.  We are experiencing this with our own software so looking at Microsoft's requirements for IIS settings maybe something the esri DEV team should be looking at.