ArcGIS Web Adaptor 11.1 App Pool freezes

[Scott_Tansley]

Hi.   

I've recently upgraded a client to ArcGIS 11.1, and we're having random problems with the new web adaptors.  I'm looking to see if anyone has observed the same issues.

So the clients ArcGIS Enterprise was deployed at 10.8.1.  There's an IIS Web Server in the DMZ.  A single host with the rest of the base deployment, which is only used for Hosted Feature Services.  WA's exist for portal and hosting.  There is a third machine with a general-purpose ArcGIS Server, federated and primarily serving Map Image Layers.  There is a Web Adaptor called server.

There have never been any repeated outage issues.  The environment was upgraded to 10.9.1 last year.  Once again no issues.

They were upgraded to 11.1 two weeks ago.  Immediately, we found that the machine was running out of memory, we noted the advice given in the new dependencies, and increased the RAM from 4 to 8GB.  It sits at around 5-6GB with no issues, and we have not seen any spike above 6GB to date.

After adding RAM it all seemed to settle for a few days.  But now, every couple of days the IIS application pool for the 'server' web adaptor will just stall.  IIS logs show 200/304 responses for everything up to the freeze/stall and 500 for everything.  There is nothing untoward in the requeste.

ArcGIS Server is still available on 6443 and can be accessed.  It just isn't receiving requests from IIS.  With Info logging turned on, it shows the last good 200 request from the WA.  Then nothing, no errors, no issues.  It's just as if it's sat there waiting for a request and not receiving it.

There have been no firewall or environmental changes recently, the only change is the upgrade to 11.1 and the addition of memory.

On the web server there is nothing in event viewer, system/admin/security or IIS application logs.

I'm blind.  It's just as if the App Pool WA says I've had enough.

The only way to bring the application back online is to restart IIS.  On the AppPool you can stop it.  But it will not start unless IIS is restarted. 

I'm currently blind.  We've external ping monitoring in place so we know when the healthCheck API fails, but there's nothing else we can do but monitor and restart at this point.

[DavidColey]

I was able to update the hosting site 'server' web adaptor after stopping the web application pools same as @RyanUthoff and @SebastienPetit  - although I did have to run the patch twice to update the 'server' WA (my federated site and portal updated previously without having to stop their respective pools)

I suspect the first time I ran the update following the method tech support told Ryan to do is because the files were still locked, and it just needed a few more minutes to clear.   

If the 'server' WA still hadn't updated then I would have ran again per @MarceloMarques  - but fortunately did not have to

[KaitlynPurcell]

All, thank you again for your responses and work with Tech Support. The very latest feedback is corroborating our investigation that the patch may not be applied completely because of a file lock. Sometimes simply re-running the patch is successful. However, as has been recently noted in this thread, and based on other reports, the best practice is to temporarily stop the ArcGIS Web Adaptor’s application pool before applying the patch. Please refer to the steps below if you are encountering the problem where the ArcGIS Web Adaptor (IIS) 11.1 Reliability Update 2 Patch is not being successfully applied to all of your instances of the ArcGIS Web Adaptor (IIS).

1. Identify all 11.1 ArcGIS Web Adaptors that have not been updated when applying the patch.
   • Open Windows Explorer, and navigate to inetpub/wwwroot/(context) folder, where (context) is the name of the ArcGIS Web Adaptor instance. Check the date stamp of the ESRI.ArcGIS.WebAdaptor.dll. A fully patched ArcGIS Web Adaptor should show a date stamp of 8/10/2023 on the ESRI.ArcGIS.WebAdaptor.dll. Repeat this check for each ArcGIS Web Adaptor instance on the machine. For example, an ArcGIS Enterprise deployment installed by the ArcGIS Enterprise Builder will create two instances of the ArcGIS Web Adaptor (IIS), one is named portal and the other is named server. In this example, the application pools will be named ArcGISWebAdaptorAppPoolportal and ArcGISWebAdaptorAppPoolserver.
   • If there is uncertainty about how many ArcGIS Web Adaptor (IIS) instances are installed on the machine, go to the Windows Control Panel > Programs > Programs and Features to see the list of installed ArcGIS Web Adaptors with their respective instance names.
2. Close the Windows Explorer that was used to view the ESRI.ArcGIS.WebAdaptor.dll date stamp. This step is important to ensure no inadvertent locking of Web Adaptor files
3. Open Internet Information Services (IIS) Manager.
4. Click the Application Pools node in the tree view.
5. Based on the results from step #1, stop all ArcGISWebAdaptor application pools that have not been updated. For example, if both the portal and server instances were not updated, right-click ArcGISWebAdaptorAppPoolportal and choose Stop. Then right-click ArcGISWebAdaptorAppPoolserver and choose Stop.
   • Please note that depending on the activity of your web server, the actual stop and release of files may take longer than is indicated by the Internet Information Services (IIS) Manager user interface.
6. Close Internet Information Services (IIS) Manager.
7. Run the ArcGIS Web Adaptor (IIS) 11.1 Reliability Update 2 Patch. Wait for the patch install process to complete for all instances of the ArcGIS Web Adaptor (IIS) you have installed.
8. Open Internet Information Services (IIS) Manager.
9. Start all ArcGISWebAdaptor application pools. For example, right-click ArcGISWebAdaptorAppPoolportal and choose Start. Then right-click ArcGISWebAdaptorAppPoolserver and choose Start.
10. Open Windows Explorer. Navigate to inetpub/wwwroot/(context) folder where (context) is the name of the ArcGIS Web Adaptor instance. Check that the ESRI.ArcGIS.WebAdaptor.dll has a date stamp of 8/10/2023. Repeat this check for every ArcGIS Web Adaptor (IIS) on the machine.
11. If you find that the ESRI.ArcGIS.WebAdaptor.dll still does not have a date stamp of 8/10/2023 after following this workflow to apply the ArcGIS Web Adaptor (IIS) 11.1 Reliability Update 2 Patch, please log an incident with Esri Tech Support.

The details provided here will be made available both in the patch page for ArcGIS Web Adaptor (IIS) 11.1 Reliability Update 2 Patch and an Esri Technical Article. We will post those links when available.

Esri plans to re-release the ArcGIS Web Adaptor (IIS) 11.1 Reliability Update 2 Patch with additional install logic so these manual steps of restarting the ArcGISWebAdaptor application pools are not necessary. This enhanced install logic would be included in any future patches for the ArcGIS Web Adaptor (IIS) 11.1.

Regards,

Kaitlyn

[KaitlynPurcell]

Hello all, please note that the following defect has been logged to address this issue:

• BUG-000161467 - ArcGIS Web Adaptor (IIS) 11.1 Reliability Update 2 Patch may not install completely to all ArcGIS Web Adaptor instances due to a file lock.

This should be published to the Support site soon - https://support.esri.com/en-us/overview

Additionally, we've published a technical article to help determine whether you are facing this issue, and the steps to successfully apply the patch - ArcGIS Web Adaptor (IIS) 11.1 Reliability Update 2 Patch may not install completely to all Web Adapt...

A note was added to the ArcGIS Web Adaptor (IIS) 11.1 Reliability Update 2 Patch page summary which points to the technical article mentioned above. 

Best,

Kaitlyn

[KaitlynPurcell]

The ArcGIS Web Adaptor (IIS) 11.1 Reliability Update 2 Patch B version is now live on the support site. The URL is: 

https://support.esri.com/en-us/patches-updates/2023/arcgis-web-adaptor-iis-11-1-reliability-update-2...

I want to highlight this note from the patch page:

Important note September 25, 2023:

The ArcGIS Web Adaptor (IIS) 11.1 Reliability Update 2 Patch has been re-released, with a B version suffix on the name, to correct install failures related to file locks. The install of this patch now includes commands to stop the ArcGIS Web Adaptor application pools in order to release any file locks and allow the patch to be applied completely to all instances of the ArcGIS Web Adaptor (IIS) 11.1. If the first attempt to run the patch does not update all of your deployed ArcGIS Web Adaptor instances, please run it a second time. A fully patched system will show all web adaptor instances, under inetpub/wwwroot/(context) folder, with a ESRI.ArcGIS.WebAdaptor.dll file dated 8/10/2023. Esri recommends that all users of the ArcGIS Web Adaptor (IIS) 11.1 download and install this corrected patch.

Regards,

Kaitlyn

[AndresEcheverri]

Has anyone implemented the The ArcGIS Web Adaptor (IIS) 11.1 Reliability Update 2 Patch B version with no additional issues? Did it fix the issues with the web-adaptor?. Thanks for any info.

[Scott_Tansley]

I have several clients running the 2B Patch with no issues.  One had a single outage shortly after installing it, but that was weeks ago with no recurrence.  It may not even have been related. 

[SebastienPetit]

We installed it without issues and no more problems since we did

[Edgar_W_Iparraguirre]

We have implemented the latest patch at a number of our clients, since then we have not received any pointers about the issue ... so we can say that the final patch did the trick.

[MarceloMarques]

FYI: I encountered the issue again with Patch B, to update the Portal WebAdaptor, the ArcGIS Server WebAdaptor worked fine thought, on my ArcGIS Enterprise machine the workaround was to stop the windows services for the datastore, arcgis server and portal then apply the webadaptor patch again.

Bug: ArcGIS Web Adaptor (IIS) 11.1 Reliability Update 2 Patch May Not Install Completely t (esri.com...