ArcSOC Optimizer for ArcGIS Monitor

[AndrewSakowicz]

Overview

ArcSOCOptimizer is a python command line utility.  It can help administrators:

1. Reduce memory by identifying low usage services with dedicated pooling and reducing the overall number of arcsoc.exe processes:
   1. changing dedicated to shared instance pool, if supported  
   2. reducing min instances for dedicated instance pool services
2. Improve performance by identifying high usage service and allocating dedicated arcsoc.exe processes if available memory:
   1. changing shared to dedicated instance pool
   2. increasing min instances for dedicated instance pool 

Get started

To run using provided executable for windows, unzip ArcSOCOptimizer.zip from

https://prof-services.maps.arcgis.com/home/item.html?id=f9c3e2864054444fb87e54575b11a96a

and execute from command line:

ArcSOCOptimizer.exe -f SampleConfig.json

SampleConfig.json

edit:false/true

False executes in read only and produces a list of recommendation.  Start with False, review and validate the results.  Once validated, execute true. 

past_days:number     

Number of past days to be analyzed to determine low and high usage.

The recommendations are based on historical data.  You may choose to increase the 7 day default.

avg_sec/day:float       

Average number of seconds per day a service is used throughout all past_days.  For example, if during the business hours service is used 10 seconds per day and 0 seconds during the weekend, the avg_sec/day=avg (10,10,10,10,10,0,0).  This can be evaluate using the combination req/s and response time, see below

shared_to_dedicated: true/false      

Allows or disallows low usage services to be moved from dedicated to shared.

dedicated_to_shared: true/false      

Allows or disallows high usage services to be moved from shared to dedicated.

host_address:array

Each arcgis server machine should be registered with ArcGIS Monitor as a host component.  This parameter takes a list of hosts (in quotes), with the names exactly as the Address of registered host, see below. 

Use case

Always test in your non-production environment first.

edit=false

Start with edit=false (default).  This will generate a report with recommendations, without making any changes. Review the recommendation and change the inputs if needed, especially avg_sec/day for low and high usage.  Repeat the process as needed.

For example, in the following use case we have the following 4 services:

1. ID 1 and ID 2 have a low usage (<5s) and already recommended shared pool.  Therefore, there is no change.
2. ID 3 has a low usage (<5s), but dedicated pool. Therefore, pool change from dedicated to shared is recommended.
3. ID 4 has a high usage and shared pool.  Therefore, pool change to dedicated and min/max instances are recommended

edit=true

Once you validate the edit=false recommendation, test executing edit=true manullay.  This mode will make changes to instances and pooling and restart services.  If satisfied, user could schedule this as a daily job, using windows scheduled task.

Note, the minProposed is calculated as the ceiling of Instances Used Avg.  In the case below, the ceiling of 1.21 is equal 2.

Memory limit

If memory available is less then specified memory_available_GB limit, the edit will not be executed.  In that case, user should lower this limit or free up memory manually.

Note, this utility is not an ArcGIS Monitor product supported feature or a supported Esri product.

[GaryNEORSD]

Looks like a great tool, but I am having issues setting it up, particularly arcgis_server in the JSON file.  Any plans on making this tool open-source by chance?

[AndrewSakowicz]

@GaryNEORSD FYI, source code included in https://prof-services.maps.arcgis.com/home/item.html?id=f9c3e2864054444fb87e54575b11a96a . Also, what issue do you have?

[GaryNEORSD]

@AndrewSakowiczThanks, I was able to access the desktop application using that link.  I mean to say is the python file(s) used to build the executable available?

[AndrewSakowicz]

@GaryNEORSD yes, I just included python files.  Please download again.  You should see source_code_xx.zip folder with the python code

[meloking_PNM]

@AndrewSakowicz I've downloaded the zipped file, extracted, and adjusted the SampleConfig.json with my ArcGIS Monitor/ArcGIS Server's values as suggested above. Could you provide guidance on what is causing the error below when I run the exe via command line as described above?

[René_Ténière]

I am having the same issues @AndrewSakowicz 

[AndrewSakowicz]

@meloking_PNM , @René_Ténière  please check component_address string for both arcgis server and host in config file

are exactly as in ArcGIS Monitor

[René_Ténière]

Thanks! That was the issue.

[LouEarly]

@AndrewSakowicz 

Im having issues with the json as well. See the image below.  The 02 machine is our ArcGIS server and the 08 machine is the server with monitor installed. When running the exe in cmd i'm getting the same error that was shown above. I've checked the URL's and they are exactly whats in monitor.

[AndrewSakowicz]

@LouEarly , it looks like a mistake in your host input.  it should not start with https..., it should be 02.gis.com.  But double check in host address, see below

[MarieCline_Delgado]

@AndrewSakowicz  Is there a final configuration process where this add-on should be imported into Monitor similar to how add-ons were imported as extensions in 10.X versions?  Or is this just intended to be automated through Windows Task Scheduler?

Also curious if the JSON accepts email configs to send the ArcSOC report via email similar to 10.X versions?  

"Email": { "email": "always", "server": "email.somewhere.com", "port": 00, "user": "", "password": "", "sender": "sender@somewhere.com", "receiverList": "receiver@somewhere.com" }

Thanks for your time!

[LouEarly]

@AndrewSakowicz Thanks for the help, I was able to get this configured and working in our DEV environment.  Which leads me to my next question. Our DEV environment has only 1 ArcGIS server.  Our PROD environment has a 5 server ArcGIS Server site accessed through a load balancer with a URL like - https://serversitename.gis.com

I've tried to configure the json using that URL instead of the individual machine names and i'm getting the following error. How is this configured when there is a multiple machine site?

[AndrewSakowicz]

@LouEarly , so you have one site with 5 machines, right?  please provide config file and ArcGIS Monitor screenshot of ArcGIS server component address and all 5 machines.  Please follow documentation stating the input needs to be as registered in ArcGIS Monitor, not your LB.

[AndrewSakowicz]

@MarieCline_Delgado , you are correct, use windows scheduled task.  Sorry, no email option available.

[LouEarly]

@AndrewSakowicz It is configured in monitor using the LB. What config file specifically are you asking for? Sorry for my confusion.

[AndrewSakowicz]

SampleConfig.json.  Note, this is not correct "configure the json using that URL instead of the individual machine names".  Please follow the documentations

[LouEarly]

Thanks for all the help...we were finally able to get this working!!

[ChrisTaylor2]

Hi @LouEarly ,

For my multi-machine site I had to include all of the participating servers in a list for the 'host_address' parameter, like:

"host_address":["server1.domain.com","server2.domain.com",...]

Hopefully that will help.

Chris

[AndrewSakowicz]

Good to hear, @LouEarly  

[MarieCline_Delgado]

@AndrewSakowicz  I have had the ArcSOC Optimizer running smoothly in list mode for about a month now.  I recently switched to edit mode and the reports are listing a couple different errors in the minMaxChange column for the services it attempts to edit.  Do you have any advice on how to solve these errors?

I don't know what to make of this error:
{"status":"error","messages":["Could not find resource or operation 'admin' on the system."],"code":404}

Here, it makes sense that an error is returned if it is trying to use the server admin token to access the portal sharing rest site, I'm just not sure what the solution is:
{"status":"error","messages":["Server machine 'https://x.y.com:7443/arcgis/sharing/rest/community/self' returned an error. 'Invalid token.'"],"code":498}

I appreciate any help you can provide! Thank you!
Marie

[AndrewSakowicz]

@MarieCline_Delgado , please verify your input.  Note you'll have to provide correct client , referer: "arcgis_server":{
"server_admin_url": "https://xxxx:6443/arcgis",
"token_url": "https://xxx:6443/arcgis/admin/generateToken",
"client":"referer",
"referer":"referer-value",
"username": "xxx",
"password": "xxx",
"password_encoding": false,
"host_address":["xxx"]

[MarieCline_Delgado]

Thanks @AndrewSakowicz !  I completely missed the referrer value.  

[MarieCline_Delgado]

Hi again @AndrewSakowicz .  Once I addressed the referrer value, I've had more success, but I'm getting a different error during the edit process.  I have run the task a few different times and received the following 2 errors but never repeating on the same service being edited (of an edit list with about 24 items each time):

10/20/2023 11:36:41 AM INFO: observer run {"success":true}
10/20/2023 11:37:06 AM ERROR: 'features'
10/20/2023 11:37:06 AM ERROR: None
Traceback (most recent call last):
  File "ArcSOCOptimizer.py", line 11, in main
  File "modules\report_builder.py", line 16, in __init__
  File "modules\report_builder.py", line 34, in _execute
  File "modules\ArcSOCOptimizer\edit_mode.py", line 16, in __init__
  File "modules\ArcSOCOptimizer\edit_mode.py", line 27, in _edit
  File "modules\ArcSOCOptimizer\host.py", line 15, in set_stats
  File "modules\ArcSOCOptimizer\host.py", line 41, in _host_id_from_address
KeyError: 'features'

10/20/2023 02:58:35 PM INFO: observer run {"success":true}
10/20/2023 02:58:44 PM ERROR: 'features'
10/20/2023 02:58:44 PM ERROR: None
Traceback (most recent call last):
  File "ArcSOCOptimizer.py", line 11, in main
  File "modules\report_builder.py", line 16, in __init__
  File "modules\report_builder.py", line 34, in _execute
  File "modules\ArcSOCOptimizer\edit_mode.py", line 16, in __init__
  File "modules\ArcSOCOptimizer\edit_mode.py", line 27, in _edit
  File "modules\ArcSOCOptimizer\host.py", line 19, in set_stats
KeyError: 'features'

Any thoughts on what could be causing this and what solution there might be?  Thanks for your time and help.

[AndrewSakowicz]

@MarieCline_Delgado , please double check host_address in your input, as per doc: 

[MarieCline_Delgado]

Hi @AndrewSakowicz ,  I double checked my host addresses and they exactly match the host address of the components.  

[AndrewSakowicz]

@MarieCline_Delgado , the 10/20/2023 11:37:06 issue looks very much like it was an incorrect host address. Did you correct it between 10/20/2023 11:37:06 and 10/20/2023 02:58:44?

For the 10/20/2023 02:58:44 issue, please verify you use the latest, version 20231012.2 from https://prof-services.maps.arcgis.com/home/item.html?id=f9c3e2864054444fb87e54575b11a96a .

Also, verify in  ArcGIS Monitor that Memory Available and CPU Utilized chars have data for this time.

[AntheaTung]

@Hi @AndrewSakowicz ,

I ran a report from ArcSOCOptimizer (source_code-20230530.1) on ArcGIS Monitor 2023.2. I also exported a report from ArcGIS Server 10.9.1 statistics on MaxInstnaceUsed in 30 days.

I compared the two results and the MaxInstanceUsed are not the same in ArcSOCOptimizer and ArcGIS Server Statistics:

ArsSOCOptimizer reports for past 30 days for NBGas_Landbase service: max instance used = 15
ArcGIS Server statistics report for 30 days for NBGas_Landbase service: max instance used = 5

Question: how does ArcSOC Optimizer collects the Max Instanced Used compared to ArcGIS Server Statistics? 

When should I use ArcSOC Optimizer and ArcGIS Server Statistics ? 

Many thanks !

Anthea

[ghaskett_aaic]

Hello.

I recently installed this and am trying to figure out if it is working properly.

The report generated doesn't seem to be gathering all the required data, or it for some reason is rounding values to (0).

We have a multi-tiered environment consisting of 6 servers (WebAdaptor, Portal, Host, Geo, Mapping, and DataStore).  Plus an independent server for ArcGIS Monitor.

Servers are all Window servers running Windows Server 2022 Standard.

Enterprise is running ArcGIS Enterprise 11.2 and Monitor is running 2023.3.

The screen shot below is showing the top 14 services.  I have been testing different dashboards, so other services should be showing values greater than (0) in avg_sec/day and instancesUsed, etc.

I am trying to figure out if this is what should be expected or if something is still a bit off.

Do we know if there are any issues related to using Enterprise 11.2?

Thanks,

George