ArcGIS Server multiple-machine deployment Experiences

[ScottCorwin]

Hi, looking for feed back on going from an singe ArcGIS GIS Server site to a multi-machine site.  Technically it's already a multi-machine site with ArcGIS Image Server on one machine and ArcGIS GIS Server on the other, both are federated.  We are running around 150-160 services utilizing over 80% of our memory sometimes spiking into the 90's when someone starts an edit session in portal.  So one potential solution would be to add another ArcGIS GIS Server.  One specific question I do have, is how does this get managed? Do I as the admin get to decide GP services on machine A and pooled instances on machine B, or does some aspect of the Server software manage this without user input?

[MichaelJenkins]

When you have multiple servers joined as a single "site", all of the services are managed the same for both servers.  In fact you can manage the services from ArcGIS Server Manager connected to either machine.

You can still set service properties, but those properties will apply to both machines.  For example, if you use a dedicated instance and set it to a maximum of 5, that is 5 per server so you are really getting a maximum of 10 instances.  If you turn off a service, it is turned off on both machines.

You cannot assign services to one machine or the other, though.  To do that you would  just federate another server to the Portal and then when you publish a service you can choose which server it will run on.

One other consideration is that both machines must access the same \config-store and \arcgisserver directories, so you would either set those up on a 3rd server or some sort of network file share, or you would put them on one of the two AGS servers and share it so the other mahine can access it too. 

[ScottCorwin]

Thank you.  If instances are pooled, now 10 avalible in your example then does it follow that requests are then divided amongst the machines? Or is this all more about redundancy should a server die, then you still have 5 instances avalible and perhaps even an active request?

Adding and federating a new server should be straight forward, were already setup for it on the directory and config store (yay), and following review of our monitor logs can see what services should on which service etc.

Thanks Again, hope the shows are good!

[MichaelJenkins]

That's right, if two machines are joined as a site then the requests are divided between the two servers.  If you look at the log files you can tell which server is providing the response - it is listed in the machine name column.  But you can't control which server will respond for individual requests or for individual services.

This is sometimes called "Scaling out", as opposed to "Scaling up", which would be just adding more memory to your single machine.

Federating a new server and assigning specific services to it is a valid approach, too, and would be called "workload separation"

[ScottCorwin]

Thank you, I like that term, "scaling out", and it looks to provide what we may need and something I will test.

Follow up question about shared directories for the config store/server directories the documentation says;

"Server performance can suffer when shared locations are used for multi-machine site directories and data, owing to several factors — network bandwidth and stability, opportunistic locking, and network drive performance among them. Use of server directories and data in shared network locations can also negatively affect performance of services under heavy load."

The system has been running with share config-store and server directories  since setup, I inherited IMO a well planned system, are you aware of performance issues and how they may manifest?  I'm not happy with our memory usage and not entirely convinced it's based on the number of services(long long story)

[MichaelJenkins]

I can't speak to what the documentation is warning about, but...

If you want to gain more insight into memory usage by services, open Task Manager and click on the "Details" tab.

You will see dozens of ArcSoc.exe processes listed and their memory usage.  Each one is a "instance" of a map service.   Task manager has some hidden columns with good info.  Right click on a column name and you should see "Select columns", then find and select "Command line"

Now you will have a column that tells you exactly which map service each ArcSOC is related to.  Look for "Dservice=mapservice name.MapServer"

With the number of services you are working with, I would go for 64GB off RAM on that machine.  32GB would likley not be enough.   Curious what JCarlson thinks about that.

[jcarlson]

Posted up above, but we run all our stuff on a machine with 16 GB RAM. I'd consider 32 in a few years, maybe, but I can't see ever needing more than that. But as I said, we do shared instances whenever possible. We're running over 250 services, too!

Great tip about the command line column, that's awesome!

[ScottCorwin]

You can also export the information from task manager, I've used this before, although I think I could have done much the same in Monitor. 

Caveat, I found this information here: https://superuser.com/questions/415360/how-do-i-find-out-command-line-arguments-of-a-running-program ran it as is without much research, it worked, but always risk when you do such. 🙂

WMIC /OUTPUT:C:\Process.txt path win32_process get Caption,Processid,Commandline

[BillFox]

if you have the resources, leave what you have as-is while you build up you new HA site from scratch

that will let you go through all the steps without potentially breaking anything or everything

jump in here: https://enterprise.arcgis.com/en/portal/11.1/administer/windows/high-availability-in-arcgis-enterpri...

[ScottCorwin]

Thank you Bill, had not really looked that way yet.  We are in the process of creating a dev site/clone of our production,. we could certainly test this idea here to.