geri_miller-esristaff

ArcGIS and High Performance Computing (HPC)

Blog Post created by geri_miller-esristaff Employee on Sep 12, 2017

Recently we have seen quite a few questions about ArcGIS and high performance computing (HPC), such as:

 

  •      I am part of a grant involving HPC – how do I run ArcGIS in HPC?
  •      What specific ArcGIS tools work in HPC environment?
  •      How do I license ArcGIS in a HPC environment?
  •      Where do I go for further questions?

 

Conceptual summary:

 

Some of the inquiries we’ve received do not distinguish between multicore computers and HPC. This could stem from general fuzziness about what some people mean by HPC, and how the term has evolved in recent years. Most modern desktop computers, including laptops, are “multicore” – meaning they have more than one CPU (central processing unit).

 

A lot of the tools in ArcGIS Desktop can take advantage of multicore hardware and parallel processing. Some tools do this automatically, some honor a geoprocessing environment, that allows one to specify how many processors one wants to use. Note that not all ArcGIS Desktop tools run parallel processing and these desktop tools do not run across multiple machines, therefore a user would need a beefy single machine to do a big job.

 

If there is an environment where multiple computers are connected together (and each of these may have multiple cores), then you have a compute cluster or distributed compute environment, which could be run in various infrastructures, commercial cloud being the likely place for such environment. Traditionally HPC, or a distributed compute environment, is described as multiple computers (hundreds) connected together and traditionally application software was specifically written for this compute environment. For a long time ArcGIS users have been asking about running ArcGIS on HPC computers, and now that is possible without additional coding and application writing.  

 

This is where ArcGIS Enterprise and specifically the GeoAnalytics Server and Image Server (Raster Analytics) server roles become useful because they are designed to run in such distributed compute environment, as well as run across multiples cores of a single machine. These roles are designed to take advantage of distributed storage (not all on one node/disk) and distributed computation (multiple computers).   

 

These roles are made available via ArcGIS Enterprise but can be accessed from a desktop application, such as ArcGIS Pro, or from a web browser (such as the Map Viewer). Running analysis tools utilizing these server roles can be much faster since the analysis tools have been optimized for the distributed computation across multiple nodes. Users utilizing these tools do not need to do anything additional (coding or writing an application) to take advantage of their distributed compute environment, the tools just know how to do it.

 

Bottom line: GeoAnalytics and Raster Analytics tools utilize distributed computing and can run across multiple machines, or multiple cores of a single machine. A base ArcGIS Enterprise deployment would need to be setup, and these additional server roles enabled.

 

Note: We also have heard some questions about ArcGIS Server clusters and associated confusion in HPC context. ArcGIS Server clusters are strongly discouraged and have been deprecated at 10.5.1. ArcGIS Server clusters were just a way to separate services, to run under a dedicated resource, i.e. “cluster”, which is different than the clustering we are talking about in HPC context.

 

Further questions:

 

As an academic user, please reach out to highered@esri.com  with any Education-related questions.

Outcomes