ArcGIS and High Performance Computing (HPC)

4985
1
09-12-2017 06:05 PM
Labels (1)
GeriMiller
Esri Regular Contributor
2 1 4,985

*Updated 2023 

We are continuing to see questions about ArcGIS and high-performance computing (HPC), such as: 

  • I am part of a grant involving HPC – how do I run ArcGIS in HPC? 
  • What specific ArcGIS tools work in HPC environment? 
  • How do I license ArcGIS in a HPC environment? 
  • Where do I go for further questions? 

Conceptual Summary 

Some of the inquiries we’ve received do not distinguish between multicore computers and HPC. This could stem from general fuzziness about what some people mean by HPC, and how the term has evolved in recent years. Most modern desktop computers, including laptops, are “multicore” – meaning they have more than one CPU (central processing unit). 

If there is an environment where multiple computers are connected (and each of these may have multiple cores), then you have a compute cluster or distributed compute environment, which could be run in various infrastructures, commercial cloud being the likely place for such an environment. Traditionally HPC, or a distributed compute environment, is described as multiple computers (hundreds) connected and traditionally application software was specifically written for this compute environment. 

A lot of the tools in ArcGIS Pro can take advantage of multicore hardware and parallel processing. Some tools do this automatically, some honor a geoprocessing environment, that allows one to specify how many processors one wants to use. Note that not all ArcGIS Pro tools run parallel processing and these tools do not run across multiple machines, therefore a user would need a large single machine to do a big job. 

For a full list of the spatial analysis tools that can utilize parallel processing, please refer to this documentation: ArcGIS Pro Parallel Processing  

ArcGIS Pro is desktop software which runs on a Windows platform, but what if you are running a Linux HPC environment? 

Deployment Options 

The vast majority of HPC environments are Linux based, for which a build for ArcGIS Pro does not exist.  

For users who have a Linux HPC, there are 2 options available. 

Option 1: Deploy ArcGIS Enterprise 

ArcGIS Enterprise and the GeoAnalytics Server and Image Server (Raster Analytics) server roles are designed to run in distributed compute environments and can also run across multiples cores of a single machine. These roles are designed to take advantage of distributed storage (not all on one node/disk) and distributed computation (multiple computers). 

With a deployment of ArcGIS Enterprise, users can then deploy any of the additional server roles that suit their analytical needs. 

These roles are made available via ArcGIS Enterprise but can be accessed from a desktop application, such as ArcGIS Pro, or from a web browser (such as the Map Viewer). Running analysis tools utilizing these server roles can be much faster since the analysis tools have been optimized for the distributed computation across multiple nodes. Users utilizing these tools do not need to do anything additional (coding or writing an application) to take advantage of their distributed compute environment. 

For additional guidance, below is documentation related to various configurations for raster analysis and deep learning: 

 Option 2: Deploy ArcGIS Server 

If a user is looking for core geoprocessing tools that are part of ArcGIS Pro (the ArcPy python library), these are found within ArcGIS Server. 

By installing and licensing ArcGIS Server, users can then programmatically run the ArcPy geoprocessing tools via scripting environments. 

Conclusion 

With a Windows HPC environment, users have the option of installing ArcGIS Pro, ArcGIS Enterprise, or ArcGIS Server. In a Linux based cluster, ArcGIS Enterprise or ArcGIS Server provide the best options. Secondly, the GeoAnalytics Server and Image Server (Raster Analytics) server roles utilize distributed computing and can run across multiple machines, or multiple cores of a single machine. A base ArcGIS Enterprise deployment would need to be setup, and these additional server roles enabled. 

Further questions 

As an academic user, please reach out to highered@esri.com with any Education-related questions. 

1 Comment