ArcGIS Pro 2.9 - Route Solver Multiprocessing in Standalone Script

[dgraham_cra]

I am using ArcGIS Pro 2.9 with an advanced single use license and a locally saved network dataset.

I am attempting to write a standalone script that takes as its input an arbitrary set of origin-destination pairs and produces as its output the travel distance (e.g., driving distance in miles) and time (e.g., driving distance in minutes) for each pair. The script creates and subsequently solves a route analysis layer using the route solver method.

The core logic of the script is as follows:

(1) load the set of origin-destination pairs from an input csv file as a point feature class

arcpy.management.XYTableToPoint(
 r"[input csv file]"
,'[output (origins|destinations) layer name]'
,'[input (origins|destinations) longitude field]', '[input (origins|destinations) latitude field])'

(2) create a new Route_ID field that concatenates the origin ID and destination ID for each pair with a pipe delimiter

arcpy.management.CalculateField(
 '[(origins|destinations) layer name]'
,'Route_ID'
,field_type = 'TEXT'
,expression = '[concatenation expression]'
,expression_type = 'ARCADE')

(3) define a function that loads the origins and destinations for each chunk and then solves for the routes

odp_FieldMappings = route.fieldMappings(
arcpy.nax.RouteInputDataType.Stops
)

odp_FieldMappings['RouteName'].mappedFieldName = 'Route_ID'

def solve_chunk(chunk):

# Define chunk start and end
chunk_index = chunk - 1

odp_chunk_start = (chunk_index * ChunkSize) + 1
odp_chunk_end = odp_chunk_start + (ChunkSize - 1)

# Load chunk origins
arcpy.conversion.FeatureClassToFeatureClass(
 '[origins layer name]'
,gdb
,'[origins chunk layer name]'
,f'OBJECTID >= {odp_chunk_start} \
     and \
   OBJECTID <= {odp_chunk_end}')

route.load(
 arcpy.nax.RouteInputDataType.Stops
,'[origins chunk layer name]'
,odp_FieldMappings
,append=False)

# Load chunk destinations
arcpy.conversion.FeatureClassToFeatureClass(
 '[destinations layer name]'
,gdb
,'[destinations chunk layer name]'
,f'OBJECTID >= {odp_chunk_start} \
     and \
   OBJECTID <= {odp_chunk_end}')

route.load(
 arcpy.nax.RouteInputDataType.Stops
,'[destinations chunk layer name]'
,odp_FieldMappings
,append=True)

# Solve
return (chunk, route.solve())

(4) feed this function into a multiprocessing pool to generate the solve results

if __name__ == '__main__':
    with mp.Pool() as pool:
        results = pool.map_async(solve_chunk, chunks)

However, frustratingly, I just can't seem to get the multiprocessing to work. When I attempt to call the solve_chunk function within map_async(), I get a 000732 error when loading the chunk origins:

ERROR 000732: Input Features: Dataset [origins layer name] does not exist or is not supported

And sure enough, when I check my project's geodatabase, I don't see the expected feature classes.

I've tried multiple ways without success to fix this issue, including using different arcpy functions (e.g., arcpy.management.MakeFeatureLayer). But nothing seems to work. I've used ArcGIS for years, but running scripts in a standalone environment rather than via the Python window is new to me, as is multiprocessing.

Can anyone help?

P.S., here are my environment settings:

ArcGIS_ProjFolder = aprx.homeFolder
m = aprx.listMaps('Map')[0]
gdb = aprx.defaultGeodatabas
arcpy.env.workspace = gdb
arcpy.env.overwriteOutput = True

[Anonymous User]

It would help if you formatted your code as python by clicking the ..., then </> symbol and pasting it into the window.

File Geodatabases do not support concurrent write operations so if your process is trying to write to it in each of its processes, it will fail. You will need to output the results to somewhere else, like a new gdb for each result, and then consolidate from that location.

[dgraham_cra]

Done! And that's helpful advice regarding file geodatabases. Is there documentation I can refer to regarding how to implement similar logic writing to and reading from in-memory locations rather than my geodatabase? I'll poke around myself as well, but any pointers would be appreciated.

[Anonymous User]

Multiprocessing in python is honestly a nightmare, especially with its inability to return objects that are not pickle-able, which is pretty much everything outside of the built-in classes and objects (lists, dictionaries, etc.,.) You can (and would have to) create or wrap your own Featureclass Class, to make it pickle-able.  I haven't gone down that far yet because it just seems ridiculous. As is the many different ways the multiprocessing can be set up, each with its own quirks.

I honestly haven't tried the in memory space yet. You can try it though, and return a path to the item in memory but it may be locked by the thread or destroyed when the script finishes executing.  I think your best bet would be creating new gdb's for each thread and returning the path to them for further reference later in your script.  Seems counter productive, but that is how arcpy/ python is built.

I return result dictionaries with paths to the features that were created, or any other information that I need from it and iterate over them.

In the worker script, return a dictionary:

...
result = {'Task': 'c_quality', 'count': 0, 'Error': None}
result['count'] = arcpy.GetCount_management(cTble).getOutput(0)

return result

In the main script, execute the workers getting their results in a list to iterate over.

 with mp.Pool(processes=3) as pool:
            p1 = pool.apply_async(CreateSales.create_sales, (logging, ))
            p2 = pool.apply_async(CreateQuality.create_quality, (logging, ))
            p3 = pool.apply_async(CreateAccountsTable.create_accounts_table, (logging, ))

            res = [p1.get(), p2.get(), p3.get()]

        for r in res:
            if r['Error'] != None:
                logging.info(
                    f'Task {r["Task"]} Failed -- Check Error Log {dt.timedelta(seconds=t.time() - tasksStartTime)}\n')

[MelindaMorang]

Hi.  I just came across this thread.  If you haven't already found a solution, we actually have a downloadable tool that multiprocesses route calculations between preassigned OD pairs in ArcGIS Pro.  You can find it here: https://github.com/Esri/large-network-analysis-tools  You can use these tools out of the box or modify them to suit your needs.  There's also some materials in that repo from DevSummit presentations on the subject.  I hope this helps!

[MelindaMorang]

While I'm here...

Esri’s Network Analyst Team is doing some research about customer workflows involving solving routes or calculating travel times and distances between known or preassigned pairs of origins and destinations.  It sounds like that's what you're doing here!  We’d like to better understand our customers’ needs in this area and may use this information to design and develop improved tools and workflows.

If you have a few minutes and are willing, could you please fill out the attached survey and return it to the e-mail address included inside the document?  Alternatively, you can reach out to me (my e-mail address is in the document) to set up a meeting.

Thank you so much!!

[MelindaMorang]

Oh wait, it's you, dgraham!  I've already talked to you in my GitHub repo.  Hi again!  Sorry for the duplicate info.

Hopefully the info here will be of use to someone else in the future who's having the same problem.

[RavenSchmidt]

Hi Melinda, these tools on GitHup only seem to do OD cost matrix analyses. Is there a way to get the routes out of the tool as well?

[MelindaMorang]

The Solve Large Analysis With Known OD Pairs tool in the GitHub repo generates routes between origins and preassigned destinations.  It uses the Route solver behind the scenes, not the OD Cost Matrix solver.  If you want routes between preassigned OD pairs, you can use this tool.

If you want some other kind of Routes, like routes with more that two stops or routes where you don't have preassigned OD pairs, then you're right: these tools don't currently have anything for your use case.

Can you explain more about what you're trying to do and the size of your problem?