Python subprocess module and 64 bit background geoprocessing in v10.1

[ChrisSnyder]

I have a script that relys on the 'subprocess' module to basically run seperate instances of arcpy as a parallel process. After installing the 'ArcGIS_BackgroundGP_for_Desktop_101sp1.exe' 64-bit geoprocessing package I have both the 32-bit and 64-bit versions of Python 2.7 on my machine. Both versions of python will import and play with arcpy just fine. However, when I attempt to launch a subprocess of arcpy with a 64 bit version of Python, the subprocess script fails upon attempting to import arcpy with the following error:

C:\Python27\ArcGISx6410.1 Python Traceback Info:   File "\\snarf\am\div_lm\ds\gis\projects\lidar_hydro_all \lidar_hydro_child_v101.py", line 7, in <module>     import arcpy Python Error Info: <type 'exceptions.ImportError'>: DLL load failed: %1 is not a  valid Win32 application.

The subprocess call is to the 64bit version of Python, and I have verified that this is the exe that is launching under subprocess.

The subprocess call I am making is:

subprocess.Popen(['C:\\Python27\\ArcGISx6410.1\\python.exe', '\\\\snarf\\am\\div_lm\\ds\\gis\\projects\\lidar_hydro_all\\lidar_hydro_child_v101.py', 'C:\\csny490\\lidar_stream_build_new', '16'], shell=False)

Funny thing is that the 'lidar_hydro_child_v101.py' script (the import arcpy part) runs just fine stand alone in 64-bit and/or 32-bit Python, but just not when called as a subprocess.

My feeling is that for 'some reason' even though I am running the script from 64-bit python and launching the subprocesses as 64-bit python, when arcpy is being imported in the 64-bit python.exe subprocess, arcpy thinks that it is in 32-bit land still...

Trying to fix things, I added a PYTHONPATH environment variable (which curiously didn't previously exist) that points to 'C:\Python27\ArcGISx6410.1', but that didn't do the trick...

I haven't uninstalled the 32-bit version of Python, but that's my next step...

Any other ideas?

[Luke_Pinner · Accepted solution]

Works for me in 64 bit python (using the following standalone script) and I have both 32 & 64 bit python installed.

import sys,subprocess  
if len(sys.argv)>1:
     import arcpy
     print arcpy
     print sys.executable
else:
     cmd=['python',__file__,'foo']
     proc=subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
     exit_code=proc.wait()
     stdout,stderr=proc.communicate()
     print '\n'.join((str(exit_code),stdout,stderr))‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

This prints:

0
<module 'arcpy' from 'c:\Program Files (x86)\ArcGIS\Desktop10.1\arcpy\arcpy\__init__.py'>
C:\Python27\ArcGISx6410.1\python.exe‍‍‍‍‍‍‍‍‍

Before running, I have a batch file to set up some environment variables:

@echo off 
set PYTHONHOME=C:\Python27\ArcGISx6410.1 
set PATH= 
set PATH=%WINDIR%;%WINDIR%\system32 
set PATH=%PYTHONHOME%;%PATH%
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

A couple of other things to try...

1 - make sure you're not calling subprocess.Popen with shell=True.
2 - try passing a custom "env" to subprocess.Popen with the above env vars, i.e (completely untested...):

env=os.environ.copy()
env['PATH']='%s;%s;%s' %
    ('C:\\Python27\\ArcGISx6410.1',
     env['WINDIR'],
     env['WINDIR']+'\\system32')
proc=subprocess.Popen(cmd, 
                      stdout=subprocess.PIPE, 
                      stderr=subprocess.PIPE,env=env)‍‍‍‍‍‍‍‍

(curtvprice did some editing to make the code visible again)

[ChrisSnyder]

Hi Luke,

Your code:

env=os.environ.copy()
env['PATH']='%s;%s;%s' %('C:\\Python27\\ArcGISx6410.1',env['WINDIR'],env['WINDIR']+'\\system32')
proc=subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE,env=env)

worked great.... I specifically traced the problem to a lack of PATH = 'C:\\Python27\\ArcGISx6410.1'.

I went ahead and appended  'C:\Python27\ArcGISx6410.1' to my system envr PATH variable permanently, and so now no need to pass a custom env definition to the subprocess.

Not sure why just changing PYTHONPATH wasn't sufficient, but I guess there must be some envr variable hierarchy in there.

Anyway, my arcpy subprocess scripts are grooving again - this time in 64 bit arcpy land.

Thanks a million!

[StacyRendall1]

Hi Chris, I am pretty sure that PYTHONPATH is a path that Python checks on load to find modules - not the path to Python. Which explains why it didn't fix your problem.

I used it at one stage to point to a folder where I had placed some modules I had written that were used by a number of different scripts located in different folders, rather than copying them each time I used them and having version mismatches, etc.

If you are on a system where you cannot modify the path I find the following command useful (and can easily be incorporated into a batch script):
set Path=C:\Apps\Python27\ArcGISx6410.1;%Path%

This is run from the Windows command prompt prior to loading Python, and will add your Python path (in this case the path to 64bit Python) to the start of the system path for the duration of the command prompt session.

[ChrisSnyder]

PYTHONPATH is a path that Python checks on load to find modules

Yep - you are correct...

set Path=C:\Apps\Python27\ArcGISx6410.1;%Path%

...gets an upvote.