Latest Contributions by BurtMcAlpine

You will need one spatial license per server, so as long as you are on the same piece of hardware you only need one license.  Arcpy should be 64Bit on server, but i am not 100% sure, but the script would be written the same for 32Bit.  (the script doesn't care 32 vs 64).  Make sure to watch the memory usage trying to process that much data an all available cores.  I would try it on one core and see the peak memory usage and then see how many processes you can safely start at one time without running out of memory and crashing. Are you going to use python multiprocessing module with Pool or some other module? ... View more

Stacy, Thanks so much.  of course I feel like a moron now.  I had the same issue with tht PP but it threw and error when it was not a tuple.  I will try that out and change my code to use mp, as it seems more arcpy friendly. Chris, You can make python 32-bit have 4gb of memory on a 64-bit OS.  You need to change the flag in the python.exe.  I found a tool to change it for me the link is below.  I found it in another forum and I have not tested it so use it with caution... http://www.techpowerup.com/forums/attachment.php?attachmentid=34392&d=1269231650. FYI ARCGIS is set to Largeaddressaware by default ... View more

1) each worker in the pool starts its own python.exe process and you can see them in the Taskmanager 2)I am not using largeaddressware.  Can you confirm that pyhton 2.6 has that capablility?   If so can you point me in that direction? I do know that ArcGIS 10 uses largeaddressaware by default and that in the next release or 2  (arcGIS 10.1 or 11 SP1 is the estimate) ArcGIS will go to 64-bit GP environment (not the entire application just the Geoprocessing environment) I was told that at the ESRI UC 2011 by an ESRI employee. ... View more

Stacy correct the mp code you modified runs the same as the code I originally posted.  That is the problem I was having originally it runs, but only using one child (one process)  so really non-parallel.  It makes the pool, but only uses one worker to process all of the jobs.  I just don't know what is wrong.  is there some setting to use mp that I am missing.  I would like to use mp as it seem to be "better" supported as in I can use arcpy.Adderror and arcpy.AddMessage.  If you make an mxd with a hand full of Shapefiles (has to be shapes for this example) and save it, then point mxd in the code to that mxd you can test it.  Also will need to adjust the workspace as am I using "D:\Temp". Does it run in parallel in your pc. In case it matters... I am running win7 64-bit, Intel processor, and 12GB of RAM.  ArcGIS 10 SP2 with Python 2.6 (as it comes with ArcGIS) most of the patches after SP2 are installed as well. ... View more

Chris in my PP example I use "In_memory", which uses part of your ram to hold data, to speed up the wrker functions which runs in parallel. Each "job" makes a unique "In_memory" object so make sure to delete it before the wrker function ends.  As for program memory leaks, I assume they are all still there and would be a much better issue if the code was run as single process, might be able to do a two part process to clean the memory out 1/2 way through if you run into memory issues, but I have not even come close on my stuff. The pp code (only one I got to work) has to be run out of process,  so "prc" in my example sets the number of processes to start (5 for my intel 980X).  Each process is a python process (32bit) with its own memory allocation, just like using spawn or a sub process.  Then all of the jobs are fed to pool of the 5 processes and those same 5 processes run through the jobs until all of the jobs are finished.  In my testing I have not had any memory issues.  If you use "In_memory" make sure not to load really big objects ( like a 1GB feature clas) in to that, also there are several limitations as to what can and cannot use "In_memory", but I use them all the time and they can really speed up the processing, some time by a factor of 4 times faster. The mp.pool "spawns" new processes as well, but I never could get it to work correctly always wanted to run everything a single process.  It would start the pool, but only feed one process.  The PP is free and seems to work well, so that is what I will be using. Please look at http://pythongisandstuff.wordpress.c.../stacyrendall/  for some known issues with mp and PP  it is towards the bottom of the page. ... View more

Stacy, Thanks for the help I worked on this for 2 days off and on, as I can benefit from several things by having a process or every layer in an mxd.    I gave up one mp and used the pp python. I was using your project as an example http://pythongisandstuff.wordpress.com/author/stacyrendall/.   I also had an install issue wiht ArcView, which compounded my issues.  The below code, which I just finished, buffers the layers in a saved mxd.  I ran it as a single thread and it took 420 seconds.  I ran using the pp libary and it took 151 seconds from python and 154 as a tool, so a significant improvement.  I never could get the jobs to start using mp, now that I have some understanding on the pp I think I will stick to it. I tested your modification.  I it still runs as a single process...  Thanks for helping your web page was very useful import arcpy import os import pp import time #forces script to run out of process no matter what import win32com.client gp = win32com.client.Dispatch("esriGeoprocessing.GpDispatch.1") def wrker(d,bf):    #pp module everything in here get to run in multiprocesses                     #much faster than a single thread     # DO NOT USE arcpy.AddMessage() OR arcpy.addError()  IN THIS PART OF THE SCRIPT.     #set enviroments for each pp process     arcpy.env.overwriteOutput = True     arcpy.env.workspace  = "D:\TEMP\Test"     arcpy.env.outputCoordinateSystem = "C:\Program Files (x86)\ArcGIS\Desktop10.0\Coordinate Systems\Projected Coordinate Systems\UTM\NAD 1983\NAD 1983 UTM Zone 16N.prj"     # Buffer distance in Feet     distance = str(bf) + " Feet"     #output file will go into the Workspace from above     Output = os.path.basename(d)[:-4] + "_Buffer" + str(bf) + ".shp"     #no .shp at the end of an In_memory feature     Tempfile = "In_memory\\" + Output[:-4]     Jobdone = d + " buffered by " + distance     Jobfail = d + " failed to buffer by " + distance     if arcpy.Exists(d):         arcpy.Buffer_analysis(d, Tempfile, distance)         arcpy.Project_management(Tempfile, Output, "C:\Program Files (x86)\ArcGIS\Desktop10.0\Coordinate Systems\Geographic Coordinate Systems\North America\NAD 1983.prj")         arcpy.Delete_management(Tempfile)         #print "Buffering " + d + " at " + distance     else:         print Jobfail     #deletes In_memory temp file as to not hold ram.     del Tempfile     #data retruned to main code and used in messaging there     return Jobdone if __name__ == '__main__':     #starts clock     clck_st = time.clock()     #Sets MXD can not use current for pp processing     mxd = arcpy.mapping.MapDocument("D:\TEMP\Untitled4.mxd")     #Gets all layer in the Current MXD     lyrs = arcpy.mapping.ListLayers(mxd)     #PP prep     # of processors to use set for max minus 1     prc = int(os.environ["NUMBER_OF_PROCESSORS"]) - 1     #using Parellel python     ppservers=()     #sets workers to prc     job_server = pp.Server(prc, ppservers=ppservers)     #sets workers to the max available     #job_server = pp.Server(ppservers=ppservers)     print "Processing all Layers in the mxd"     arcpy.AddMessage("Processing all Layers in the mxd")     #list to hold all of the jobs     jobs = []     #List of buffer values     bfs = [101, 200, 201, 400, 401, 750, 751, 1001,            1500, 1501, 2000, 2001, 2500]     #Loops through every layer and gets source data name and path     for lyr in lyrs:         d = lyr.dataSource         arcpy.AddMessage("Passing " + d + " to processing pool")         for bf in bfs:             jobs.append(job_server.submit(wrker,(d,bf), (), ("arcpy","os","time")))             #wrker(d)             msg = "Passing " + d + " buffer at " + str(bf) + " Feet to processing pool"             print msg             arcpy.AddMessage(msg)     # Retrieve results of all submited jobs     for job in jobs:         print job()         arcpy.AddMessage(job())     #script is finished     print "Processing completed in " + str(int(time.clock() - clck_st)) + " seconds"     arcpy.AddMessage("Processing completed in " + str(int(time.clock() - clck_st)) + " seconds")     time.sleep(5) ... View more