Creating multipart polygons with ArcPy? Dissolve Failure?

5914
14
Jump to solution
08-24-2015 01:56 PM
ChadCordes
New Contributor II

I need the functionality of the editor > merge command for use in a python script tool. I realize this function is not directly accessible through ArcPy, but I am hoping someone can help with a work around?

The Issue: I have a series of single part polygon features which do not touch or overlap, but have common field values. I need all these features to exist as multipart features for the random point generation to operate properly and consider all these regions equally. However, the dissolve function inconsistently fails leaving single part polygons.

Essentially, I am trying to go from single part geometries to multipart by creating feature layer selections and dissolving on that feature layer selection.

I have included a code snippet below. The code works and does exactly as required, but it is inconsistent. The dissolve fails when preceded by other scripts or when significant edits with debugging have been made to other parts of this script. However, when called from a new .mxd document it runs consistently well.

Any advice and support would be gratefully accepted!
Thanks!

#Custom exceptions.
class LicenseException(Exception):
    pass


#Import system modules.
import arcpy
import sys
import os
import traceback
from arcpy import env
from arcpy.sa import *


#Enable geoprocessor logging.
arcpy.SetLogHistory(True)


try:
    #---Check the installed ArcGIS version.---
    iv = arcpy.gp.GetInstallInfo()


    version = " ArcGIS %s : %s" % ('Version', iv.get('Version','Version Unknown'))


    if version.find("10.") > 0:
        ArcGIS10 = True
    else:
        ArcGIS10 = False
        arcpy.AddError("ERROR: This tool requires ArcGIS version 10.1 or greater...exiting script.")
        sys.exit("")


    del iv, version, ArcGIS10


    #---Check the ArcGIS administrative license.---
    if arcpy.CheckProduct("ArcInfo") == "AlreadyInitialized" or arcpy.CheckProduct("ArcInfo") == "Available":  
        pass
    else:
        raise LicenseException()


    #---Check out extension licenses.---
    if arcpy.gp.CheckExtension("spatial") == "Available":
        arcpy.CheckOutExtension("spatial")
    else:
        arcpy.AddError("ERROR: Spatial Analyst extension is unavailable. Please enable Spatial analyst and try again...exiting script.")
        sys.exit("")


    #---Get tool share folder file path.---
    mxd = arcpy.mapping.MapDocument("Current")
    mxdpath = mxd.filePath
    (filepath, filename) = os.path.split(mxdpath)
    MCP_Path = str(filepath)


    #---Set the geoprocessing environment.---
    workspace = MCP_Path + "\OutputData\ToolOut.gdb"
    scratchspace = MCP_Path + "\OutputData\ScratchFolder\Scratch.gdb"


    env.scratchWorkspace = scratchspace
    env.workspace = workspace
    env.overwriteOutput = True


    #---Read user input.---
    BI = arcpy.GetParameterAsText(0)


    BIdesc = arcpy.Describe(BI) 


    #---Execute Reclass.---
    DS_RC_filename = BIdesc.name + "_DS_RC"
    DS_RC_out = os.path.join(workspace, DS_RC_filename)


    DS_Reclass = Reclassify(BI, "Value", 
    RemapRange([[-1000,-75,8],[-75,-50,7],[-50,-35,6],[-35,-20,5],[-20,-12,4],[-12,-6,3],[-6,-3,2],[-3,-1,1],[-1,0,"NODATA"]]), "NODATA")
    DS_Reclass.save(DS_RC_out)


    #---Execute raster to polygon conversion.---
    DS_Polygon_filename = BIdesc.name + "_DS_Polygon"
    DS_Polygon = os.path.join(workspace, DS_Polygon_filename)


    arcpy.RasterToPolygon_conversion(DS_RC_out, DS_Polygon, "SIMPLIFY", "VALUE")


    if arcpy.Exists(DS_RC_out):
        arcpy.Delete_management(DS_RC_out)
    del DS_RC_filename, DS_Polygon_filename 


    #---Select polygons of areas >= 1000m2.---
    DS_PS_out_filename = BIdesc.name + "_DS_PSelect"
    DS_PS_out = os.path.join(workspace, DS_PS_out_filename)


    sql = '{0} >= {1}'.format("Shape_Area",'1000')
    if arcpy.Exists("DS_PS"):
        arcpy.Delete_management("DS_PS")
    DS_PS = arcpy.management.MakeFeatureLayer(DS_Polygon,"DS_PS",sql)
    arcpy.CopyFeatures_management("DS_PS", DS_PS_out) #Done to hopefully stabilize the following selection


    arcpy.Delete_management(DS_PS)
    del DS_PS_out_filename, sql


    #---------------------Perform Dissolve----------------------------------    
    DS_F_out_filename = BIdesc.name + "_DS"
    DS_F_out = os.path.join(workspace, DS_F_out_filename)


    arcpy.Dissolve_management(DS_PS_out,DS_F_out,"grid_code","#","MULTI_PART","DISSOLVE_LINES") 


    del BsMmaxS, sql


    #---Check to ensure dissolve function worked properly.---
    sql = '{0} = {1}'.format("grid_code",1)
    arcpy.MakeTableView_management(DS_F_out, "DS_F_out_TV",sql)
    count = int(arcpy.GetCount_management("DS_F_out_TV").getOutput(0))
    if count > 1:
        del sql
        if arcpy.Exists(DS_F_out):
            arcpy.Delete_management(DS_F_out)
        arcpy.Delete_management("DS_F_out_TV")
        arcpy.AddError("Geoprocessing Error due to memory limitations with 32bit processing. Assuming 64bit processing is unavaiable, please close your ArcMap instance, open it again and reprocess...exiting script.")
        #Above error prompt is my current best guess since the polygons can be quite complex and can contain many verticies. 
        sys.exit("")
    else:
        arcpy.Delete_management("DS_F_out_TV")
        del sql


    #Continue code here..
Tags (3)
0 Kudos
14 Replies
JoshuaBixby
MVP Esteemed Contributor

If you run the code two or three times in a row using the same dataset, does it work each time or only the first time?

0 Kudos
ChadCordes
New Contributor II

It runs flawlessly each time.
In fact, it runs flawlessly no matter what raster dataset is input at the top of the script.

The problem only arises when I run the 2 script tools which precede this one. They're used to create the raster which is input in this script.

However, they also run flawlessly, are very stable, and seem to be bug free.

Once the tool errors, it will continue to error unless I close the .mxd and reopen it. Then it will run smoothly again.


The only other time the dissolve fails is when I make edits/debug this tool.

Once the logic is correct. The dissolve will continue to fail until I restart the .mxd.  

0 Kudos
curtvprice
MVP Esteemed Contributor

I am seeing a similar deal with Dissolve run at the end of a large script. The dissolve works fine from ArcMap and run from a clean Python session with similar environment. Here is my error message:

21:18:53 43.63 Dissolve
ERROR: File "Q:\tools\nhr_raster\HRStepL.py", line 1160, in RasterProc
    "OBJECTID COUNT", "MULTI_PART")
Executing: Dissolve lyrCat C:\Working\StepL_0604_20151123\work.gdb\vpucat_poly1 GRIDCODE "OBJECTID COUNT" MULTI_PART DISSOLVE_LINES
Start Time: Mon Nov 23 21:18:56 2015
Sorting Attributes...
Dissolving...
ERROR 999999: Error executing function.
ERROR: Invalid Topology [Cannot open cache file.]
ERROR: Failed to execute (Dissolve).
Failed at Tue Nov 24 01:23:57 2015 (Elapsed Time: 4 hours 5 minutes 1 seconds)

Here's what happens when run from a straight Python command line:

Executing: Dissolve C:\Working\StepL_0604_20151123\work.gdb\vpucat_poly0 C:\Working\StepL_0604_20151123\work.gdb\vpucat_poly0a Grid
ode "objectid count" MULTI_PART DISSOLVE_LINES
Start Time: Tue Nov 24 10:27:58 2015
Sorting Attributes...
Dissolving...
Succeeded at Tue Nov 24 10:30:09 2015 (Elapsed Time: 2 minutes 11 seconds)

I have set ARCTMPDIR to a writeable folder (Esri KB 25995) and have even tried running half the polygons and get the same behavior. Given how things are hanging before they crash i suspect there is a resource issue here (ie lack of RAM) and that things eventually time out.

I don't get the problem (I don't think) when running 64 bit, which is even more evidence that this is a resources issue. (Along with your experience that restarting the mxd lets it work - restarting the mxd would clear layers and other cruft from memory.)

I am going to go back and try deleting some layers from memory in the hope that will solve my problem. My scripts clean up memory in a finally block but I think there is enough stuff in this script that I need to do some pre-cleanup!

I'm also going to try to add some garbage collection to hopefully release some memory that Python may be holding onto. Garbage collection happens automatically but a complicated script may benefit from a gc.collect() before the max possible memory may be needed.

import gc
gc.collect()

UPDATE: I've started an incident with Esri support and they pretty much told me that Dissolve is not stable with really large inputs (my input that was failing is a Raster To Polygon output of Watershed() output with about 70,000 polygons). The workaround is to tile the data and dissolve smaller datasets, or run in x64 python (for example, x64 background processing). (Not an option for me, unfortunately.)

DanielThomas1
New Contributor III

Curtis,

Recently, I've ran into the same problem in try to handle multiple overlapping buffers and dissolving into a multipart feature.  The only solution I found to work was to depart from arcpy and use these python libraries noted on this page: GIS with Python, Shapely, and Fiona - macwright.org.

After making the modifications for my Windows based python, I was able to use the union example to create the feature I needed.

ChadCordes
New Contributor II

Hi Curtis,

This essentially turned out to be the culprit. The ESRI support people took almost a month working with me to understand then sort out the issue. Their solution was always to just use a 64bit processing environment; or as a work around, restart the map document as I was doing.

Apparently there was some mysterious behind the scenes virtual memory cache (they never really explained this too well) which was becoming overloaded. The script tool which I was running before the dissolve created a DEM. This was loading up the cache. Then when I would run the script tool which called the dissolve geoprocess (which itself has a very high virtual memory requirement) overloaded the cache. This doesn't make a whole ton of sense because from what I understand all virtual memory caches are cleared when a tool is closed. Anyway, it really only poses an issue when working within a 32bit OS.

Little late now, but just thought I'd follow up. Thanks for your effort!
Cheers