How to setup BATCH process for my Script.

4330
29
Jump to solution
01-11-2013 10:07 AM
ChristopherClark1
Occasional Contributor
Ok, I have the following python script which migrates and maps fields and the values of those fields into a target Feature Class.
# Import arcpy module import arcpy  arcpy.env.workspace = "G:\ChrisGIS\WDFW_NWIFC_EVENT_17100101.gdb" arcpy.env.overwriteOutput = "True"                def GetFieldMappings(inputFC, targetFC, dico):     field_mappings = arcpy.FieldMappings()     field_mappings.addTable(inputFC)      for input, output in dico.iteritems():         field_map = arcpy.FieldMap()         field_map.addInputField(inputFC, input)                 field = field_map.outputField                 field.name = output         field_map.outputField = field         field_mappings.addFieldMap(field_map)         del field, field_map      return field_mappings  #Variables inputFC = "WDFW_CHFA_2HEM_evt_1" targetFC = "WDFW_CHFA_2HEM_evt_olay_1"  dico = {'Permanent_Identifier': 'Permanent_ID',                 'STRM_NAME':        'LLID_STRM_NAME',     'DATE_SRVY': 'SOURCE_DATE',     'SPPCODE': 'SPECRCODE',     'SRC_AGENCY': 'SOURCE_AGENCY',     'SRC_WHO': 'SOURCE_WHO'}      Mapper = GetFieldMappings(inputFC, targetFC, dico) # Process: Append arcpy.Append_management(arcpy.env.workspace+"\\"+inputFC, arcpy.env.workspace+"\\"+targetFC,"NO_TEST",Mapper)

I have several feature classes to run this script on. AN exmaple of the Feature Classes that I need to run my script on are :

WDFW_CCT_2HEM_evt_olay
WDFW_CHFA_2HEM_evt_olay
WDFW_CHMF_2HEM_evt_olay
WDFW_CHSP_2HEM_evt_olay
WDFW_CHSU_2HEM_evt_olay
WDFW_COHO_2HEM_evt_olay
WDFW_DBT_2HEM_evt_olay
WDFW_KOK_2HEM_evt_olay
WDFW_RBT_2HEM_evt_olay
WDFW_SOCK_2HEM_evt_olay

These feature classes are represented in several geodatabases (see image)
[ATTACH=CONFIG]20670[/ATTACH]
I could simply go to each individual geodatabase and run the script for each featureclass in that geodatabase. This however seems redundant and I would like to figure out how to batch my scrip or automate this process. All geodatabases are located in one working directory.

I appreciate any help or advice!
Tags (2)
0 Kudos
29 Replies
ChristopherClark1
Occasional Contributor
I think I have it correct now, but when I execute the code
# Import arcpy module
import arcpy
arcpy.env.overwriteOutput = True
from os import path as p
        
def GetFieldMappings(inputFC, targetFC, dico):
    field_mappings = arcpy.FieldMappings()
    field_mappings.addTable(inputFC)

    for input, output in dico.iteritems():
        field_map = arcpy.FieldMap()
        field_map.addInputField(inputFC, input)        
        field = field_map.outputField        
        field.name = output
        field_map.outputField = field
        field_mappings.addFieldMap(field_map)
        del field, field_map

    return field_mappings

#Variables
dico = {'Permanent_Identifier': 'Permanent_ID',
                'STRM_NAME':        'LLID_STRM_NAME',
    'DATE_SRVY': 'SOURCE_DATE',
    'SPPCODE': 'SPECRCODE',
    'SRC_AGENCY': 'SOURCE_AGENCY',
    'SRC_WHO': 'SOURCE_WHO'}

arcpy.env.workspace = r"G:\ChrisGIS\COMBO_fishdist_events"
for ws in arcpy.ListWorkspaces("*", "FileGDB"):
 listFCS = arcpy.ListTables("WDFW*evt")
 arcpy.env.workspace = ws
 for inputFC in listFCS:
  outTab = r'in_memory\%s_2' %inputFC
  targetFC = inputFC +"_olay"   
  Mapper = GetFieldMappings(inputFC, targetFC, dico)
  # Process: Append
  arcpy.CreateTable_management(ws, outTab)
  arcpy.Merge_management([p.join(ws,inputFC),outTab],p.join(ws,targetFC),Mapper)
  print 'Appended "{0}" to "{1}" in {2}'.format(inputFC, targetFC, p.basename(ws))


I get this error

Runtime error <class 'arcgisscripting.ExecuteError'>: ERROR 999999: Error executing function. Failed to execute (CreateTable).
0 Kudos
by Anonymous User
Not applicable
I have made some dummy gdb's with your table names and the input fields from the dico.  I am able to get it to go through all the tables and merge, but there is a problem with your GetFieldMappings() function since the field names aren't changing in the targetFCs.  I am trying to figure it out, I have never worked with field mappings objects before. I'll post the full code if I can figure it out.
0 Kudos
ChristopherClark1
Occasional Contributor
Thanks Caleb for the loads of help. If you want, I can send you a zip file with a sample of my data. I wonder why I am getting a create table error and you are not. I will work more with it. The field mapping object is pretty awesome and useful once it is figured out. It was pretty complex for me.

Here are some links that helped me out :

http://forums.arcgis.com/threads/33060-field-mapping-in-python-for-ArcGIS-10

http://blogs.esri.com/esri/arcgis/2012/08/30/field-mapping-and-python-scripting/

send me a message if you want me to email me some of my data.

Thanks again!
0 Kudos
by Anonymous User
Not applicable
Alright, I got it! I went back to the append instead of merge. I created a dummy table in a gdb in a different location than the original workspace with the new field names (the values in the 'dico' dictionary). I then used that as a template for the create table (for targetFC) and appended the inputFCs to the targetFCs.

# Import arcpy module
import arcpy
arcpy.env.overwriteOutput = True
from os import path as p

def GetFieldMappings(fc_in, fc_out, dico):
    field_mappings = arcpy.FieldMappings()
    field_mappings.addTable(fc_in)

    for input, output in dico.iteritems():
        field_map = arcpy.FieldMap()
        field_map.addInputField(fc_in, input)        
        field = field_map.outputField      # Missing underscore  
        field.name = output
        field_map.outputField = field
        field_mappings.addFieldMap(field_map)
        del field, field_map

    return field_mappings

#Variables
dico = {'Permanent_Identifier': 'Permanent_ID',
        'STRM_NAME':'LLID_STRM_NAME',
 'DATE_SRVY': 'SOURCE_DATE',
 'SPPCODE': 'SPECRCODE',
 'SRC_AGENCY': 'SOURCE_AGENCY',
 'SRC_WHO': 'SOURCE_WHO'}

template = r'C:\gdblooptest\template\template.gdb\Template_table'    
##arcpy.env.workspace = r"G:\ChrisGIS\COMBO_fishdist_events"
for ws in arcpy.ListWorkspaces("*", "FileGDB"):
    arcpy.env.workspace = ws
    listFCS = arcpy.ListTables()
    for inputFC in listFCS:
        targetFC = inputFC +"_olay"
        # Process: Append
        arcpy.CreateTable_management(ws, targetFC, template)
        Mapper = GetFieldMappings(inputFC, targetFC, dico)
        arcpy.Append_management(inputFC,targetFC,'NO_TEST',Mapper)
        print 'Appended "{0}" to "{1}" in {2}'.format(inputFC, targetFC, p.basename(ws))


Here is a shot of the template dummy table
[ATTACH=CONFIG]20675[/ATTACH]

Here is the original table ( I just added some random values)
[ATTACH=CONFIG]20676[/ATTACH]

And here is the targetFC table after appended:
[ATTACH=CONFIG]20677[/ATTACH]
0 Kudos
ChristopherClark1
Occasional Contributor
Caleb, You are the Man (or woman?)!! Thanks! I ran it once real quick and got a new error which I know how to correct. One of the fields in my template table only has 50 characters max and some of the values in the original table under that same field are 100 characters, so I think I just have to change that and it will work. But, I have to go to dinner with some folks, bummer, I hate to wait!

But I will correct the field character value when i get home and run this.

Why did we have to have a template table??

Thanks for helping me this entire day. Our thread rules!
0 Kudos
by Anonymous User
Not applicable
Haha, I am a man indeed.  I think I know why you got an error.  We actually do need to include a test to exclude those tables that end in '_olay' because it will keep looping through the ws (as new tables are created) and create tables that end in '_olay_olay' etc.  These tables will cause an error since they will not have the original field names in them when they get passed into the GetFieldMappings().

Also, we needed to have a dummy template just so that the append tool would work properly.  When the field names don't match you have to use NO_TEST, as you already knew, but the fields will not be appended if they do not already exist in the table and  the fields that are matched through the field map would not actually be inserted into the table.  Essentially you would be appending the fields to a table with only an OID field which will not work.

This was a tricky one though.  I had never worked with field mappings before.  Here is the optimal solution:

# Import arcpy module
import arcpy, sys, traceback
arcpy.env.overwriteOutput = True
from os import path as p

def GetFieldMappings(fc_in, fc_out, dico):

    field_mappings = arcpy.FieldMappings()
    field_mappings.addTable(fc_in)

    for input, output in dico.iteritems():
        field_map = arcpy.FieldMap()
        field_map.addInputField(fc_in, input)        
        field = field_map.outputField      
        field.name = output
        field_map.outputField = field
        field_mappings.addFieldMap(field_map)
        del field, field_map

    return field_mappings
  

#Variables
dico = {'Permanent_Identifier': 'Permanent_ID',
        'STRM_NAME':'LLID_STRM_NAME',
 'DATE_SRVY': 'SOURCE_DATE',
 'SPPCODE': 'SPECRCODE',
 'SRC_AGENCY': 'SOURCE_AGENCY',
 'SRC_WHO': 'SOURCE_WHO'}


template = r'C:\gdblooptest\template\template.gdb\Template_table'
arcpy.env.workspace = r"G:\ChrisGIS\COMBO_fishdist_events"

for ws in arcpy.ListWorkspaces("*", "FileGDB"):
    arcpy.env.workspace = ws
    listFCS = arcpy.ListTables()
    for inputFC in listFCS:
        if not inputFC.endswith('_olay'):
            targetFC = inputFC +"_olay"
            # Process: Append
            if arcpy.Exists(targetFC):
                arcpy.Delete_management(targetFC)
                print 'deleted %s' %targetFC
            arcpy.CreateTable_management(ws, targetFC, template)
            Mapper = GetFieldMappings(inputFC, targetFC, dico)
            arcpy.Append_management(inputFC,targetFC,'NO_TEST',Mapper)
            print 'Appended "{0}" to "{1}" in {2}'.format(inputFC, targetFC, p.basename(ws))
print 'done'

0 Kudos
ChristopherClark1
Occasional Contributor
I keep getting a few errors. I was able to resolve one of them but the other two are giving me a hard time. For some reason when I run the script it starts working in the first gdb. After it has successfully created a few "olay" tables it hangs up and gives me the error

Runtime error <type 'exceptions.RuntimeError'>: FieldMappings: Error in adding table to field mappings

For some reason a olay table is created (CHSU). The odd thing is this "CHSU" feature class does not exist in the gdb. Therefore when the script tries to add the original table into the field mappings object it crashes and delivers the above error.

So, I could not solve this issue. I decided to remove that gdb from my workspace (temporarily) and try to run the script on what remained in the workspace. Well that failed as well with the resulting error :

Runtime error <class 'arcgisscripting.ExecuteError'>: ERROR 000278: Failed on input OID 1, could not write value â?? â?? to output field SOURCE_DATE Failed to execute (Append).

Caleb, I could easily zip up 2 or 3 gdb if you wanted to recreate these errors. I am really stumped on the first one. the CHSU feature class does exist is some of the other gdb in my workspace. Is my listFCS = arcpy.ListTables("WDFW*evt") argument making a list for all the tables no matter what gdb they are in? If so this is the error. There are many times where the tables in one gdb are the same as another, but there are some tables (like the CHSU ones) that may only occur in 9 out of 10 gdb or 5 out of 20 etc.. In other words if this argument is the issue do I need to change it so it makes a list of tables for each gdb?
0 Kudos
by Anonymous User
Not applicable
I sent you a private message with my email info.  I'll take a look at it.
0 Kudos
ChristopherClark1
Occasional Contributor
Caleb, thanks for all the hardwork, time, and advice. It finally worked. Completely automated!

Very much appreciated!
0 Kudos
MichalisAvraam
New Contributor III
... 
So, I could not solve this issue. I decided to remove that gdb from my workspace (temporarily) and try to run the script on what remained in the workspace. Well that failed as well with the resulting error : 

Runtime error <class 'arcgisscripting.ExecuteError'>: ERROR 000278: Failed on input OID 1, could not write value â?? â?? to output field SOURCE_DATE Failed to execute (Append).


Did you ever figure out this error? I keep getting it too on a similar process running the Merge tool.
0 Kudos