Model Builder not Appending Data Correctly ?!

6847
18
02-10-2015 12:55 AM
timdunlevie
Occasional Contributor

Hi guys,

 

I'm using ArcGIS 10.2...and I'm getting some frustrating results using Model Builder to append data to a Feature Class within a FGDB.

 

I have 6 datasets (a mix of shapefiles and FC in a FGDB)

The steps I perform in the model include:

1. add fields to these datasets

2. calculate fields

3. append to a "master" FC in a FGDB using schema type "NO_TEST" and field map the various fields from the input datasets to the Target.

 

for some reason the data is not appending correctly. The field mapping looks correct.

I've deleted features from the Target FC (empty FC) and tried appending over again, yet the same (wrong) fields map to the target fc.

is it holding this data in memory ???

Is MB always so buggy ?

 

many thanks,

Tim

Tags (1)
0 Kudos
18 Replies
JoeBryant2
Occasional Contributor II

Sure: The model I am running is joining customer account information tables the I have exported from our billing database to Parcel Points we get from the County, based on APN. I am using these points as "placeholders" for water meters (wServiceConnections) that will have their locations updated later using GPS. Most of our meters have already been located and created; these meters are the leftovers we have not been able to collect yet that need to be represented in the feature class for accurate hydraulic modelling. 

We recently migrated to the Local Government Information Model, so the field names in our billing system don't exactly match the fields names for wServiceConnections in the LGIM, except for the extra fields from the billing database I have added to the schema. But, because these fields are the result of a JOIN, their names don't exactly match (outputs from my Joins create the field name with the source table name as a prefix). So the field mapping feature of the Append Tool could save me a lot of time as I go through each meter zone in our District and create the meters that have not been collected yet.

I had thought about the "Poor Man's" way; it will significantly increase the size of the model, which is already kind of large. If I have to, that's the way I will go.

PS: I got your field map parameter suggestion in a separate post - I will try that first. Thanks!

"failure-using-model-builder-iteration-to-append-many-shapefiles"

0 Kudos
curtvprice
MVP Esteemed Contributor

From what I'm reading from your description, it sounds like you will need to create a field map in a python function to do what you want. But do that you will need logic to create a map of old names to new name, for example will prefix_table_name_SERVED always map to NUMBER_SERVED no matter what the prefix table name?

0 Kudos
JoeBryant2
Occasional Contributor II

Yes. For example: FeatureClassName_JoinedTableName_Full_Addr always field maps to ADDRESS in wServiceConnections. As I run the model on different zones in our district, the FeatureClassName changes (I create a selected subset of Parcel Points in a local file geodatabase), and JoinedTableName changes as I export separate tables of customer meter information from the billing system for each zone. But "..._Full_Addr" is always the name at the end of the field that will map to ADDRESS.

I am attempting your Field Map parameter suggestion, but it is currently adding a lot of manual work to each run of the model, particularly because there are a lot of fields and I can't see the full field name in the migration mapping dialog - I have to expand the window and scroll through the list for each selection.

The Poor Man's approach is looking more attractive - rename all the fields once in the model and Append using TEST. I had just hoped that Model Builder was up to the task of what seems like a very common workflow: join a table to a collections of points, select out features that have already been created, and load the new features into the master feature class, while create tables for the unmatched rows and points left over.

0 Kudos
curtvprice
MVP Esteemed Contributor

Here's a shot at it. No guarantees, but this is a way to use python to create field maps on the fly. You'd use the Calculate Value expression: make_maps(r"%Input features%"), paste the code below into the code block parameter, and set the output data type to Field Map so you could connect it as a parameter to the Append tool.  You'd also have to hard code your from-name and to to-names in the code block into the dictionary I called renames

Your mileage may vary, but this an approach to consider. You may be stuck with the "poor man method", but that would sure be a lot extra work.

def FieldNameMap(tbl, maps):
    """Easy field mappings generator

    arguments

      tbl - input feature class, table, or table view
      maps - field names map list (';'-delimited string of field name pairs)

    example

      import arcpy
      Maps = "Shape_Area AREA;BID BID2;AREASQMI AREAMI2"
      Mapper = FieldNameMap("temp.dbf", Maps)
      # to debug:
      print Mapper.exportToString().replace(";","\n")
      # Output will only have fields: AREA, BID2, AREAMI2
      # (plus ObjectID and Shape fields, if a feature class)
      arcpy.Merge_management("temp.dbf", "temp2.dbf", Mapper)

    author

    Curtis Price, U.S. Geological Survey, cprice@usgs.gov
    Not reviewed/approved use at your own risk
    """

    field_mappings = arcpy.FieldMappings()
    mapList = maps.split(';')

    for rec in mapList:
        fromName,toName = rec.split()
        # create a new field map
        field_map = arcpy.FieldMap()
        # populate it and add to field_mappings
        try:
            field_map.addInputField(tbl, fromName)
            field = field_map.outputField
            field.name = toName
            field_map.outputField = field
            field_mappings.addFieldMap(field_map)
        except:
            raise Exception(
                "FieldNameMap: Cannot not map fields ({}) in {}".format(rec,tbl))
    return field_mappings


# set up dictionary of from and to names
# first entry is db name, second is output name
renames = {"DRAINS":"DRAINS", 
           "PERSONS":"PEOPLE"}
def make_maps(tbl):
    fnames = [f.name.upper() for f in arcpy.ListFields(tbl)]
    # create field pairs list
    maps = [] 
    for f in fnames:
        for r in renames:
            if f.find(r) > -1: 
                maps.append("{} {}".format(f,renames[f]))
                break # done for this field
    maps = ";".join(maps)
    return FieldNameMap(tbl, maps).exportToString  # pass back to MB as string
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍
JoeBryant2
Occasional Contributor II

Curtis, thanks. That's awesome that you were able to mock that up and share it with me and everyone who may stumble across this post. 

I am going to return to this script the next time I need to model an Append and I have a little more time to get familiar with Python.

I was able to get the Poor Man's version setup (20 Alter Field operations plus some field calculating) and it works with the Append Tool when you set the field map as a parameter and leave it blank as you mentioned. I'm about to put it through it's paces, so we'll see if it holds up.

PaulLohr
Occasional Contributor III

Tim,

I am also frustrated by the bugs in Modelbuilder. It has been around long enough that a lot of these problems could be corrected by now. It is rare that I build a Modelbuilder process and it simply works without research time and a call to tech support.

timdunlevie
Occasional Contributor

Thanks Paul.

I might revert to a python script instead…as I know this works !!!

0 Kudos
CaryLancaster
New Contributor II

100% agree with you Paul.  it is 2018 and still the bugs are there.

0 Kudos
curtvprice
MVP Esteemed Contributor

I think it's important not to ask too much more of Model Builder than the environment is designed to support. If your workflow is complex (as in this example, where field mappings must be modified on each run based on the inputs), you may have to go to Python scripting. Even then, prototyping the process in ModelBuilder can save you a lot of time if the workflow and tools are complicated.

0 Kudos