Append GP Tool Field Map Bug

724
2
04-17-2023 09:43 AM
JohnStreeb
Occasional Contributor

After upgrading from ArcGIS Pro 2.9 to 3.1.1 I am experiencing some issues with the Append GP Tool that appear to be undocumented and are causing problems for a number of existing geoprocessing workflows, including various scripted workflows.  So, I'm trying to find out if this is a bug that is planning to be corrected, was an intentional change, or otherwise get more information so I can plan accordingly.

In the past, both in ArcMap and in ArcGIS Pro through at least v2.9 (I jumped right to 3.1.1, so I do not know at intermediary versions like 3.0 what the behavior is), if you opened the Append GP tool interactively, and set the "Field Matching Type" to "Use the field map to reconcile field differences", the auto-populated field map, in my experience, would auto map any source dataset fields to a target dataset field with an exactly matching field name.  However, now in ArcGIS Pro 3.1.1, some fields where there is an exact match in the field name, type, and length between the source and target datasets aren't auto-matching any more.  I can confirm from personal testing that any fields in the source dataset that are configured as Editor Tracking fields but the corresponding field in the target dataset is not, are not auto-matched any more, where they were previously.  The post at https://community.esri.com/t5/data-management-questions/append-data-management-gp-tool-field-mappin-... suggests that this is also affecting fields with Attribute Rules configured in the source dataset but not in the target dataset, again even when the name and properties all match.

Now, I know, when running the tool interactively, you can visually identify these with the "(0)" next to the field and then manually configure a field match; but frankly, that's a degradation in ease of use in the newer version of the software, which seems a poor choice by itself. 

However, the issue really comes in where the Append tool is used in existing scripted workflows, such as automated tasks to take data from a number of local source datasets and append them into a single regional dataset for example.  In my case for example, the source and target datasets contain overlapping sets of fields with the same name, type, length, etc... (each may have additional fields not found in the other, but they contain a core set of fields that are the same).  However, since the source datasets are directly edited by users and the regional (target) dataset is not, editor tracking is enabled on source datasets but not on target dataset, so the editor tracking data from source datasets can be preserved in the target database; though now these fields in my target dataset are Null after the 3.1.1 upgrade.  In the past, based on how the Append tool worked, if in the script you passed sources and the target with the "schema_type" property set to "NO_TEST", it would automatically append all data from the source fields into the identically named target dataset fields where there were identically named fields, without any explicit creation of or customization of a FieldMappings (and any fields without identical field names in both would simply be ignored).  In v3.1.1 however, this no longer is the case, and some fields, as noted above, even though they are identically named and with the same properties, no longer auto match.

So, since ESRI is typically pretty good about maintaining backwards compatibility through changes to GP tools and there is no documentation I've been able to find specifying this functionality was intentionally removed, I'm left wondering if this is a bug or if I'm missing something or what.  However, since this is breaking existing workflows, is causing errors in other ESRI solutions (see https://community.esri.com/t5/data-loading-tools-questions/issue-copying-from-editor-tracking-fields...), and there is no documentation related to it, I was hoping someone could chime in with a bit more information about this issue so I can downgrade my software for now, rewrite my scripts, or otherwise plan accordingly?

2 Replies
BillHeigh
Occasional Contributor

I am not able to help you with an answer, however I can confirm that the functionality of the Append tool is different than it was in Desktop. I was never able to get the Append to bring over matching attributes in Desktop, so I've been using NO_TEST ever since.

A couple of things I discovered as I transitioned to Pro was 1) not all the 'matching' fields matched up as expected, and 2) there is often a temporary path that seems to cause issues for me.

 

BillHeigh_1-1682532839410.png

 

I had success by X-ing out the Source entry and re-mapping it by picking the layer from the dropdown, which seems like it's coming right from the TOC as one might expect.

I want to take a moment here and point out that speaking for myself, I find it confusing that the Inputs are on the right side of this dialog, and the Target is on the left. There are several changes to the interface that take some getting used to, but I digress.

This re-mapping had to be done all the way down the list to get the expected results. This was not required in Desktop and is quite a bit of extra work compared to the way it works in Desktop.

0 Kudos
JohnStreeb
Occasional Contributor

As an update, regarding the auto-field mapping of the editor tracking fields specifically, ESRI has officially logged a bug.

BUG-000157802

Synopsis: The Field Map in the Append tool do not recognize the matching fields of a layer with Editor Tracking Enabled in ArcGIS Pro

0 Kudos