Batch Upload of ArcGIS Pro Field Calculations to AGOL

MErikReedAugusta

Right now, we have a field effort that is deployed via AGOL and Field Maps, and occasionally we find ourselves needing to run a Field Calculation on the dataset without wanting to download the whole thing, run the calculation, and then re-upload.

No problem, you say, because you can just go through Portal in ArcGIS Pro to access your AGOL data. You're right; that's great!

Only... if you try to run a Field Calculation on a column, it takes eons to process, compared to the same calculation on a dataset local to your machine. Near as I can tell, it seems like it refreshes the data from AGOL after every calculation, which is the added time sink.

Adding an option to bundle all of those calculations together and have them post to AGOL together at once would be a godsend for this. Most of the time, these are calculations that don't rely on other features, so there's no real need to update AGOL one at a time. Batch-updating after the Field Calculation is done would be preferable.

I could see this handled on the Share toolbar, most likely. Maybe a toggle switch to hold all updates to AGOL until you toggle it back. OFF, your changes stay local to your machine. ON, it syncs between the two, maybe with a settable option for which side to favor in case of collisions of edits (though I would argue best practice would be to avoid these in the first place wherever possible).

KoryKramer · ‎10-25-2023

Are you able to choose SQL for the calcuation?

See https://pro.arcgis.com/en/pro-app/latest/tool-reference/data-management/calculate-field.htm

SQL expressions support faster calculations for feature services and enterprise geodatabases. Instead of performing calculations one feature or row at a time, a single request is sent to the server or database, resulting in faster calculations.

Only feature services and enterprise geodatabases support SQL expressions. For other formats, use Python or Arcade expressions.

JonathanNeal · ‎10-26-2023

@MErikReedAugusta You should get a larger performance increase by using SQL for FeatureService and egdb data. If you are finding that this is not the case let us know.

MErikReedAugusta · ‎10-26-2023

@KoryKramer @JonathanNeal

I vaguely remember SQL not being significantly faster and/or not being feasible for the calculations we've run in the past, but I'll have to run tests on this again to be positive.

What I know of SQL tells me that this isn't always going to be a viable solution, though, because I can't always rework the expression into a SQL expression (though maybe my SQL is just too rusty).

For context of scale: Last time we did testing on this a few months ago, it was a script that ran on our entire local dataset (FGDB, Python calculations) in a few minutes. Running it on the same dataset hosted in ArcGIS Online was measured in hours.

Is it theoretically possible to implement the option of having Python or Arcade flip a setting and have ArcPro send a single request to the server/database like this in the future?