Benchmark GP tool execution times against previous version

Bud

Edited.

I came across an issue in ArcGIS Pro 2.9.12 where the Make Route Event Layer geoprocessing tool was much slower than previous versions (2.6.8).

Route Event Layers — Slow performance in ArcGIS Pro 2.9.12, but not in 2.6.8

That performance issue is a significant problem for my unit. We can't upgrade from 2.6.8 to 2.9.12 because route event layers are unuseable in 2.9.12. (My organization will upgrade to 3.x eventually; that's beyond my control.)

I'm wondering if that kind of performance issue could have been caught by benchmarking GP tool execution times against previous versions of Pro to find issues. Could that be incorporated into existing automated Esri testing mechanisms?

For example, if that had been done, then it would have been clear that the Make Route Event Layer tool is significantly slower in 2.9.12 compared to previous versions. That would have raised a red flag, and Esri could have addressed the Route Event Layer performance issues before releasing 2.9.12.

SSWoodward

Thanks for bringing up this important topic, @Bud . We do try our very best to address performance issues and understand it can be frustrating when things like this happen.

The specific issue you've brought up was introduced in Pro 2.9.6 and 3.1 and has been addressed in the coming release of ArcGIS Pro 3.3

Since this Idea is in reference to internal testing practices and not an implementable feature in ArcGIS Pro, it will be closed.

If you're looking for more information on guidelines for Ideas on the ESRI communities pages, you can refer to the submission guidelines at the link below.

ESRI Ideas Submission Guidelines

RTPL_AU

@SSWoodward From a paying customer perspective if you already have internal testing processes that detect regressions, these regressions should be public and part of the release notes, and if not, this is still a valid idea and should be moved a more appropriate section such as "things that Esri could do better that isn't a product feature".
Just closing an Idea related to our experience of Pro that is "not an implementable feature in ArcGIS Pro" does not seem very nice?
Then pointing an active and experienced contributor to your guidelines comes across a bit harsh.
So many Ideas are related to Pro performance and issues surrounding them, that you have to, by now, realise that you have a problem and that you have loyal customers trying to help.

Bud

@SSWoodward Thanks for providing this info:

The specific issue you've brought up was introduced in Pro 2.9.6 and 3.1 and has been addressed in the coming release of ArcGIS Pro 3.3

That certainly helps. We will try upgrading to 2.9.5 instead of 2.9.12.

I like @RTPL_AU's comment:

...if you already have internal testing processes that detect regressions, these regressions should be public and part of the release notes...

@SSWoodward, if I were to create a new post as described below, would you close it?

Idea: Documentation — Provide GP tool execution time benchmarks, including comparison against previous Pro versions
Label: Documentation & Help

I'm open to input on the title/content of such an idea.

MarcoBoeringa

@RTPL_AU wrote:

@SSWoodward From a paying customer perspective if you already have internal testing processes that detect regressions, these regressions should be public and part of the release notes, and if not, this is still a valid idea and should be moved a more appropriate section such as "things that Esri could do better that isn't a product feature".

I think I would even go a step further: regarding the nature of the bugs when there is a new software release, there are two clearly distinguishable ones in my opinion:

- 1) Bugs in newly added functionality that doesn't affect any of the existing functionality of ArcGIS and allows the current user base to continue using Pro unhindered as they did with the previous release, and to potentially upgrade to the latest release with the minor caveat of needing to avoid the affected functionality. While it might be a nuisance that some fancy new feature is not usable due to such an issue, such bugs in my opinion are relatively harmless and lower priority from the perspective of the user base (unless there is a risk that the bug causes major damage to e.g. an enterprise geodatabase, but such bugs are rare).

- 2) Regression bugs that directly affect the usability of existing functionality of ArcGIS, and make it impossible to use and / or upgrade to the latest release either due to a severe performance issue, or tools / functionality being truly broken and totally unusable.

In my opinion, the second type of bugs - regression bugs - is far worse than the first, and should always be top priority for fixing as they make it impossible for the existing user base to use or upgrade to the latest version while continuing their regular workflow.

The specific issue you've brought up was introduced in Pro 2.9.6 and 3.1 and has been addressed in the coming release of ArcGIS Pro 3.3

In my opinion, regression bugs should always be fixed in the next patch release, not in a major or minor release! That is, this bug should have been fixed in a 2.9.x or 3.1.x patch, not in Pro 3.3. Or, if the original fix was developed based on the 3.3 code base, it should have been back ported to all previous releases affected.

I realize time constraints may make it impossible to fix stuff in the next patch release, e.g. 2.9.7, but that still means the fix should have been part of 2.9.8 or so.

If you do not follow such practices, you may end up with a perpetually broken product, because regressions are never fixed in the release cycle as used by the existing user base.