branch versioning vs. mass updates

3257
17
Jump to solution
02-02-2023 07:25 AM
RiccardoKlinger
Occasional Contributor

Hi folks, 

we are switching from traditional versioning to branch versioning as we are also switching from direct db access to a servcie based architecture. 

In the past we made bulk updates on tables directly via sql but disabled versioning prior these activities and re-enabling the traditional versioning after this update. This is against the "recommodation" from esri but it was working quite good. 

As the new branch versioning is new to us we also want to follow the proposed workflow from Esri but as a consequence we will most likely see a grow of our tables by factor 2 with every bulk update. This will be "mad" after a few weeks in our use case....
Of course we could also stop the feature service, disable versioning do our "mass updates" via sql or feature servcie and enable versioning again. But this will b e not in line with the esri policy, will it?

Are there any other users facing a similar problem?  How have you overcome this problem? 

I have seen this comment from the 2022 UC 

Can the same feature class be set to use both traditional and branch versioning?A dataset can only have one registration type. It would either be traditional or branch versioned.This pattern should be consistent for all datasets found within a feature dataset.

 

Looking forward your replies.

 

Best, Riccardo

17 Replies
RiccardoKlinger
Occasional Contributor

Hi Marcelo,
I don't use network utility. So I don't need to read this. I know, that direct sql access is not "supported" yet I can write directly into the db in this table.
And as  I mentioned above: I would only write directly to the db, after deregister as versioned in the db connection/sde. after direct sql access I would simply register the table as versioned again.

0 Kudos
MarceloMarques
Esri Regular Contributor

I understand. yes that will work if you unregister as versioned first.

| Marcelo Marques | Principal Product Engineer | Esri |
| Cloud & Database Administrator | OCP - Oracle Certified Professional |
I work with Enterprise Geodatabases since 1997.
“ I do not fear computers. I fear the lack of them." Isaac Isimov
0 Kudos
RiccardoKlinger
Occasional Contributor

Why is branch-versioning so "utility-network" centric? I don't get it.

Yet, thanks for the links.

0 Kudos
MarceloMarques
Esri Regular Contributor

"branch versioning" was created because of the new ArcGIS Enterprise - Portal + ArcGIS Server service architecture in mind to better support editing workflows via the web and in particular, of course, "utilities" electric, water, gas, oil, etc. to modernize their workflows, as consequence the new "utility network" replaces the old "geometric network", and the new "utility network" only works with "branch versioning". I hope this clarifies your question. -Marcelo : )

| Marcelo Marques | Principal Product Engineer | Esri |
| Cloud & Database Administrator | OCP - Oracle Certified Professional |
I work with Enterprise Geodatabases since 1997.
“ I do not fear computers. I fear the lack of them." Isaac Isimov
0 Kudos
RiccardoKlinger
Occasional Contributor

David, I feel you!

As you described it, it would be the way to work with the db while the table is still registered as "branch versioned table".... If I swith off the versionining for the table, the process is even simpler. I will then make the bulk edits, and switch on the versioning again.

0 Kudos
MarceloMarques
Esri Regular Contributor

Hello @RiccardoKlinger 

Please, bear with me. I can see 2 scenarios about "branch versioning vs. mass updates" that you have already described.

A. if the branch versioned featureclass is small with just some million rows and the actual featureclass table size in the Enterprise Geodatabase RDBMS Storage is around let's say 20GB or so.

  1. then you can open Pro
  2. connect to Portal
  3. add the Feature Service to the map
  4. and perform the mass attribute update using the Pro editing tools directly in the branch version sde.default
  5. can even use model builder to automate the process for example.
  6. hence there is no need in this scenario to unregister as branch versioned the featureclass(es)

B. on the other hand if the branch versioned featureclass is quite large with 500 milllions rows for example and the featureclass table size in the Enterprise Geodatabase RDBMS storage is around let's say 1 TB.

  1. then open Pro
  2. reconcile and post all branch versions child of sde.default
  3. close Pro
  4. take a full database backup
  5. stop the ArcGIS Server Feature Service(s)
  6. need to do this because of locks that might prevent from unregister as branch versioned
  7. open pro and unregister as branch version the featureclass(es) then close Pro
  8. perform the mass attribute update using SQL statement
  9. sure thing the mass field update will be faster via SQL
  10. here consider to ask your database administrator to place the database in nologging (oracle), simple recovery mode (sql server) before you do the mass update via SQL, this will speed up the mass update via SQL
  11. after the update is done the database administrator can change the database back to logging (oracle), full recovery mode (sqlserver)
  12. open pro and register as branch version the featureclass(es) again and close Pro
  13. start the ArcGIS Server Feature Service(s)
  14. test the feature service and make sure you can make edits in a new branch version child of sde.default
  15. if everything is fine then take another full database backup

@RiccardoKlinger this is pretty much the same process that you have already described in your original question, but I just wanted to create a step by step to clarify for anyone reading this discussion.

I do not see any major problems with Scenario B and you have also mentioned that this is working fine for you, hence although this might not be officially sanctioned by Esri yet. I do not see any other way of doing the branch version mass update for very large featureclasses, Scenario B is the more appropriate way that avoids any data corruption.

Thanks,

| Marcelo Marques | Principal Product Engineer | Esri |
| Cloud & Database Administrator | OCP - Oracle Certified Professional |
I work with Enterprise Geodatabases since 1997.
“ I do not fear computers. I fear the lack of them." Isaac Isimov
ArizonaGIS
New Contributor III

Hello. When I try to create a new child version using the ArcGIS Python API to write some updates, I receive this message below. I wanted to know if anyone has successfully created a temporary child version, applied edits, reconciled/posted the updates to the parent, and deleted the child version in the process.

Error creating version: Failed to execute. Parameters are not valid.
ERROR 000837: The workspace is not the correct workspace type.
WARNING 000379: The value may not be valid because of empty domain or does not exist in the domain
Failed to execute (CreateVersion).

from arcgis.gis import GIS
from arcgis.features import FeatureLayer, Feature
import arcpy
import time

# Record the start time
start_time = time.time()

# Connect to the GIS
gis = GIS(url="YOUR PORTAL URL", username="YOUR USERNAME", password="YOUR PASSWORD")

# Path to the SDE file (i.e., the database connection file)
database = r"P:\GIS.sde"

# Define version names
parent_version = "sde.DEFAULT"
child_version = "Child_Version"

# Try to create the child version
try:
    arcpy.CreateVersion_management(database, parent_version, child_version, "PUBLIC")
    print(f"Created version: {child_version}")
except Exception as e:
    print(f"Error creating version: {e}")
    arcpy.AddError(e)
    sys.exit()  # stop the script if version creation fails

# Create a feature layer from the feature class
feature_layer = arcpy.management.MakeFeatureLayer(feature_class, "FeatureLayer").getOutput(0)

# Switch to the new version
arcpy.ChangeVersion_management(feature_layer, "TRANSACTIONAL", f"{child_version}")

# Get the feature service
service_id = "SERVICE ID"
service = gis.content.get(service_id)
layer = service.layers[0]

# Fetch the features to be updated
features = layer.query()  # You may need to refine this query

# Create a dictionary to match the prefixes to the correct "St_PreTyp" value
prefixes = {
    "Interstate": "Interstate"
}

# Loop through the features
for feature in features:
    # Get the "St_Name" value
    st_name = feature.attributes["St_Name"]
    
    if st_name is not None:
        update_required = True
        # Check if St_Name contains only the Spanish prefix
        if st_name.strip() in ["Calle"]:
            update_required = False  # No update required

        # If update is required, loop through the prefixes to find a match
        if update_required:
            for prefix, pretyp in prefixes.items():
                if st_name.startswith(prefix):
                    # If additional characters are present, parse and update the fields
                    feature.attributes["St_PreTyp"] = pretyp
                    feature.attributes["St_Name"] = st_name.replace(prefix, "").lstrip().replace("Highway", "").lstrip()
                    break
    else:
        feature.attributes["St_PreTyp"] = None

# Submit the edits
layer.edit_features(updates=features)

# After performing your operations, reconcile and post the changes from the child version to the parent version
arcpy.ReconcileVersions_management(database, "ALL_VERSIONS", parent_version, child_version, "LOCK_ACQUIRED", "NO_ABORT", "BY_OBJECT", "FAVOR_TARGET_VERSION", "POST", "DELETE_VERSION")

# Switch back to the parent version
arcpy.ChangeVersion_management(database, "TRANSACTIONAL", parent_version)

# Delete the child version
arcpy.DeleteVersion_management(database, child_version)

# Record the end time
end_time = time.time()

# Calculate the total time in minutes
total_time = (end_time - start_time) / 60

print("Successfully updated parsing for feature class")
print("Total time taken: {:.2f} minutes".format(total_time))

 

0 Kudos
JOSEPHAUSBY
New Contributor

Hi there,

What was your final solution to bulk edit data using pure T-SQL.  We also have 100's of ETL processes that push/pull data directly into SQL from Esri databases.  We now are using Utility Network and branch versioning.  

0 Kudos