Hi -
One of my users contacted me because a python script he is running has started giving him some issues after our organization pushed Pro version 3.0.0 to our workstations. His script (below) is exporting a feature layer to a CSV, which is then being pulled into R for further analysis. Previously, the CSV would export and any null values in the attribute table would just be blank/empty. Now, they are all coded to "None" which has thrown off his processes in R. He added logic into the R script to deal with these new "None" values, however I was curious if there was some kind of change at Pro version 3.0.0 which could have caused this. His Py script had been updated to Python 3 awhile ago and was running fine. We are uncertain what the problem could be, but the issues started happening right around when we upgraded ArcGIS Pro.
I also notice that if I go to the hosted feature layer in ArcGIS Online and export to a CSV from the Item Details page, null values are empty and not coded as "None". Same if I view the feature layer in AGOL or Pro.
Any ideas on what could be causing this would be appreciated.
Here is the Python script:
#!C:\\Program Files\\ArcGIS\\Pro\\bin\\Python\\envs\\arcgispro-py3 python
# coding: utf-8
# In[60]:
import arcgis
from arcgis.gis import GIS
from arcgis.gis import *
import os
import getpass
portal_url = rhttps://mass-eoeea.maps.arcgis.com/
# ArcGIS user credentials to authenticate against the portal
username = 'USERNAME'
password = 'PASSWORD'
gis = GIS(portal_url, username, password)
# ID of the feature service you want to export
featureServiceID = "0d081b35bbe2466290926f0f09faffe5"
# Get username
username = getpass.getuser()
# Output format Shapefile | CSV | File Geodatabase
out_format = 'CSV'
# Local folder where the data will be downloaded (include slash at the end)
download_folder = "C:\\Users\\" + username + PATH
# Address of your ArcGIS portal
portal_url = rhttps://mass-eoeea.maps.arcgis.com/
download_file = download_folder + 'download.zip'
# ID of Field Data
item_id = featureServiceID
filename = "stage_turbidity" + "." + out_format
outpath = download_folder + filename
# Get content and save to csv
my_item = gis.content.get(item_id)
my_layer = my_item.tables[0].query().sdf
my_layer.to_csv(outpath, index = False)
#Download photos
download_folder_photos = download_folder + "Photos\\"
#Query to get list of object ids in layer
featureLayer = my_item.tables[0]
featureObjectIds = featureLayer.query(where='1=1', return_ids_only=True)
#Loop through features in layer
emptyAttachments = 0
for j in range(len(featureObjectIds['objectIds'])):
currentObjectId = featureObjectIds['objectIds'][j]
currentObjectIdAttachments = featureLayer.attachments.get_list(oid=currentObjectId)
if len(currentObjectIdAttachments) > 0:
#Loop through feature attachments and download to appropriate folder
for k in range(len(currentObjectIdAttachments)):
attachmentId = currentObjectIdAttachments[k]['id']
attachmentName = currentObjectIdAttachments[k]['name']
attachmentParentId = currentObjectIdAttachments[k]['parentGlobalId']
FileName = '{}'.format(attachmentName)
AttachmentPath = '{}\\{}'.format(download_folder_photos, FileName)
FinalFileName = '{}.{}'.format(attachmentParentId,"jpg")
FinalAttachmentPath = '{}\\{}'.format(download_folder_photos, FinalFileName)
if not os.path.isfile(FinalAttachmentPath):
featureLayer.attachments.download(oid=currentObjectId, attachment_id=attachmentId, save_path=download_folder_photos)
os.rename(AttachmentPath, FinalAttachmentPath)
print("Finished downloading csv file.")
# Run batch file to process csv outputs and import into Database
file = "C:\\Users\\" + username + PATH.bat
os.system('"' + file + '"')
print ("Stage/Turbdiity Data Processed with R Script" )
Thank you,
Erica
