Workforce API - Missing Data...

1367
11
08-18-2017 02:15 AM
JonathanGroves
New Contributor III

This morning, assignments returned by the export_assignments_to_csv.py script (part of the standalone scripts collection) are missing newer assignments - I can see none since 1st August. When I review the Dispatcher interface I can still see the newer assignments.

It seems there is some sort of problem with the API.

How best to address this please?

0 Kudos
11 Replies
CraigGillgrass
Esri Regular Contributor

Jonathan, are you still running into this issue?

If so, can you post the command line arguments you're using?

0 Kudos
JonathanGroves
New Contributor III

Hi, Craig.

Thanks so much for responding!

Yes - every day. I really can't see what has changed my end...

#First we delete the old temporary output file:
sudo cp -f /home/pi/arcgis/AssignmentsDownload.blank /home/pi/arcgis/AssignmentsDownload.csv;
# Build up string:
q="sudo python /home/pi/arcgis/export_assignments_to_csv.py "
# add output file:
q+="-outCSV \"/home/pi/arcgis/AssignmentsDownload.csv\" "
# add username:
q+="-u \"redacted\" "
# password:
q+="-p \"redacted\" "
# url:
q+="-url \"redacted\" "
# PID:
q+="-pid  \"redacted\" "
# Logfile:
q+="-logFile \"./reporter_log.txt\" "
# Date format:
q+="-dateFormat \"%d/%m/%Y %H:%M:%S\" "
# Timezone:
q+="-timezone \"US/Eastern\" "
# Execute string (calls Python script):
echo $q;
This has been working fine for weeks and weeks - it's run by a cron job every morning - but last week the more recent Assignments are missing from the CSV. This is part of a more complex system but the data is definitely missing from the source CSV file.
Any help gratefully received.
Kind regards,
Jonathan
0 Kudos
CraigGillgrass
Esri Regular Contributor

Jonathan,

At this point, I can suggest a couple ways to track this down, but a solution may take some time; have you tried confirming the queries you are using return the correct features through the REST endpoint?

When you say "the more recent assignments are missing"  what does more recent mean?  Is this the creation date of the assignments?

0 Kudos
JonathanGroves
New Contributor III

Hi Craig, thanks for your response.

Please could you point me to the information I need to access the REST endpoint to do this?

"More recent" means "assignments which have been updated (as in, status changed, notes added etc) since about 1st August". It's hard to be more precise - I'm interested in extracting Assignments which have Notes fields populated and (sadly) the date a Note was left is not provided by the script. I have a Python script which filters out items from the AssignmentsDownload.csv which have Notes fields populated - it builds a new CSV of these and sends them to a bunch of interested parties. These "more recent" Assignments are missing from the AssignmentsDownload.csv (which is the output of your standalone script). Bizarrely, some of these missing Assignments have previously been extracted successfully. Something changed around the 8th of August, but it wasn't my end...

I've got used to dealing with script runtime errors caused by strange characters entered in Notes by site staff - and my follow-on script now correctly handles commas in the Notes (which, of course, upset csv formatting!) - but this is definitely something different. Happy to email a sample AssignmentsDownload.csv which is missing the data if it's of help, but can't see that it would be...

Very grateful for your help on this.

J

0 Kudos
CraigGillgrass
Esri Regular Contributor

Hi Jonathan,

Yes, you can access the REST endpoint for services through the Service URL link on the item details page.  Scroll to the bottom of the Service URL page and you'll see a number of Supported Operations; one being Query.  You can use this to prove out the queries you're using.

Little stumped by what happened on August 8th; there were no changes on our end for Workforce.  Can you provide an example of the command line parameters you specify when you run the script?

Craig

0 Kudos
JonathanGroves
New Contributor III

Thanks, Craig.

Not sure where the "service URL page" is but I'll investigate...

The command line parameters I specify are shown in the bash script extract earlier in this thread.

Kind regards,

JG

0 Kudos
Kylie
by Esri Regular Contributor
Esri Regular Contributor

Jonathan - You can access the service URL page through the item page in ArcGIS Online. Here is an example of the item page for an assignments layer, and under the layers, there is a service URL link. That takes you to the service URL page.

Esri Education Team
0 Kudos
JonathanGroves
New Contributor III

Awesome – thanks so much for this.

Kind regards,

Jonathan

0 Kudos
JonathanGroves
New Contributor III

Hi.

Failed to make head nor tail of the REST endpoint so am unable to verify if the data all appear in there.

Please advise what the constraints are for Notes fields - in terms of maximum length, characters used etc - for use with the standalone scripts. I have previously seen an odd character (an I with acute accent) crash the download to csv script, and I'm starting to think my current problem might be due to Notes field length after all.

A Notes-related suggestion:

As a Project Owner, I want to have a filter allowing Dispatchers to view all Assignments which have a Notes field containing data, so that they can easily see any exceptions reported by site personnel.

Kind regards,

Jonathan

0 Kudos