How to publish a feature service to ArcGIS Online from an mxd USING PYTHON?

23504
44
Jump to solution
03-25-2013 07:06 PM
BenoitMetzger
New Contributor II
Hi All,

I would like to automate publishing feature services - from an mxd using Python to my AG Online account.

I did this successfully manually using the menu: Share as -> Service, Publish a service - Feature Access Capabilities etc...

Using Python, I managed to publish the service definition file on AG Online using:
arcpy.mapping.CreateMapSDDraft()
arcpy.mapping.AnalyzeForSD()
arcpy.StageService_server()
arcpy.UploadServiceDefinition_server()

But I could not find a way to publish a feature service to AG Online! I must be missing something. Any help would be really appreciated.

Thanks in advance!
Tags (2)
44 Replies
KevinHibma
Esri Regular Contributor
From the previous testing I've done, the following should do what you want:


  • Log on to the machine, open ArcMap, and sign in (make note of the account you logged into Windows with)

  • Make sure you've checked the automatic sign in on the Portal sign in box.

  • Create your scheduled task (it must run under the same windows user account from above)


The key here is to log in once, have it be remembered, then have the task run under the user account which had logged in previously.

Of course, make sure the SignInToPortal tool has been commented out.
If I get time later this afternoon I'll give this another go.
0 Kudos
ErikEndrulat
Occasional Contributor
I tried the approach you suggested but was unable to successfully publish the service as a scheduled task (or manually while not logged in through Desktop). I'm very interested in any other suggestions you or other folks may have that would enable running the update FS as a scheduled task.

Thanks!
Erik
0 Kudos
JeffMoulds
Esri Contributor
Erik,

Are you getting an error when running the python script? Or are you having a problem creating a scheduled task?
0 Kudos
ErikEndrulat
Occasional Contributor
I get the error when running the python script, and the scheduled task fails (presumably from the same error but I haven't logged the errors yet).
0 Kudos
JeffMoulds
Esri Contributor
Are you getting a different error now that you implemented the new sign in methodogy? Can you send me the exact error message? Thanks.
0 Kudos
ErikEndrulat
Occasional Contributor
Correction on previous post: I attempted to run the publishing script again today from within my python IDE (pyscripter) and it executed correctly without having signed into ArcGIS Online manually through Desktop. I hadn't changed anything in my script from last week before executing it, so it's possible that logging out and back in on the system may account for it working now.

However, I confirmed that I am still unable to get it to work though a scheduled task. I logged the error to a text file, and the following was recorded:

Executing: UploadServiceDefinition C:\Workspace\KYEM_Test\KYEM_TEST3.sd "My Hosted Services" KYEM_TEST3 # FROM_SERVICE_DEFINITION # STARTED OVERRIDE_DEFINITION SHARE_ONLINE PRIVATE NO_SHARE_ORGANIZATION #
Start Time: Mon Sep 16 14:06:05 2013
Failed to execute. Parameters are not valid.
ERROR 000732: Server: Dataset My Hosted Services does not exist or is not supported
WARNING 001404: You are not signed in to ArcGIS Online.
Failed to execute (UploadServiceDefinition).
Failed at Mon Sep 16 14:06:05 2013 (Elapsed Time: 0.02 seconds)



The script (only minor changes from Esri's sample):

import arcpy, os, sys
from datetime import datetime
import xml.dom.minidom as DOM

arcpy.env.overwriteOutput = True

# Define global variables from User Input
mapDoc = 'KYEM.mxd'
serviceName = 'KYEM'
shareLevel = 'PRIVATE'              # Options: PUBLIC or PRIVATE
shareOrg = 'NO_SHARE_ORGANIZATION'  # Options: SHARE_ORGANIZATION and NO_SHARE_ORGANIZATION
shareGroups = ''                    # Options: Valid groups that user is member of

tempPath = sys.path[0]

#SignInToPortal function does not work in 10.2
#for 10.1: uncomment below line to enable sign in
#for 10.2:  (1): Sign in to arcgis desktop and check 'sign in automatically'
#           (2): Schedule task to run publishFS.py script, using same user as in step 1
##arcpy.SignInToPortal_server('','','http://www.arcgis.com/')

sdDraft = tempPath+'/{}.sddraft'.format(serviceName)
newSDdraft = 'updatedDraft.sddraft'

try:
    os.remove(serviceName+'.sd')
    print 'SD already exists, overwriting..'
    SD = os.path.join(tempPath, serviceName + '.sd')
    print 'File removed and overwritten'
except OSError:
    print 'No SD exists, writing one now.'
    SD = os.path.join(tempPath, serviceName + '.sd')

try:

    # create service definition draft
    analysis = arcpy.mapping.CreateMapSDDraft(mapDoc, sdDraft, serviceName, 'MY_HOSTED_SERVICES')


    # Read the contents of the original SDDraft into an xml parser
    doc = DOM.parse(sdDraft)

    # The follow 5 code pieces modify the SDDraft from a new MapService
    # with caching capabilities to a FeatureService with Query,Create,
    # Update,Delete,Uploads,Editing capabilities. The first two code
    # pieces handle overwriting an existing service. The last three pieces
    # change Map to Feature Service, disable caching and set appropriate
    # capabilities. You can customize the capabilities by removing items.
    # Note you cannot disable Query from a Feature Service.
    tagsType = doc.getElementsByTagName('Type')
    for tagType in tagsType:
        if tagType.parentNode.tagName == 'SVCManifest':
            if tagType.hasChildNodes():
                tagType.firstChild.data = 'esriServiceDefinitionType_Replacement'

    tagsState = doc.getElementsByTagName('State')
    for tagState in tagsState:
        if tagState.parentNode.tagName == 'SVCManifest':
            if tagState.hasChildNodes():
                tagState.firstChild.data = 'esriSDState_Published'

    # Change service type from map service to feature service
    typeNames = doc.getElementsByTagName('TypeName')
    for typeName in typeNames:
        if typeName.firstChild.data == 'MapServer':
            typeName.firstChild.data = 'FeatureServer'

    #Turn off caching
    configProps = doc.getElementsByTagName('ConfigurationProperties')[0]
    propArray = configProps.firstChild
    propSets = propArray.childNodes
    for propSet in propSets:
        keyValues = propSet.childNodes
        for keyValue in keyValues:
            if keyValue.tagName == 'Key':
                if keyValue.firstChild.data == 'isCached':
                    keyValue.nextSibling.firstChild.data = 'false'

    #Turn on feature access capabilities
    configProps = doc.getElementsByTagName('Info')[0]
    propArray = configProps.firstChild
    propSets = propArray.childNodes
    for propSet in propSets:
        keyValues = propSet.childNodes
        for keyValue in keyValues:
            if keyValue.tagName == 'Key':
                if keyValue.firstChild.data == 'WebCapabilities':
                    keyValue.nextSibling.firstChild.data = 'Query,Create,Update,Delete,Uploads,Editing'

    # Write the new draft to disk
    f = open(newSDdraft, 'w')
    doc.writexml( f )
    f.close()

     # Analyze the service
    analysis = arcpy.mapping.AnalyzeForSD(newSDdraft)

    if analysis['errors'] == {}:
        # Stage the service
        arcpy.StageService_server(newSDdraft, SD)

        # Upload the service. The OVERRIDE_DEFINITION parameter allows you to override the
        # sharing properties set in the service definition with new values.
        arcpy.UploadServiceDefinition_server(SD, 'My Hosted Services', serviceName, '', '', '', '', 'OVERRIDE_DEFINITION','SHARE_ONLINE', shareLevel, shareOrg, shareGroups)

        print 'Uploaded and overwrote service'

        # Write messages to a Text File
        txtFile = open(tempPath+'/{}-log.txt'.format(serviceName),"a")
        txtFile.write (str(datetime.now()) + " | " + "Uploaded and overwrote service" + "\n")
        txtFile.close()

    else:
        # If the sddraft analysis contained errors, display them and quit.
        print analysis['errors']

        # Write messages to a Text File
        txtFile = open(tempPath+'/{}-log.txt'.format(serviceName),"a")
        txtFile.write (str(datetime.now()) + " | " + analysis['errors'] + "\n")
        txtFile.close()

except:

    print arcpy.GetMessages()
    # Write messages to a Text File
    txtFile = open(tempPath+'/{}-log.txt'.format(serviceName),"a")
    txtFile.write (str(datetime.now()) + " | Last Chance Message:" + arcpy.GetMessages() + "\n")
    txtFile.close()
JeffMoulds
Esri Contributor
When you got it to work in an IDE, did you have ArcMap open at the same time? And when you ran it as a task, where you logged out of the machine?

With the new security enhancements introduced at 10.2, it is necessary to have ArcMap open to get the sign-in information, even with "sign me in automatically" checked on. I realize that this is not ideal as it makes nightly auto-updates to hosted features services nearly impossible. A time frame for getting this to work is still being investigated on our end.

We also have another bug where overwriting an existing hosted feature service sometimes fails - usually every second time for most services. We are investigating a fix for this issue as well. Let me know if you are hitting this issue as well - try running your script several times in a row (perhaps 3 times).

I will let you know when I have more information.
0 Kudos
MikkelHylden
Occasional Contributor

I realize that this post is over two years old now, but I'm trying to run this script using 10.3 - has this requirement to be logged in through ArcMap been corrected?  I'm trying to set up exactly the situation you described - a nightly auto-update. 

Any ideas are greatly appreciated..

0 Kudos
ErikEndrulat
Occasional Contributor
Jeff:

Yes, I have successfully run this script while ArcMap was not open, though the ArcGIS Connection globe is present in the taskbar and indicates that i'm signed into ArcGIS Online. I have not yet been able to run the task while logged out,

I've also run into the problem where the script fails when run several consecutive times, I get this error:

Executing: UploadServiceDefinition C:\Workspace\KYEM_Test\KYEM_TEST3.sd "My Hosted Services" KYEM_TEST3 # FROM_SERVICE_DEFINITION # STARTED OVERRIDE_DEFINITION SHARE_ONLINE PRIVATE NO_SHARE_ORGANIZATION #
Start Time: Thu Sep 19 08:41:03 2013
ERROR 001566: Service overwrite error: failed to delete the service.
ERROR: code:400, Item 'e18aae36f2874deba39e380611c59745' does not exist or is inaccessible., Bad syntax in request.
Failed to execute (UploadServiceDefinition).


However, when attempted again it typically works, so this seems like a relatively minor problem that can be solved with a 'try until no error' loop.

I'd definitely appreciate it if you let me know when you have more information, specifically about the sign-in issue.

Thanks,
Erik
0 Kudos
JeffMoulds
Esri Contributor
Even though you are able to run the script from an IDE with ArcMap closed, you may eventually find that over time, the AGO connection invalidates itself. I dont know exactly how long it will be valid for. For example, if I shut down ArcMap (without my IDE open), the connection goes away after a couple minutes. However, if I have ArcMap open, then run my script from an IDE, then close ArcMap, the connection stays alive for longer until I shut down the IDE - I am trying to get an answer on how this exactly works and how long the token is valid for. A workaround to this in your python script would be to open arcmap and force it to connect, then execute the code to publish the service, then kill arcmap. Something like:
from subprocess import Popen, PIPE
import win32com.client
import time
import os
proc = Popen(["C:\\Program Files (x86)\\ArcGIS\\Desktop10.2\\bin\\ArcMap.exe", "c:\\temp\\states.mxd"], stdout=PIPE) # put in a dummy mxd to get around the start dialog
time.sleep(10)
shell = win32com.client.Dispatch("WScript.Shell")
shell.SendKeys("%f", 0) # sends ALT+F to the UI, which signs you in - just opening ArcMap doesnt sign you in. You have to click the file menu
< code to overwrite a hosted feature service goes here >
os.system("taskkill /im arcmap.exe /f")


And yes, 'Try until no exception is raised' condition should get around the issue of it failing every second time.

It's ugly, but it should work if you are logged in. I havent thouht of a workaround that'll work if you are logged off your machine.
0 Kudos