I've set up a geoprocessing service for a client which is an ETL tool. Most of the time it works, but occasionally it fails with an error similar to the following. On each occasion I'm running it for the same area of interest, same map layer to extract from and same format (shape file), ie. all the input parameters are the same each time, yet sometimes it works, sometimes it fails.
Error executing tool. MyExtract Job ID: jc9b53901a9dd43d489210ecb9fbe7533 : Traceback (most recent call last): File "\agisdev\arcgisserver\directories\arcgissystem\arcgisinput\GeoMy\ExtractDataTask.GPServer\extracted\v101\geoMy\ExtractData.py", line 522, in zipFolder = clipAndConvert(layers, areaOfInterest, featureFormat, rasterFormat, coordinateSystem) File "\agisdev\arcgisserver\directories\arcgissystem\arcgisinput\GeoMy\ExtractDataTask.GPServer\extracted\v101\geoMy\ExtractData.py", line 368, in clipAndConvert numFeatures = clipFeatures(lyr, aoi, featureFormat, zipFolderPath, scratchFolderPath, convertFeaturesDuringClip, dctColsMaster, dctRowsMaster, dctColsExport, wsExport) File "\agisdev\arcgisserver\directories\arcgissystem\arcgisinput\GeoMy\ExtractDataTask.GPServer\extracted\v101\geoMy\ExtractData.py", line 246, in clipFeatures with arcpy.da.SearchCursor(scratchpath, ["OID@"]) as cursor: RuntimeError: cannot open 'C:\Users\agis-srv\AppData\Local\Temp\geoMy\extractdatatask_gpserver\jc9b53901a9dd43d489210ecb9fbe7533\scratch\scratchfolder\ctLand_Dispositions.shp' Failed to execute (My Extract). Failed to execute (ExtractDataTask). Failed to execute (MyExtract).
The error seems to be that it can't find this file:
'C:\Users\agis-srv\AppData\Local\Temp\geoMy\extractdatatask_gpserver\jc9b53901a9dd43d489210ecb9fbe7533\scratch\scratchfolder\ctLand_Dispositions.shp'
Why is it creating a scratch folder here? I've looked for this file in \agisdev\arcgisserver\directories\arcgisjobs, where I'd expect it to be but the job folder, jc9b53901a9dd43d489210ecb9fbe7533, doesn't exist here at all, not in any of the subfolders.
What's going on? Is it perhaps running out of memory occasionally and having to use the server's temp folder as its scratch workspace?