Hey guys - need some help!
I'm having an issue where ArcGIS Server will not publish map services sourced to an MS SQL group listener alias. I can publish the same service when sourced directly to the SQL cluster - same data, same process (x,y event layer), except one publishes without issue, and the other not.
I have tried publishing the service using the ArcMap GUI. I've tried saving a service definition draft and staging/uploading using Server Tools. I've tried publishing from within Manager - nothing works when using the group listener. I've also tried manually setting APPLICATIONINTENT and MULTISUBNETFAILOVER as per Connections to highly available SQL Server databases—Help | ArcGIS Desktop. I've tried different naming conventions, not immediately starting the service, using different folders, creating new ones, registering the group listener db using different naming conventions, re-applying SQL user perms... nothing works.
I can connect to the instance in ArcCatalog and manipulate/geoprocess in ArcMap without issue... it's the publishing that almost immediately fails:
As per recommendation, I've added perms to every directory I've run across related to ArcGIS/Esri - I have no problems publishing other services, just ones related to the group listener.
I noticed others have had issues connecting - Group listener for Database is not connecting - again, I have no issues from connecting from within ArcMap - just with publishing.
I've spun down all services, restarted Publishing Tools, recycled the ArcGIS Server Windows Services process, restarted the server, restarted the Web Adaptor server, adjusted min/max processes in Manager for each service... nothing works.
Here's some server logs that may help from around the time processing failed... I bolded some of the things that looked interesting and found some stuff on Geonet, but nothing conclusive. Keep in mind, this is from multiple attempts back-to-back... I think I was loading from manager at the time and tried to publish a few times after failing:
SEVERE Apr 3, 2018, 9:49:18 AM Error executing tool. Publish Service Definition Job ID: j8bbd31a22a804aa39ea8f9d769a0e30c System/PublishingTools.GPServer
SEVERE Apr 3, 2018, 9:49:18 AM Error executing tool. Publish Service Definition Job ID: j8bbd31a22a804aa39ea8f9d769a0e30c : ERROR 001360: Failed to get basic item metadata for service definition upload. ERROR 001362: Failed to extract manifest from uploaded service definition. Failed to execute (Publish Service Definition). System/PublishingTools.GPServer
SEVERE Apr 3, 2018, 9:49:18 AM Failed to extract manifest from uploaded service definition. System/PublishingTools.GPServer
SEVERE Apr 3, 2018, 9:49:18 AM Failed to get basic item metadata for service definition upload.: ERROR: code:404, Could not find resource or operation 'i6d8d0dff-1297-4b6f-a71c-4c90a7463298' on the system. No resource could be found at that address. System/PublishingTools.GPServer
SEVERE Apr 3, 2018, 9:49:14 AM Error executing tool. Publish Service Definition Job ID: je029c3ad0688414592aff0969b65a73v System/PublishingTools.GPServer
SEVERE Apr 3, 2018, 9:49:14 AM Error executing tool. Publish Service Definition Job ID: je029c3ad0688414592aff0969b65a73v : ERROR 001360: Failed to get basic item metadata for service definition upload. ERROR 001362: Failed to extract manifest from uploaded service definition. Failed to execute (Publish Service Definition). System/PublishingTools.GPServer
SEVERE Apr 3, 2018, 9:49:14 AM Failed to extract manifest from uploaded service definition. System/PublishingTools.GPServer
SEVERE Apr 3, 2018, 9:49:14 AM Failed to get basic item metadata for service definition upload.: ERROR: code:404, Could not find resource or operation 'i6d8d0dff-1297-4b6f-a71c-4c90a7463298' on the system. No resource could be found at that address. System/PublishingTools.GPServer
SEVERE Apr 3, 2018, 9:49:11 AM Error executing tool. Publish Service Definition Job ID: j655e3bccf3ee497f876nn76c7c4b4f5b System/PublishingTools.GPServer
SEVERE Apr 3, 2018, 9:49:11 AM Error executing tool. Publish Service Definition Job ID: j655e3bccf3ee497f876nn76c7c4b4f5b : ERROR 001360: Failed to get basic item metadata for service definition upload. ERROR 001362: Failed to extract manifest from uploaded service definition. Failed to execute (Publish Service Definition). System/PublishingTools.GPServer
SEVERE Apr 3, 2018, 9:49:11 AM Failed to extract manifest from uploaded service definition. System/PublishingTools.GPServer
SEVERE Apr 3, 2018, 9:49:11 AM Failed to get basic item metadata for service definition upload.: ERROR: code:404, Could not find resource or operation 'i6d8d0dff-1297-4b6f-a71c-4c90a7463298' on the system. No resource could be found at that address. System/PublishingTools.GPServer
SEVERE Apr 3, 2018, 9:49:07 AM Error executing tool. Publish Service Definition Job ID: jfa1c4220fb794068be1yu7b29713aa55 System/PublishingTools.GPServer
SEVERE Apr 3, 2018, 9:49:07 AM Error executing tool. Publish Service Definition Job ID: jfa1c4220fb794068be1yu7b29713aa55 : ERROR 001487: Failed to update the published service with the server-side data location. Please see the server's log for more details. ERROR 001369: Failed to create the service. Failed to execute (Publish Service Definition). System/PublishingTools.GPServer
SEVERE Apr 3, 2018, 9:49:07 AM Failed to create the service.: Updating the server connection string for layer MY_MAP_DB.MAPUSER_MY_MAP_DB.%tbl_map_AGV_1 failed. Attempted connection string was ENCRYPTED_PASSWORD=MY_ENCRYPTED_PASSWORD;SERVER=AGXXXXXXX;INSTANCE="DSID=e851aec4-00c2-4795-87e6-n6f2652552b7";DBCLIENT=sqlserver;DB_CONNECTION_PROPERTIES=AGXXXXXXX,1031;DATABASE=MY_MAP_DB;USER=MapUser_MY_MAP_DB;AUTHENTICATION_MODE=DBMS. Table name is MY_MAP_DB.MAPUSER_MY_MAP_DB.%tbl_map_1. Please verify the data exists on the server. System/PublishingTools.GPServer
SEVERE Apr 3, 2018, 9:49:07 AM DoAfterSDExtraction failed. System/PublishingTools.GPServer
SEVERE Apr 3, 2018, 9:49:07 AM SwizzleService failed. System/PublishingTools.GPServer
SEVERE Apr 3, 2018, 9:48:42 AM Error executing tool. Publish Service Definition Job ID: j16h4ce5d483946b29e28e20j2360c3d7 System/PublishingTools.GPServer
SEVERE Apr 3, 2018, 9:48:42 AM Error executing tool. Publish Service Definition Job ID: j16h4ce5d483946b29e28e20j2360c3d7 : ERROR 001360: Failed to get basic item metadata for service definition upload. ERROR 001362: Failed to extract manifest from uploaded service definition. Failed to execute (Publish Service Definition). System/PublishingTools.GPServer
SEVERE Apr 3, 2018, 9:48:42 AM Failed to extract manifest from uploaded service definition. System/PublishingTools.GPServer
SEVERE Apr 3, 2018, 9:48:42 AM Failed to get basic item metadata for service definition upload.: ERROR: code:404, Could not find resource or operation 'i8baadf8d-6f3f-44b3-8de9-814e1b6b9315' on the system. No resource could be found at that address. System/PublishingTools.GPServer
SEVERE Apr 3, 2018, 9:48:38 AM Error executing tool. Publish Service Definition Job ID: ja20l40af8ee489fe9146479e8153a213 System/PublishingTools.GPServer
SEVERE Apr 3, 2018, 9:48:38 AM Error executing tool. Publish Service Definition Job ID: ja20l40af8ee489fe9146479e8153a213 : ERROR 001487: Failed to update the published service with the server-side data location. Please see the server's log for more details. ERROR 001369: Failed to create the service. Failed to execute (Publish Service Definition). System/PublishingTools.GPServer
SEVERE Apr 3, 2018, 9:48:38 AM Failed to create the service.: Updating the server connection string for layer MY_MAP_DB.MAPUSER_MY_MAP_DB.%tbl_map_AGV_1 failed. Attempted connection string was ENCRYPTED_PASSWORD=MY_ENCRYPTED_PASSWORD;SERVER=AGXXXXXXX;INSTANCE="DSID=e851aec4-00c2-4795-87e6-n6f2652552b7";DBCLIENT=sqlserver;DB_CONNECTION_PROPERTIES=AGXXXXXXX,1031;DATABASE=MY_MAP_DB;USER=MapUser_MY_MAP_DB;AUTHENTICATION_MODE=DBMS. Table name is MY_MAP_DB.MAPUSER_MY_MAP_DB.%tbl_map_1. Please verify the data exists on the server. System/PublishingTools.GPServer
SEVERE Apr 3, 2018, 9:48:38 AM DoAfterSDExtraction failed. System/PublishingTools.GPServer
SEVERE Apr 3, 2018, 9:48:38 AM SwizzleService failed. System/PublishingTools.GPServer
SEVERE Apr 3, 2018, 9:47:48 AM Error executing tool. Publish Service Definition Job ID: jd42c6ele01384ffca5089facv0e0fd01 System/PublishingTools.GPServer
SEVERE Apr 3, 2018, 9:47:48 AM Error executing tool. Publish Service Definition Job ID: jd42c6ele01384ffca5089facv0e0fd01 : ERROR 001487: Failed to update the published service with the server-side data location. Please see the server's log for more details. ERROR 001369: Failed to create the service. Failed to execute (Publish Service Definition). System/PublishingTools.GPServer
SEVERE Apr 3, 2018, 9:47:48 AM Failed to create the service.: Updating the server connection string for layer MY_MAP_DB.MAPUSER_MY_MAP_DB.%tbl_map_AGV_1 failed. Attempted connection string was ENCRYPTED_PASSWORD=MY_ENCRYPTED_PASSWORD;SERVER=AGXXXXXXX;INSTANCE="DSID=e851aec4-00c2-4795-87e6-n6f2652552b7";DBCLIENT=sqlserver;DB_CONNECTION_PROPERTIES=AGXXXXXXX,1031;DATABASE=MY_MAP_DB;USER=MapUser_MY_MAP_DB;AUTHENTICATION_MODE=DBMS. Table name is MY_MAP_DB.MAPUSER_MY_MAP_DB.%tbl_map_1. Please verify the data exists on the server. System/PublishingTools.GPServer
SEVERE Apr 3, 2018, 9:47:48 AM DoAfterSDExtraction failed. System/PublishingTools.GPServer
SEVERE Apr 3, 2018, 9:47:48 AM SwizzleService failed. System/PublishingTools.GPServer
WARNING Apr 3, 2018, 9:47:34 AM Unable to process request. json Rest
WARNING Apr 3, 2018, 9:47:20 AM Unable to process request. json Rest
SEVERE Apr 3, 2018, 9:47:06 AM Error executing tool. PublishServiceDefinition Job ID: j72e65h16b9c2402e910sep8cfc6c152n System/PublishingTools.GPServer
SEVERE Apr 3, 2018, 9:47:06 AM Error executing tool. PublishServiceDefinition Job ID: j72e65h16b9c2402e910sep8cfc6c152n : ERROR 001487: Failed to update the published service with the server-side data location. Please see the server's log for more details. ERROR 001369: Failed to create the service. Failed to execute (Publish Service Definition). System/PublishingTools.GPServer
SEVERE Apr 3, 2018, 9:47:06 AM Failed to create the service.: Updating the server connection string for layer MY_MAP_DB.MAPUSER_MY_MAP_DB.%tbl_map_AGV_1 failed. Attempted connection string was ENCRYPTED_PASSWORD=MY_ENCRYPTED_PASSWORD;SERVER=AGXXXXXXX;INSTANCE="DSID=e851aec4-00c2-4795-87e6-n6f2652552b7";DBCLIENT=sqlserver;DB_CONNECTION_PROPERTIES=AGXXXXXXX,1031;DATABASE=MY_MAP_DB;USER=MapUser_MY_MAP_DB;AUTHENTICATION_MODE=DBMS. Table name is MY_MAP_DB.MAPUSER_MY_MAP_DB.%tbl_map_1. Please verify the data exists on the server. System/PublishingTools.GPServer
SEVERE Apr 3, 2018, 9:47:06 AM DoAfterSDExtraction failed. System/PublishingTools.GPServer
SEVERE Apr 3, 2018, 9:47:06 AM SwizzleService failed. System/PublishingTools.GPServer
WARNING Apr 3, 2018, 9:43:58 AM Unable to process request. json Rest
WARNING Apr 3, 2018, 9:42:43 AM Unable to process request. json Rest
WARNING Apr 3, 2018, 9:41:09 AM Unable to process request. json Rest
WARNING Apr 3, 2018, 9:39:52 AM Unable to process request. json Rest
WARNING Apr 3, 2018, 9:39:24 AM Unable to process request. json Rest
WARNING Apr 3, 2018, 9:36:08 AM Unable to process request. json Rest
SEVERE Apr 3, 2018, 9:29:14 AM Error executing tool. PublishServiceDefinition Job ID: jc5be0a81lb174eca8e2820fx0eda233k System/PublishingTools.GPServer
SEVERE Apr 3, 2018, 9:29:14 AM Error executing tool. PublishServiceDefinition Job ID: jc5be0a81lb174eca8e2820fx0eda233k : ERROR 001487: Failed to update the published service with the server-side data location. Please see the server's log for more details. ERROR 001369: Failed to create the service. Failed to execute (Publish Service Definition). System/PublishingTools.GPServer
SEVERE Apr 3, 2018, 9:29:14 AM Failed to create the service.: Updating the server connection string for layer MY_MAP_DB.MAPUSER_MY_MAP_DB.%tbl_map_AGV_1 failed. Attempted connection string was ENCRYPTED_PASSWORD=MY_ENCRYPTED_PASSWORD;SERVER=AGXXXXXXX;INSTANCE="DSID=e851aec4-00c2-4795-87e6-n6f2652552b7";DBCLIENT=sqlserver;DB_CONNECTION_PROPERTIES=AGXXXXXXX,1031;DATABASE=MY_MAP_DB;USER=MapUser_MY_MAP_DB;AUTHENTICATION_MODE=DBMS. Table name is MY_MAP_DB.MAPUSER_MY_MAP_DB.%tbl_map_1. Please verify the data exists on the server. System/PublishingTools.GPServer
SEVERE Apr 3, 2018, 9:29:14 AM DoAfterSDExtraction failed. System/PublishingTools.GPServer
SEVERE Apr 3, 2018, 9:29:14 AM SwizzleService failed. System/PublishingTools.GPServer
Also, the GP error output:
Executing: UploadServiceDefinition "C:\Users\me\AppData\Local\ESRI\Desktop10.6\Staging\arcgis on gisserver_6443 (admin)\TEST_SERVICE_3.sd" "C:\Users\me\AppData\Roaming\Esri\Desktop10.6\ArcCatalog\arcgis on gisserver_6443 (admin)" # # FROM_SERVICE_DEFINITION # STARTED USE_DEFINITION NO_SHARE_ONLINE PRIVATE NO_SHARE_ORGANIZATION #
Start Time: Tue Apr 3 09:47:04 2018
Submitted.
Executing...
Failed.
ERROR: the server's publishing job didn't succeed.
Failed to execute (UploadServiceDefinition).
Failed at Tue Apr 3 09:47:06 2018 (Elapsed Time: 2.29 seconds)
And, some info on my set-up:
- ArcGIS Server 10.6
- ArcGIS Server Web Adaptor 10.6
- Windows Server 2012 r2
- MS SQL Server 2012 r2
- 64 gb memory (usually, about 30 free)
- CPU cycles look good
Ever since upgrading to 10.6, the stability I enjoyed with 10.3 has gone down the tube - it's very frustrating... from McAfee conflicts to difficulties in publishing and more! I hope Esri patches this, soon! As I understand, some very deep things under the hood changed in 10.6; maybe it's rearing its ugly head now.