Added scale to image service cache, but tile generation refuses to complete

[deleted-user-zpcJw1u0IXiO]

I have an image service I've successfully cached previously. It had four empty scales 1:50, 1:100, 1:250, and 1:500 I created in order that the Geocortex viewer would permit zooming closer than the cached levels.

I have found need to populate the cache at 1:500, and was able to get perhaps a third of the service extent to do so. The table below shows the Cache Status where it is stuck, despite that Level 17 is currently underway for the dozenth or so attempt. 

Level	Scale	Size	Expected Tiles	Completed Tiles	Percent	In Progress
0 1,000,000 0.18 MB 6 6 100 1 500,000 0.61 MB 18 18 100 2 250,000 2 MB 55 55 100 3 125,000 4.36 MB 210 180 85.71 4 100,000 6.76 MB 312 275 88.14 5 64,000 9.87 MB 760 624 82.11 6 50,000 13.92 MB 1,200 960 80 7 32,000 27.75 MB 2,886 2,356 81.64 8 25,000 39.45 MB 4,653 3,744 80.46 9 16,000 65.76 MB 11,242 9,150 81.39 10 10,000 162.53 MB 28,405 23,183 81.62 11 8,000 228.27 MB 44,352 36,058 81.3 12 5,000 471.78 MB 113,390 91,868 81.02 13 4,000 742.54 MB 177,408 143,395 80.83 14 2,500 1.91 GB 453,560 366,520 80.81 15 2,000 2.79 GB 708,400 572,390 80.8 16 1,000 11.01 GB 2,832,450 2,285,258 80.68 17 500 4.47 GB 11,327,500 2,583,061 22.8 18 250 0 MB 45,310,000 0 0 19 100 0 MB 283,126,752 0 0 20 50 0 MB 1,132,434,765 0 0

The trouble is the tile count never changes, despite my running Manage Map Server Cache Tiles with each attempt using an Update Extent shapefile covering a progressively smaller fraction of the area needed. So after three hours on an extent of just 30 sq. miles, the job details URL reports 0% completion on each line, with a different Estimated Time Remaining that grows larger and larger before dropping back to begin growing larger again.

The Logs in AGS Manager report what I suspect may be the reason for no tile creation:

SEVERE

Aug 15, 2016, 1:21:23 PM

Error exporting image Wait time of the request to the service 'public/swoop2015_mosaic.ImageServer' has expired.

Rest

SEVERE

Aug 15, 2016, 1:21:23 PM

Unable to process request. No instances for 'public/swoop2015_mosaic.ImageServer' were available for 60.006 seconds. Wait timeout exceeded.

public/swoop2015_mosaic.ImageServer

SEVERE

Aug 15, 2016, 1:20:41 PM

Error exporting image Error handling service request :Processing request took longer than the usage timeout for service 'public/swoop2015_mosaic.ImageServer'.

Rest

SEVERE

Aug 15, 2016, 1:20:38 PM

Processing request took longer than the usage timeout for service 'public/swoop2015_mosaic.ImageServer'. Server request timed out. Check that the usage timeout is appropriately configured for such requests.

public/swoop2015_mosaic.ImageServer

I don't know what to try next. I have already restarted the image service with no change in behaviour.

Short of contacting Esri tech support, what should I try to correct this issue?

Thanks,

Justin

[JenniferMcCall4]

Could it be that another process is holding on to the instances need to cache this?  Have you tried re-starting the ArcGIS Server service in windows?

What does your CPU sit at over the 3 hours?

[deleted-user-zpcJw1u0IXiO]

Thank you Jennifer. I didn't notice the CPU usage before cancelling the job shortly after my post. How can I determine what other process might be holding on to the instance, which I assume you mean to be the image service? Perhaps it might be that I publish the same service dynamically to handle the larger, non-cached scales. Should I stop that service to see if that affects things?

To restart ArcGIS Server service in windows, do you mean in Manager, or in Windows Services?

[deleted-user-zpcJw1u0IXiO]

I've launched the MMSCT again, after stopping the dynamic image service, and CPU usage ranges between 20% and 65%, but the server has active map users. I've still not restarted any services.

[JenniferMcCall4]

Is this percentage where you expect the CPU to sit while caching, or while there is no caching and active users?

[JenniferMcCall4]

Hi Justin,

I was referring to the ArcGIS Server service within Windows Services.  Stopping and then starting it should remove any potential locks from other services or even from processes previously run on this image service.  After starting the windows service again, wait a minute or two until ArcGIS Server and all arcgis services (ex; the image service) are back up and running.  Then try to cache just level 17 again and see if it makes a difference.

When you publish the service, are you checking the box to manually create the cache after the service is published?

Another source of information you could check is the Status.gdb within the cache folder.  It will only be visible through Windows Explorer, so copy this gdb to another location (ex; your Desktop) and then you can open the GDB and feature classes within ArcMap.  This may provide more information on what might be happening with your service.  I would recommend copying the GDB after you have tried to re-cache level 17 so any information about the job will be populated in the feature classes.

[deleted-user-zpcJw1u0IXiO]

okay thank you. I'll try restarting as you described. I realized just moments ago the mistake of turning off the dynamic image service (as I said I did), because the cached service needs the dynamic one as its source. And AGS logs screamed at me for the source service not being found. Silly me.

I looked at status.gdb last week, and that's how I knew which areas remained uncached, and used that to create polygons to use as the small update extents.

[deleted-user-zpcJw1u0IXiO]

yes, the service was already set for manual cache creation. 

I did restart the Windows ArcGIS Server service, but neglected to cancel caching before doing so. I clicked Cancel Job in the Job Details page, but after 10 minutes, it still says esriJobCancelling, so I don't know what more to do to make it actually stop so that I may restart it

[JenniferMcCall4]

Stopping the ArcGIS Server service should kill the job, not leave it cancelling.  How long did you wait between stopping it and starting it up again.  It's beneficial to wait at least 30 seconds.

[deleted-user-zpcJw1u0IXiO]

I had left it only a second or two before starting it again. So I've stopped it once more, this time waiting more than 30 seconds, but with all services back up, the job still reports as being in progress (in Manager Cache Status).