Raster mosaic datasets: Overview Quality control

NicholasO_Connor · ‎05-05-2016

Dear ESRI Support and GIS Users.

Can anyone help me with how best to get good quality overviews? I know this is a little subjective…my case:

I create the Raster Mosaic Dataset in a fGDB and add the following tifs to it.

200 tifs, each tif 20MB in size, all tifs fit together like squares on a chessboard

Each tif:

Number of bands: 3

Cell size 1.25 x 1.25

Format: TIFF

Source Type: Thematic

Pixel Type: unsigned integer

Pixel Depth: 8 bit

NoData Value 256,256,256

ColourMap: absent

Pyramids: absent

Compression: LWZ

TOOL PARAMETERS

DefineOverviews

Overview Tile Parameters

Pixel size: 1.25

Number of Levels: 12

Rows: 5120

Columns: 5120

Overview Sampling Factor: 2

Overview Image Parameters

Resampling Method: BILINEAR

Compression: LWZ

Calculate item visibility – take Default settings.

RESULTS

Scale 1:8000 –= viewing the original tif, not the overviews – quality good

Scale 1:8000 – 20000 = viewing the overviews – quality poor. I know this is a little subjective – but the image is quite
blurred, hard to read labels…

TESTED:

[1] I have tried changing the Number of Levels and Overview Sampling Factor

[2] I have tried changing the Resampling Method and Compression

Nothing seems to help.

QUESTIONS

Is the problem in the original tifs? For example the source Type or Compression used?

Is the problem in how I define the Overview Generation?

Feedback welcome and appreciated.

PeterBecker · ‎05-05-2016

Try the following:

Option 1:

Set the sampling factor to Nearest - This may make the images look much sharper, but I think you will find the images too grainy.

Option 2:

Change the overview based pixel size to 2.25m and set 'Force Overview Sampling Tiles' to true. This will change the factor at which the pyramids are created. This can have a positive effect when accessing the imagery in webmaps. For more details see https://blogs.esri.com/esri/apl/2014/06/24/designing-and-optimizing-image-services-for-high-performa...

Option 3:

Instead of creating overviews generate tile cache. You will need to use the 'Manage Tile Cache' Tool or use http://esriurl.com/RasterTileCacheTools. Generate the tile cache using ArcGIS Online schema and [2.388...] pixelsize. Set the MaxSource Cell size to 2.5. What happens here is that the tile cache is generated. The pyramids for this are created with a gaussian sampling which leads to better quality overview. Once the tile cache has been generated add it back to the mosaic dataset (its a raster dataset) and use it as the overview. Note if you are not using the capabilities of image services for more advanced metadata and processing you could cache it form a lower resolution and just use tile cache.

NicholasO_Connor · ‎05-06-2016

Hello Peter

Thank you very much for your replay – greatly appreciated. Sorry I should have been a little more clear.

The Raster Mosiac Datasets are being accessed only by ArcMap Users connecting to SDE 10.2.2 (Oracle 11.2) – with the actual tifs and overviews stored on the file system. I only generate overviews, I do not create any pyramids. Performance is excellent – the only issue is with the overview quality from certain tifs we have delivered. An example being one in the details above.

Question:

Do you feel that the Source Type in the original tif here Thematic (or Generic) plays a role combined with the Resampling Method BILINEAR and Compression as LWZ?

...Sorry I assume it is difficult to answer this without actually testing the data.

However I will also follow-up too on some of your suggestions to see if we can improve our workflow.

Kind Regards

PeterBecker · ‎05-06-2016

Nicholas

Source Type will not make a difference. The resampling type used in the overviews will make a difference. Using Nearest the overviews will be sharper, but may get to look grainy at small scales. Bilinear will be smoother. Cubic will give results that are a bit sharper than Bilinear. Suggest you try these first.

NicholasO_Connor · ‎05-12-2016

Hello Peter

Once again thank you for your feedback so far changing the resampling type has made no difference.

Some more information that maybe helpful. Below shows the details of the tifs from 2015 which have poor overviews generated. The ones we had delivered in 2011 the overviews once generated are very good. The extent of the tifs from 2011 and 2015 is the same.

The difference being in Source Type, NoData Value, Compression, Mensuration and Statistics. Could any of these play a role in the quality of the overview generation?

2015 (poor overviews)

Each tif:

Number of bands: 3

Cell size 1.25 x 1.25

Format: TIFF

Source Type: Thematic

Pixel Type: unsigned integer

Pixel Depth: 8 bit

NoData Value 256,256,256

ColourMap: absent

Pyramids: absent

Compression: LWZ

Mensuration: None

Statistices : not calculated

2011 (good overviews)

Each tif:

Number of bands: 3

Cell size 1.25 x 1.25

Format: TIFF

Source Type: Generic

Pixel Type: unsigned integer

Pixel Depth: 8 bit

NoData Value

ColourMap: absent

Pyramids: absent

Compression: PACKBITS

Mensuration: Basic

Statistices : calculated

PeterBecker · ‎05-12-2016

Nothing stands out as incorrect.The 2015 are defined as thematic, but if when you create the overviews you define the sampling as bilinear so there should be no issue there.

Can you include a screen shot to indicate what you mean by poor and good overviews. If I can see the artifacts you are seeing I may be able to determine the cause.

NicholasO_Connor · ‎05-13-2016

Hello Peter

I have uploaded two images (copyright © Data: Federal Office of Topography Swisstopo) – as screenshots from
ArcMap at 1:25,000. Both sets of Overviews were generated in the same way but from different year tifs 2011 and 2015. See workflow below with details on how I defined the Overviews:

arcpy.CreateMosaicDataset_management
arcpy.AddRastersToMosaicDataset_management
arcpy.DefineOverviews_management

DefineOverviews x:\test\test.gdb\Raster_1101
x:\test\overviews # "620000 170000 690000 242000" 1.25 # 5120 5120 2
NO_FORCE_OVERVIEW_TILES BILINEAR LZW 80

arcpy.BuildOverviews_management

arcpy.CalculateCellSizeRanges_management

1101_good_1to25k.jpg = good quality overviews (1st picture)

1602_bad_1to25k.jpg = poor quality overviews (2nd Picture)

I hope you can see the difference?

PeterBecker · ‎05-13-2016

Nicholas

The issue is related to the sampling use for the pyramids and overviews. To me it looks like the 2015 images have pyramids that were created using Nearest sampling. I looks to me that the system finds pyramids with the files (which have been created using nearest sampling and is using them). This could have happened if you added the tif files to ArcMap originally. It may have asked to create pyramids and the default is nearest sampling. Please double check the source images do they have .OVR files associated with them. Is so delete. Check in the properties of the rasters (in catalog), does it state the pyramids exist. If you add the rasters to ArcMap does it prompt to generate pyramids? If not then the system has found pyramids. It is possible that if the original data is read only that the pyramids have been put in a cache directory. Try copying and renaming on of the original TIF files and creating overview or pyramids from this but ensure sampling is bilinear.

NicholasO_Connor · ‎05-16-2016

Hello Peter

Thank you once again for following up on my issue. Very much appreciated.

I did the following:

For each 20 tifsI made a copy.... into a seperate Folder with a rename
Executing:CopyRaster x:\test\1602\109M.tif x:\test\1602_copy\2.tif # # 256 NONE NONE # NONE NONE

Executing: AddRastersToMosaicDataset x:\test\fGDB\test.gdb\Raster_1602_copy "Raster Dataset"
UPDATE_CELL_SIZES UPDATE_BOUNDARY NO_OVERVIEWS # 0 1500 # # SUBFOLDERS ALLOW_DUPLICATES NO_PYRAMIDS NO_STATISTICS NO_THUMBNAILS # NO_FORCE_SPATIAL_REFERENCE

Executing: DefineOverviews x:\test\fGDB\test.gdb\Raster_1602_copy x:\test\overviews_quality\1602_copy\overviews # "620000 170000 690000 242000" 1.25 12 5120 5120 2 NO_FORCE_OVERVIEW_TILES BILINEAR LZW 80

Executing: BuildOverviews x:\test\fGDB\test.gdb\Raster_1602_copy # DEFINE_MISSING_TILES GENERATE_OVERVIEWS GENERATE_MISSING_IMAGES REGENERATE_STALE_IMAGES

The result was the same poor quality overview images.

To answer your questions abount the copied raster data:

>>Check in the properties of the rasters (in catalog), does it state the pyramids exist.
No

>>If you add the rasters to ArcMap does it prompt to generate pyramids?
Yes

>>It is possible that if the original data is read only that the pyramids have been put in a cache directory
No

Any further help? Perhaps I could send you directly my tif data for testing?

PeterBecker · ‎05-16-2016

Nicholas

Can you send me some sample data to test out pbecker _at_ esri.com. Else send me a mail an I will provide a share location.