OptimizeRasters

[EmilyWindahl]

This post provides a starting point for users to share questions, answers, and general comments regarding the OptimizeRasters tools and scripts. 

OptimizeRasters is an open-source, community-supported set of tools for accomplishing three tasks: converting raster data to optimized Tiled TIF or MRF files, moving data to cloud storage, and creating optimized raster proxy files. The result is more efficient, scalable, and elastic data access with a lower storage cost.

Run into an issue using OptimizeRasters? Have suggestions for using or improving the tool? Share them here.

[BrianCarson2]

I'm trying to run the Optimize Rasters script to generate a MRF which will reside in S3 from a directory of TIFF images (8-bit unsigned). However, I'm running into a warning which is causing the tool to fail 17 seconds into execution. The warning is as follows:

Executing: OptimizeRasters Imagery_to_MRF_LERC Local Profile " " C:\arcgis\ArcTutor\Raster\Data\Amberg_tif # "Amazon S3" pyadav brianmrftesting brianmrftesting/optimizerasters \\sertao\rasterdrive\MRF_for_AWS_s3\OutputTempFolder_OR \\sertao\rasterdrive\MRF_for_AWS_s3\RasterProxyOutputFolder \\sertao\rasterdrive\MRF_for_AWS_s3\CacheFolder # "Mode mrf;RasterFormatFilter tif,tiff,TIF,TIFF,img,jp2,JP2,IMG,JPG,jpg,sid,SID;ExcludeFilter tmp,rrd,idx,lrc,mrf_cache,pjp,ppng,pft,pzp,ovr,aux.xml,aux,tfw,TFW,pjg;IncludeSubdirectories true;Compression LERC;Quality #;LERCPrecision #;BuildPyramids true;PyramidFactor #;PyramidSampling avg;PyramidCompression LERC;NoDataValue #;BlockSize 512;Scale #;KeepExtension false;Threads 4;GDAL_Translate_UserParameters #"

Start Time: Wed May 30 11:25:44 2018

Running script OptimizeRasters...

log-msg:Using output endpoint> https://brianmrftesting.s3-us-west-1.amazonaws.com/

log-critical:maximum recursion depth exceeded while calling a Python object

log-critical:Unable to initialize the (S3) upload module! Check module setup/credentials. Quitting..

log-status:[Failed!]

Completed script OptimizeRasters...

Succeeded at Wed May 30 11:25:59 2018 (Elapsed Time: 15.86 seconds)

I have checked to ensure that my setup/credentials are correct, but I'm unable to get past the error message.

Thanks,

Brian

[LorneDmitruk2]

Hi,

I've been testing out the MRF format and raster proxies and I have run into issues. I've been using ArcGIS Desktop 10.6.1 and ArcGIS Pro 2.3.2

I started by using OptimizeRasters to create MRF files from the original GeoTIFFs and then copied the files to a networked drive. I then created the raster proxies as a csv file, as shown below:

ObjectID;Raster
1;<MRF_META><CachedSource><Source clone='true'>H:/MRFTest/18933E593082N.mrf</Source></CachedSource><Raster><Size c='3' x='10000' y='10000'/><PageSize c='1' x='512' y='512'/><Compression>LERC</Compression><DataFile>D:/test/18933E593082N.mrf_cache</DataFile><IndexFile>D:/test/18933E593082N.mrf_cache</IndexFile></Raster><Rsets model='uniform' scale='2'/><GeoTags><BoundingBox maxx='19683.00000000' maxy='5931579.00000000' minx='18933.00000000' miny='5930829.00000000'/><Projection>PROJCS['NAD_1983_3TM_114',GEOGCS['NAD83',DATUM['North_American_1983',SPHEROID['GRS 1980',6378137,298.257222101,AUTHORITY['EPSG','7019']],AUTHORITY['EPSG','6269']],PRIMEM['Greenwich',0],UNIT['degree',0.0174532925199433],AUTHORITY['EPSG','4269']],PROJECTION['Transverse_Mercator'],PARAMETER['latitude_of_origin',0],PARAMETER['central_meridian',-114],PARAMETER['scale_factor',0.9999],PARAMETER['false_easting',0],PARAMETER['false_northing',0],UNIT['metre',1,AUTHORITY['EPSG','9001']]]</Projection></GeoTags><Options>V2=ON</Options></MRF_META>
2;<MRF_META><CachedSource><Source>H:/MRFTest/18933E593082N.tif</Source></CachedSource><Raster><Size c='3' x='10000' y='10000'/><PageSize c='1' x='512' y='512'/><Compression>LERC</Compression><DataFile>D:/test/18933E593082N.mrf_cache</DataFile><IndexFile>D:/test/18933E593082N.mrf_cache</IndexFile></Raster><Rsets model='uniform' scale='2'/><GeoTags><BoundingBox maxx='19683.00000000' maxy='5931579.00000000' minx='18933.00000000' miny='5930829.00000000'/><Projection>PROJCS['NAD_1983_3TM_114',GEOGCS['NAD83',DATUM['North_American_1983',SPHEROID['GRS 1980',6378137,298.257222101,AUTHORITY['EPSG','7019']],AUTHORITY['EPSG','6269']],PRIMEM['Greenwich',0],UNIT['degree',0.0174532925199433],AUTHORITY['EPSG','4269']],PROJECTION['Transverse_Mercator'],PARAMETER['latitude_of_origin',0],PARAMETER['central_meridian',-114],PARAMETER['scale_factor',0.9999],PARAMETER['false_easting',0],PARAMETER['false_northing',0],UNIT['metre',1,AUTHORITY['EPSG','9001']]]</Projection></GeoTags><Options>V2=ON</Options></MRF_META>
3;<MRF_META><CachedSource><Source clone='true'>H:/MRFTest/18933E593157N.mrf</Source></CachedSource><Raster><Size c='3' x='10000' y='10000'/><PageSize c='1' x='512' y='512'/><Compression>LERC</Compression><DataFile>D:/test/18933E593157N.mrf_cache</DataFile><IndexFile>D:/test/18933E593157N.mrf_cache</IndexFile></Raster><Rsets model='uniform' scale='2'/><GeoTags><BoundingBox maxx='19683.00000000' maxy='5932329.00000000' minx='18933.00000000' miny='5931579.00000000'/><Projection>PROJCS['NAD_1983_3TM_114',GEOGCS['NAD83',DATUM['North_American_1983',SPHEROID['GRS 1980',6378137,298.257222101,AUTHORITY['EPSG','7019']],AUTHORITY['EPSG','6269']],PRIMEM['Greenwich',0],UNIT['degree',0.0174532925199433],AUTHORITY['EPSG','4269']],PROJECTION['Transverse_Mercator'],PARAMETER['latitude_of_origin',0],PARAMETER['central_meridian',-114],PARAMETER['scale_factor',0.9999],PARAMETER['false_easting',0],PARAMETER['false_northing',0],UNIT['metre',1,AUTHORITY['EPSG','9001']]]</Projection></GeoTags><Options>V2=ON</Options></MRF_META>
4;<MRF_META><CachedSource><Source>H:/MRFTest/18933E593157N.tif</Source></CachedSource><Raster><Size c='3' x='10000' y='10000'/><PageSize c='1' x='512' y='512'/><Compression>LERC</Compression><DataFile>D:/test/18933E593157N.mrf_cache</DataFile><IndexFile>D:/test/18933E593157N.mrf_cache</IndexFile></Raster><Rsets model='uniform' scale='2'/><GeoTags><BoundingBox maxx='19683.00000000' maxy='5932329.00000000' minx='18933.00000000' miny='5931579.00000000'/><Projection>PROJCS['NAD_1983_3TM_114',GEOGCS['NAD83',DATUM['North_American_1983',SPHEROID['GRS 1980',6378137,298.257222101,AUTHORITY['EPSG','7019']],AUTHORITY['EPSG','6269']],PRIMEM['Greenwich',0],UNIT['degree',0.0174532925199433],AUTHORITY['EPSG','4269']],PROJECTION['Transverse_Mercator'],PARAMETER['latitude_of_origin',0],PARAMETER['central_meridian',-114],PARAMETER['scale_factor',0.9999],PARAMETER['false_easting',0],PARAMETER['false_northing',0],UNIT['metre',1,AUTHORITY['EPSG','9001']]]</Projection></GeoTags><Options>V2=ON</Options></MRF_META>
5;<MRF_META><CachedSource><Source clone='true'>H:/MRFTest/18933E593232N.mrf</Source></CachedSource><Raster><Size c='3' x='10000' y='10000'/><PageSize c='1' x='512' y='512'/><Compression>LERC</Compression><DataFile>D:/test/18933E593232N.mrf_cache</DataFile><IndexFile>D:/test/18933E593232N.mrf_cache</IndexFile></Raster><Rsets model='uniform' scale='2'/><GeoTags><BoundingBox maxx='19683.00000000' maxy='5933079.00000000' minx='18933.00000000' miny='5932329.00000000'/><Projection>PROJCS['NAD_1983_3TM_114',GEOGCS['NAD83',DATUM['North_American_1983',SPHEROID['GRS 1980',6378137,298.257222101,AUTHORITY['EPSG','7019']],AUTHORITY['EPSG','6269']],PRIMEM['Greenwich',0],UNIT['degree',0.0174532925199433],AUTHORITY['EPSG','4269']],PROJECTION['Transverse_Mercator'],PARAMETER['latitude_of_origin',0],PARAMETER['central_meridian',-114],PARAMETER['scale_factor',0.9999],PARAMETER['false_easting',0],PARAMETER['false_northing',0],UNIT['metre',1,AUTHORITY['EPSG','9001']]]</Projection></GeoTags><Options>V2=ON</Options></MRF_META>
6;<MRF_META><CachedSource><Source clone='true'>H:/MRFTest/18933E593307N.mrf</Source></CachedSource><Raster><Size c='3' x='10000' y='10000'/><PageSize c='1' x='512' y='512'/><Compression>LERC</Compression><DataFile>D:/test/18933E593307N.mrf_cache</DataFile><IndexFile>D:/test/18933E593307N.mrf_cache</IndexFile></Raster><Rsets model='uniform' scale='2'/><GeoTags><BoundingBox maxx='19683.00000000' maxy='5933829.00000000' minx='18933.00000000' miny='5933079.00000000'/><Projection>PROJCS['NAD_1983_3TM_114',GEOGCS['NAD83',DATUM['North_American_1983',SPHEROID['GRS 1980',6378137,298.257222101,AUTHORITY['EPSG','7019']],AUTHORITY['EPSG','6269']],PRIMEM['Greenwich',0],UNIT['degree',0.0174532925199433],AUTHORITY['EPSG','4269']],PROJECTION['Transverse_Mercator'],PARAMETER['latitude_of_origin',0],PARAMETER['central_meridian',-114],PARAMETER['scale_factor',0.9999],PARAMETER['false_easting',0],PARAMETER['false_northing',0],UNIT['metre',1,AUTHORITY['EPSG','9001']]]</Projection></GeoTags><Options>V2=ON</Options></MRF_META>
7;<MRF_META><CachedSource><Source clone='true'>H:/MRFTest/18933E593382N.mrf</Source></CachedSource><Raster><Size c='3' x='10000' y='10000'/><PageSize c='1' x='512' y='512'/><Compression>LERC</Compression><DataFile>D:/test/18933E593382N.mrf_cache</DataFile><IndexFile>D:/test/18933E593382N.mrf_cache</IndexFile></Raster><Rsets model='uniform' scale='2'/><GeoTags><BoundingBox maxx='19683.00000000' maxy='5934579.00000000' minx='18933.00000000' miny='5933829.00000000'/><Projection>PROJCS['NAD_1983_3TM_114',GEOGCS['NAD83',DATUM['North_American_1983',SPHEROID['GRS 1980',6378137,298.257222101,AUTHORITY['EPSG','7019']],AUTHORITY['EPSG','6269']],PRIMEM['Greenwich',0],UNIT['degree',0.0174532925199433],AUTHORITY['EPSG','4269']],PROJECTION['Transverse_Mercator'],PARAMETER['latitude_of_origin',0],PARAMETER['central_meridian',-114],PARAMETER['scale_factor',0.9999],PARAMETER['false_easting',0],PARAMETER['false_northing',0],UNIT['metre',1,AUTHORITY['EPSG','9001']]]</Projection></GeoTags><Options>V2=ON</Options></MRF_META>‍‍‍‍‍‍‍‍

When I tried to load the csv file into fGDB mosaic data set as table the process failed with the following error messages:

 2019-05-15T15:29:20.325: Error: 8004206c: Failures reported while processing specified command item. [1]
 2019-05-15T15:29:20.325: Error: 80042019: Chained Error ID: 0X80042019
 2019-05-15T15:29:20.325: Error: 80042019: Could not build mosaic dataset item. [ID: 1, URI: 'D:\Projects\SlimWebMap\2017_MRF_Proxies.csv|1']
 2019-05-15T15:29:20.325: Error: 80040207: An invalid SQL statement was used.‍‍‍‍

I got around this problem by loading the csv into the fGDB, and the raster proxy files loaded without a problem.

Here's the issue. When I add the mosaic data set to a map in ArcGIS Pro, the foot prints displayed, but no image, when I zoomed to the source resolution I got an image displayed on the screen and then when I started to zoom out ArcGIS Pro crashes. Then when trying to reopen the project, unless I pause the drawing ArcGIS will crash. I got a similar result in ArcMap. So I ran analyse on the mosaic data set and got the following report back indicating the paths to the files can't be accessed:

This seems strange as I have full read/write permissions on the folders where the piles are stored.

Next I ran export mosaic dataset paths and everything appears to be okay:

Last thing I did was to check the cache folder and I did see that some mrf_cache files had been created.

Now when I looked at the footprint table in ArcMap and open the information for the raster field, the preview is displayed and when I check the properties for the path everything looks good as near as I can tell. 

So any thoughts as to why the mosaic data set is causing both ArcMap and ArcGIS Pro to crash? 

MRF‌ #OptimizeRasters #Raster Proxy File mosaic datset‌ 

[AbhijitDoshi]

Lorne,

There might be few things that might have gone wrong

• In your case the mrf_cache path for multiple files is same.
• Can you paste the converted MRF string ( open in notepad can copy), it seems that there is a mismatch in some of the properties, i need to verify it
• the CSV has tif as well as MRF, ideally you should keep them separate

[LorneDmitruk2]

Hi Abhijit,

The cache path is pointing to the table I created in the fGDB, not the csv file as I was unable to create the mosaic dataset from the csv file.

Here's a sample mrf file, it's the first file:18933E593082N.mrf

<MRF_META>
<Raster>
<Size x="10000" y="10000" c="3" />
<PageSize x="512" y="512" c="3" />
<Compression>JPEG</Compression>
<Quality>95</Quality>
</Raster>
<GeoTags>
<BoundingBox minx="18933.00000000" miny="5930829.00000000" maxx="19683.00000000" maxy="5931579.00000000" />
<Projection>PROJCS["NAD_1983_3TM_114",GEOGCS["NAD83",DATUM["North_American_1983",SPHEROID["GRS 1980",6378137,298.257222101,AUTHORITY["EPSG","7019"]],AUTHORITY["EPSG","6269"]],PRIMEM["Greenwich",0],UNIT["degree",0.0174532925199433],AUTHORITY["EPSG","4269"]],PROJECTION["Transverse_Mercator"],PARAMETER["latitude_of_origin",0],PARAMETER["central_meridian",-114],PARAMETER["scale_factor",0.9999],PARAMETER["false_easting",0],PARAMETER["false_northing",0],UNIT["metre",1,AUTHORITY["EPSG","9001"]]]</Projection>
</GeoTags>
<Rsets model="uniform" scale="2" />
</MRF_META>

Cheers

[AbhijitDoshi]

Hi Lorne, 

below is the issue number for this 

https://github.com/Esri/OptimizeRasters/issues/88

It would be fixed in 2 -3 days on github. In the mean time you can create Raster proxies by edit the compression in the Create Raster Proxy template from LERC to JPEG and it should fix the issue for you 

[LorneDmitruk2]

Hi Abhijit,

I downloaded the latest update from Github and the proxies are now working

without crashing ArcGIS Pro.

Cheers

[MikeSchonlau]

Hi. I've used Optimize Rasters successfully in the past to shrink many sets of orthoimagery tiles. I tried running the process against a new set of tiles (1" GSD RGB jpg's w/jgw's) and it actually resulted in TIF files that are larger than the original jpg's. My parameters are shown below. Any thoughts on if I should have modified something in the parameters? I used OptimizeRasters v. 2.1.

Thanks

Mike

[TilmannSteinmetz2]

We would like to run OptimizeRasters on our supercomputer, in a Linux environment. How does that get installed and configured?

I've successfully used it for 'manual' conversion of large volumes of NetCDF (nc) files to MRF, including LERC compression, using an install on my Windows PC. Here, I use a Python2.7 which came with my ArcMap.

The installation instructions on github provide a link to an .exe installer and nothing for Linux, although, under the Requirements section, it says that it can be used on Linux. How?
(see below)

There is the "Lambda" function installer available under https://github.com/Esri/OptimizeRasters/blob/master/Setup/Lambda/OptimizeRastersLambdaFX.zip - would I be using the contents of this zip file? is it as easy as unzipping it on my Linux target and scheduling the process to run with my configuration?

Thank you for any help on this -

Tilmann

@EmilyWindahl @AbhijitDoshi 

[JamesTurrin]

I'm using Optimize Rasters to convert 4-band (R,G,B,NIR) aerial photography tiffs to mrfs. When I view the resulting mrfs in ArcGIS Pro, the 4th band is being interpreted as an alpha band, rather than NIR. Has anyone else encountered this problem? Is there a way to prevent this behavior? I've tried entering gdal translate user parameters into Optimize Rasters to over-ride default color interpretation during conversion. Examples of some of the gdal translate parametes I've tried are:

-colorinterp alpha

-colorinterp red,green,blue,alpah

-colorinterp_4 alpha

-co PHOTOMETRIC=RGB

-co COLORINTERP=ALPHA

-co COLORINTERP_4=ALPHA

Nothing seems to work. 

I know I can set the transparency for a raster through Symbology, but this isn't a feasible approach when I have many thousands of rasters to convert.

Any advice is welcome.