Performance issues - can't (or barely can) reproject 300 million points in ArcGIS10

766
2
02-10-2011 06:39 PM
AndyRitchie
Occasional Contributor
I have a pretty big raster that's currently gridded in NAD83 State Plane feet that I wanted to re-grid as accurately as possible in the corresponding NAD83 UTM zone, so I exported the grid as points to a geodatabase, and reprojected from State Plane to UTM into a shapefile. The point file is 306,660,952 points, 16 bit integer values.

I started this about four hours ago. geoprocessor went through the % complete bar pretty quickly, but it's been stuck at 100% and it WAS slowly writing output for the last 3.5 hours or more... but in the last few minutes it appears that it stuck at 100% and stopped writing output. The file sizes are:

.dbf - 1,268,832 KB
.shp - 2,089,840 KB
.shx - 597,098 KB

Oh. It just finished.

Anyway, I quickly became frustrated with the write speed, so I started to perform the same task in parallel using gdal (1.8) and cs2cs (4.7.1). In a fraction of the time (about 43 minutes), I was able to export the same raster to an XYZ file using gdal_translate (1.8), and convert it to UTM using CS2CS (4.7.1). on the same machine. The write speed of Arc, and the cpu overhead, didn't change appreciably while I was doing this.

For the gdal/cs2csthe raster started out as a 211 MB geotif, exported to XYZ as a 5.75 GB XYZ (integer feet), and transformed into a 8.89 GB UTM xyz.

Food for thought.

(WIN7x64 Dual 6-core CPUs (Xeon X5650 @ 2.67 GHz)/ 12 GB RAM / using different 2 TB SATA 3 read/write drives)
0 Kudos
2 Replies
KyleShannon
New Contributor III
Next time, use gdalwarp, it does all of that together and the -multi flag uses parallel processing.
0 Kudos
ChrisSnyder
Regular Contributor III
No doubt gdal is faster for many things, but...

1) Why not just project it in raster format? Since there is no datum transformation, I don't (think?) you would  get any meaningfully added precision by converting to point format and projecting that.

2) You listed the .shp file size as being 2,089,840 MB, which is basically the 32-bit 2.1GB file size limit... Are you sure ALL your points made it to the shapefile?
0 Kudos