POST
|
Thank you for your response. Recommendations: Based on your response and your script, I have the following recommendations: Before executing Focal Statistics, specify arcpy.env.cellSize to your zone raster (10m) arcpy.env.snapRaster to your value raster Before executing Zonal Statistics or Zonal Statistics as Table, you can save the output from Focal Statistics on disc, to check how much time it takes to create this raster dataset. You can also check the cell size to make sure that it is 10m (OPTIONAL). You can specify the output from Focal Statistics as value raster to Zonal Statistics or Zonal Statistics as Table. At this point, the output from Focal Statistics, if not saved earlier will be saved at 10m cell size, which should take substantially less time. Comments: I have also put together some comments below to explain my recommendations. 1. Cell size: It looks like you have not explicitly specified the environment cell size. The Zonal Statistics or Zonal Statistics as Table tool is executing with default cell size, which is Maximum of inputs. So, your Zonal Statistics output cell size is 10m. The Zonal Statistics as Table output is also calculated based on 10m cell size. 2. Focal Statistics: Since there is no analysis environment cell size specified, Focal Statistics is calculated with 0.16m cell size, resulting in a large raster. You are not immediately saving this large raster, and passing the output from Focal Statistics to Zonal Statistics or Zonal Statistics as Table. Since you are not saving the Focal Statistics output, it feels like Focal Statistics is executing quickly. In reality, Focal Statistics is returning a raster function, which stores the analytical model only, but not a raster dataset on disc. If you are not interested in the Focal Statistics output as the final result, you can use this raster function output to another tool/raster function. However, depending on the nature of the subsequent operation, the raster function will be extended with additional analytical operation (function chain) or first a dataset will be created, and then used as input in the analytical tool. Usually, the local and focal operations create function chain, where a raster dataset can be created by saving the final output at the end of the analysis, which makes the execution faster. If a raster function or a function chain is used as input to zonal or global tools/raster functions, a dataset will first be created internally, before executing that in the tool. In this case, the local/focal operation is not chained together with the zonal/global operation. 3. Zonal Statistics as Table: You are specifying the output of Focal Statistics as input to Zonal Statistics or Zonal Statistics as Table. Due to the nature of the operation (zonal), the tool requires a raster dataset as value. So, it triggers the save internally, which is creating a large raster with 0.16m cell size. This raster is then internally resampled to a 10m value raster for performing the analysis with a 10m zone raster. All this extra work in Zonal Statistics as Table is probably taking the additional time, which can be avoided by specifying analysis cell size and snap raster before performing Focal Statistics. It is not very clear yet why the tool is executing faster in Pro compared to your standalone python script. I am hoping by specifying the cell size and snap raster explicitly, you will see a performance improvement. We can then take a closer look at your workflow in Pro to understand the source of the execution time difference. Please let me know if this works with your dataset, and if you have additional questions. Thanks, Sarmistha
... View more
03-04-2022
08:00 PM
|
0
|
1
|
1659
|
POST
|
Hi Michael, thank you for reporting this. I am a Senior Product Engineer from the Spatial Analyst team, I have a few questions for you: What is the environment Cell size within Pro and the standalone Python environment? I am assuming you are setting Parallel processing factor to 75% as you mentioned in your comment for both within Pro and Python. Are you setting it as a percentage or as a number of processors. Have you saved the output of Focal statistics before using it as input in Zonal Statistics tools in Pro? If so, what is the output format? And what is the cell size of this output? Are you setting the Output Coordinate System environment in either Pro or Python? If so, is it same as the value_raster or something else?
... View more
03-03-2022
11:02 AM
|
0
|
1
|
675
|
POST
|
Hi Luba, We have executed Zonal Statistics as Table tool successfully with a large number of zones (million+), and with large rasters. So, the problem could be related to the specific data you are using, or the specific version of the software. So, I would like to take a closer look at your use case, where Zonal statistics as Table is failing with ERROR 999998. Zonal Statistics as Table tool supports overlapping polygons from ArcGIS Pro version 2.5 onwards, so it should not be a problem. So, there is no need to convert the zone features to a raster to execute the tool. Sometimes, this kind of problem can happen with mosaic or image service, due to the limit set on a mosaic or on the server. If you are using a mosaic dataset, please check the mosaic dataset property from the Catalog, and check what is the Maximum Number of Rasters Per Mosaic (under Image Properties). Please make sure, that this number is equal to or greater than the number of tiles you are trying to analyze. If you are using an image service, you could be hitting a limit by the server of how many cells you can process. Can you please try a larger cell size for the full analysis extent (same as your zone polygons)? By adjusting extent, and cell size, you can avoid hitting the server limitation, if there is any. I will follow up with you requesting additional information, if the problem still persists. Thanks, Sarmistha
... View more
02-04-2022
05:26 PM
|
0
|
0
|
924
|
POST
|
Hi @Robert_Greene There are several tools available in the Spatial Analyst extension to perform this analysis. Extract Multi Values to Points tool will append your input points, and the extracted cell value will be added as a new field to the input points. Sample tool will create a table with the point location and the cell values. Extract Values to Points tool will create a new point feature with the extracted cell values. 2. From the problem you stated above, I am assuming you have one raster indicating wind speed (magnitude, not direction). The solution for wind direction will be different. You can use the Buffer tool, and the Zonal Statistics tool or the Zonal Statistics as Table tool to solve this problem with wind speed. First, create the buffer around the point. Use the Buffer tool to create a buffer with the desired distance around the point, and use the buffer output as zone, and wind speed as value raster in one of the following tools: Zonal Statistics tool will create a raster output, with the mean wind speed. Zonal Statistics as Table tool will create a table output, with the mean wind speed. Zonal Statistics as Table tool can also calculate statistics on overlapping buffer zones, if any, on ArcGIS Pro v2.5 or later. Please note that the points and subsequently buffer polygons must have unique IDs, so that the mean is computed for each location. Note: If you want to calculate the mean of wind direction, you should calculate Circular Mean, which will be available in a future release. Hope this helps. Thanks, Sarmistha
... View more
01-12-2022
10:46 AM
|
2
|
0
|
1125
|
IDEA
|
Hi Omer, my understanding of this problem is that you want to use the output of a mathematical expression (also known as map algebraic expression in raster analysis) as the value raster input to the Zonal Statistics tool. Technically, it is a two-steps process. The first step is where the mathematical expression is executed, and the second step is where the zonal operation is executed. Using ArcPy: In ArcPy, you can write it using 2 expressions or as one complex expression as follows: import arcpy from arcpy.sa import * in_zone = Raster('zone.tif') in_ras1 = Raster('ras1.tif') in_ras2 = Raster('ras2.tif') # Approach 1: Using two expressions. However, in a standalone script the map algebraic expression will be executed when Zonal Statistics is executed. out_mapalgebra_ras = in_ras1 + in_ras2 out_zs_ras = ZonalStatistics(in_zone, 'Value', out_mapalgebra_ras) # Approach 2: Using map algebraic expression as the value raster input out_zs_complex_ras = ZonalStatistics(in_zone, 'Value', (in_ras1 + in_ras2)) # Approach 3: Creating input raster objects in the complex expression while executing Zonal Statistics out_zs_complex_ras2 = ZonalStatistics(Raster('zone.tif'), 'Value', (Raster('ras1.tif')+ Raster('ras2.tif'))) Using Geoprocessing tools: Use the Raster Calculator tool to execute a mathematical expression (also known as map algebraic expression in raster analysis). Then use the output of Raster Calculator, as the value raster input in the Zonal Statistics tool. You can also use Model Builder to automate this workflow. Hope this helps! Sarmistha
... View more
11-30-2021
12:56 PM
|
0
|
0
|
2063
|
POST
|
Thank you for sharing the data and for reporting this issue BabakJfard. It seems like the ZonalStatisticsAsTable tool has some issue in ArcGIS Pro 2.8.2 with the MakeNetCDFRasterLayer tool output as a value raster input. We will take a closer look and will address it in the upcoming release. For a workaround, please use the new MakeMultidimensionalRasterLayer tool to add the PM25 layer to your analysis in Pro. This tool creates a raster layer from a netCDF file with specified variables and dimensions: Try, arcpy.md.MakeMultidimensionalRasterLayer(r"C:\data\V4NA03_PM25_NA_200007_200007-RH35.nc", "PM25_MMRL", "PM25") Then use the raster layer in your Zonal tool, as you have done above. Your new output should contain the same number of zones, as in your CONUS counties dataset. Hope that helps! Thanks, Sarmistha
... View more
09-10-2021
03:54 PM
|
0
|
0
|
578
|
POST
|
Hi BabakJfard, Thank you for finding and posting the workaround. Even though projecting data from GCS to a PCS works for you, the underlying problem remains. The Zonal Statistical as Table tool should work with Geographic Coordinate System, and you do not have to project them before the analysis. If you can share the data with me, I can investigate it further. Thanks, Sarmistha
... View more
09-07-2021
12:05 PM
|
0
|
1
|
1794
|
POST
|
Hi BabakJfard, Thank you for reporting this. I tried to download your data, but could not access it, can you please make it public, so that I can further investigate it. If you cannot make the data public, please let me know, so that I can reach out to you directly and get the data. I have a few questions for you: Are the zones missing in the ZonalStatisticsAsTable dbf output, or in the TableToTable_conversion csv output? Do you also get the correct number of zone if you run ZonalStatistics? Do you get the correct number of zones in the output table if you use a raster zone? You can create a raster zone by converting your feature input zone to a raster zone using the PolygonToRaster tool using the same Cell Size, Snap Raster and Output Coordinate System as that of the value raster. Thanks, Sarmistha
... View more
09-03-2021
02:16 PM
|
0
|
1
|
1831
|
POST
|
On further investigation, it seems like the problem is related to using a multidimensional raster with multiple variables as input to zonal tools. The workaround is to create another multidimensional raster with only one variable and use it as an input to the Zonal Statistics and Zonal Statistics as Table tools. You can create a multidimensional raster with one variable in multiple ways: You can use the Make Multidimensional Raster Layer tool and specify the variable you want and use the output as input to zonal tools. This will not create a new multidimensional raster file on disk. You can also use the Subset Multidimensional Raster tool, however, it will create a new multidimensional raster file on disk. Creating a new file may not be required for your analysis. If you want to automate the workflow in Python using arcpy, you can use the Subset function to create a temporary multidimensional raster with only one variable.
... View more
05-12-2021
10:31 AM
|
0
|
0
|
976
|
POST
|
Thank you @dazu for reporting this issue and for attaching the project. Please let me know which software version you are using; I have used Pro 2.7.1 to test your cases. Zonal Statistics: From your project I see that you could successfully run the Zonal Statistics tool processing as multidimensional (1st run of Zonal Statistics from History), creating the correct output ZonalSt_asMulti.CRF. However, when you see the result in Contents, it shows the statistics of the tp variable, which is positive. The other 2 variables (e and evaow) has negative values, as you can see from the input value raster, era5_subset.CRF data. To view the other variables, from the Multidimensional tab, choose the variable you want to view to see the values. These values seem correct to me. Zonal Statistics as Table: Based on the inputs you have used, the output from Zonal Statistics as Table seems incorrect, I am investigating it further. Please let me know if that answers your question. Thanks, Sarmistha
... View more
05-11-2021
12:55 PM
|
0
|
0
|
993
|
POST
|
Thanks for reporting this issue Simon. I was able to run ZonalStatisticsAsTable tool in a model by looping through features in a feature class, as well as feature class in a workspace in Pro 2.7, using Iterate Feature Selection and Iterate Feature Classes respectively. In both the models, I could produce expected output table. From the image you have shared, I am not exactly sure which other tools you have used. So, based on the information you provided, I tried the following in ZonalStatisticsAsTable tool in the models below: Iterate Feature Selection- 2 zones from 1 shapefile, text zonefield with length 15, tif value raster, Mean statistics and an fgdb output table Iterate Feature Classes- 2 shapefile, text zonefield with length 15, tif value raster, Mean statistics and an fgdb output table Here is the screenshot of my models: The issue you have reported might be related to your data, or other tools or environment in the model. Can you share the model, and the data with us so that we can reproduce? Please feel free to submit a bug. Also, if you are looping through features in a feature class, what is the workflow you are trying to solve. Incase you are working with overlapping polygons, you do not have to use an iterator, as ZonalStatisticsAsTable tool can process overlapping polygons in Pro 2.7 and create statistics for each polygon zone. Thanks, Sarmistha
... View more
01-26-2021
05:36 PM
|
0
|
0
|
1603
|
BLOG
|
Many new and improved capabilities are available to the Spatial Analyst extension in ArcGIS Pro 2.7. At the top of the list are two brand new capabilities, the Storage Capacity tool in the hydrology toolset and the Surface Parameters tool in the surface toolset. In addition, new capabilities and performance improvements are available across a broad range of analytical areas. Where do I get it? ArcGIS Pro 2.7 was released on December 16, 2020. Download from here To see a complete summary of all the changes that have been made for this release, have a look at What’s New for ArcGIS Pro 2.7. This release also sees the introduction of a version switcher in the web help content. In any topic, click on the Other versions drop-down to see the help content for the previous release. 1. Density analysis The Kernel Density tool now supports the ability to use the geodesic method to calculate the density with a barrier feature. This creates better results for point features over a large extent. In the previous release, barriers were only supported for the planar distance method. The Kernel Density raster function now supports the ability to specify a barrier input. 2. Distance analysis A new parameter was added to the Optimal Path As Line tool. This option allows you to choose to create overlapping paths, or to now produce non-overlapping paths that can be used for networking. Performance has been improved for the planar distance method for the Distance Accumulation, Distance Allocation, and Optimal Region Connections tools as well as the Distance Accumulation and Distance Allocation raster functions. The performance has also been improved for the Least Cost Path raster function. 3. Generalization analysis The Boundary Clean tool now supports parallel processing for improved performance on larger datasets. This ability to smooth the boundaries between zones in raster data is now also available as the new Boundary Clean raster function. 4. Hydrology analysis The new Storage Capacity tool creates a table and optional chart of elevations and corresponding storage capacities for an input surface raster based on a zone input. It calculates the surface area and total volume of the underlying region at a series of elevation increments. Storage Capacity tool elevation volume curve example 5. Overlay analysis The Locate Regions tool has significantly improved performance compared to previous releases. This tool is a useful part of a suitability workflow to identify the best regions that meet specified size requirements and spatial constraints. 6. Segmentation and classification analysis The Export Training Data For Deep Learning tool has two new options available for the format of output metadata tables, Multi-labeled Tiles and Export Tiles. 7. Surface analysis Some significant new capabilities for surface analysis are available to Spatial Analyst in ArcGIS Pro 2.7. The new Surface Parameters tool determines various parameters of a raster surface such as aspect, slope and several types of curvature. These parameters are calculated on a cell-by-cell basis, by fitting a local quadratic or biquadratic surface around each target cell. You can define a specific neighborhood size to use for the calculations, or to have the tool to allow the neighborhood size to vary with local landscape changes. Illustration of adaptive neighborhood sizes based on local landscapes. There are two additional tools now available for surface analysis. Use the Add Surface Information tool to add various surface elevation properties to the attribute table of a feature class. Use the Interpolate Shape tool to convert 2D point, polyline or polygon feature class into a 3D feature class by interpolating z-values from a surface. 8. Zonal analysis For zonal statistics analysis, it is now possible to calculate median and percentile statistics for floating point input rasters. For the percentile type, a new parameter is available that specifies the method of percentile interpolation to be used when the number of values from the input raster to be calculated are even. Also supported now is multidimensional processing for inputs with different dimensions. These changes are available for the Zonal Statistics and Zonal Statistics as Table tools and the Zonal Statistics raster function. 9. ArcPy functions & Geoprocessing functions Following is the list of updates made in the Spatial Analyst module. ArcPy functions The following new ArcPy functions are available in the arcpy.sa module in this release, listed by category: Band Indices—BAI, NBR, NDBI, NDMI, and NDSI Classification—Classify, LinearUnmixing, RegionGrow, and SegMeanShift Conversion—RasterizeFeatures Reclass—ZonalRemap A Spatial Analyst license is required to use the entries in the Classification category. The entries in the Band Indices, Conversion and Reclass categories do not require any extension license. Geoprocessing functions in Pro 2.7 The following new geoprocessing functions are available in the arcpy.sa module, listed by toolset: Hydrology—StorageCapacity Surface—AddSurfaceInformation, InterpolateShape, and SurfaceParameters Spatial Analyst resources Refer back to the Spatial Analyst resources blog to keep tabs on current developments and to find links to useful and interesting content. Here are some recent posts that you might find interesting: The new Suitability Modeler: Changing the way suitability models are created Reclassify Raster Data in ArcGIS Pro Spatial Analyst Extension Summary 2020 has been different for all of us, to say the least, but working from home has not stood in the way of the Spatial Analyst team making new and improved capabilities available to you! Check out the new ArcGIS Pro 2.7 (December 2020) release and try out the new capabilities. Share your views, analysis stories, suggestions, and feedback with us on the ArcGIS Spatial Analyst Geonet page.
... View more
12-18-2020
04:42 PM
|
1
|
0
|
794
|
POST
|
Hi Ivan, Thank you for the data and the details. I used your data in Zonal Statistics as Table with zone field = GID_1, and statistics type = Sum, and by keeping the Geoprocessing environment default, and got the correct output in ArcGIS Pro 2.6.3. However, if I change the cell size or the snap raster, the output will be different as expected due to the resampling happening with the values in the value raster during the zonal analysis. We need to make sure that you are using the default environment. Please send me all the parameters that you have used in the Zonal Statistics as Table tool, and the environment values of cell size, output coordinate system, extent and snap raster, so that I can investigate further. Thank you, Sarmistha
... View more
12-18-2020
03:56 PM
|
0
|
1
|
1578
|
POST
|
Hi Ivan, I am looking at your post, and I would like to know more about it before I try to explain the result. Can you please let me know: What version of the software you are using? What is the cell size of the value raster? Are you executing the tool with a default environment or specifying anything such as extent, cell size, snap raster or output coordinate system, or a combination? If you are interested to share the data with me for further investigation, let me know. I look forward to your response. Thanks, Sarmistha
... View more
12-15-2020
05:14 PM
|
0
|
1
|
1609
|
Title | Kudos | Posted |
---|---|---|
2 | 01-12-2022 10:46 AM | |
1 | 12-18-2020 04:42 PM | |
1 | 04-16-2019 04:54 PM | |
1 | 08-12-2020 01:12 PM | |
3 | 09-07-2017 05:55 PM |
Online Status |
Offline
|
Date Last Visited |
02-14-2024
10:41 PM
|