Create an nDSM Using Raster Functions

[GuenterDoerffel]

Sources and further reading

The data I use for the blog is open government data from Vienna, Austria. Find it at Offene Daten Österreich | data.gv.at. This site is in German. 

See the help topic Use mosaic dataset items in raster function templates for raster function templates (RFT) that work on an Item Group level. I will only reference the UI and workflows in ArcGIS Pro. The template would work in ArcMap the same way.

nDSM—Two model suggestions

At any location, the Normalized Digital Surface Model (nDSM) is the difference between the built-up/overgrown surface, or DSM, and the bare ground, or DTM, elevation. In this calculation, the bare ground is 0 everywhere by definition. Objects like trees or buildings will be represented as absolute heights above local terrain. The simplest raster function to create nDSM from DSM and DTM is to only use the Arithmetic  function (available as a raster core function):

However, elevation datasets may contain errors. For DSM data derived from lidar, this risk is quite high (birds, planes). QC of the data is a prerequisite. In cities, the highest buildings are usually known; with vegetation, this might become more difficult. See topics like Locate outliers (3D Analyst) for that, or fixing elevation dataset voids using the Elevation Void Fill function.

In other cases, the DSM might be more correct (a recent excavation site is correct in the DSM but not in the [maybe older] DTM). This is easier to fix—assuming an nDSM can never be below 0. This fix could be built into the raster function chain, using the Less Than and the Con functions (both available with ArcGIS Spatial Analyst and/or ArcGIS Image Analyst licenses):

This version of the nDSM raster function chain will substract the DTM from the DSM in the Arithmetic function and send the result into the Less Than function. The second raster for the Less Than function is a constant raster of 0. Less Than is a Boolean function: If the elevation value is less than the constant value (of 0), the condition is True, else False. This Boolean result is handed over to the Con function, which performs a conditional if-then-else and will either assign the (positive) value of the Arithmetic function to the output cell, or the value of 0.

In the two use cases below, I will use the second model and explain all the settings. If you build on top of this, you can customize it for further needs.

Two use cases—Many data sources

The following description is basically independent of the data sources used as the elevation data source. It could be ASCII point data, lidar, or the result of drone processing, for example. The only important prerequisite is that you need to have DSM and DTM data of an overlapping area. This is where the two use cases differ:

• Use case—Separate MDs: DSM and DTM have different origin/tiling/schema/dates ==> are in different MDs.
• Use case—One MD: DSM and DTM have essentially a 1:1 relationship (like the same tiling schema and dates) ==> and here can be held in the same MD.

Use case: Separate MDs

When collecting DSM and DTM datasets over time and adding them to an (always improving/recent) MD, there might be different datasets coming into both from the sources mentioned above or even from a land survey. You can maintain them independently and create the nDSM as a function, available on one of them or both—you decide. Following is the workflow to add the nDSM raster function to the DSM MD created:

 Here are the two source MDs side by side: At left, the DSM (central Vienna, Stephansdom area); at right, the DTM with the rasters listed below in TableView. In the Catalog window, it looks like this:

In this Catalog view, right-click your DSM layer and select the Manage Processing Templates entry:

If there are no processing templates yet, or to create the one you need, add a new template:

The Function Editor will open with an empty template called None. Open the properties first:

This will display the Edit Properties form—see the important settings noted in red:

1.  This is the name displayed later in the UI for selecting this template—choose one that is unique and self-explanatory.
2.  The function template you create will apply to the mosaic as such—this is important.
3.  Think about creating and adding a visual for the function—not mandatory, though—then click OK to confirm.

The (still) empty Function Editor appears—time to add the functions. Open the Raster Functions pane:

In the Search box of the Raster Functions pane, enter "Arith" to quickly find the Arithmetic function. Drag it into the Function Editor:

It now looks like this:

Double-click the Arithmetic function to open the settings, and fill in accordingly:

1.  No need to do something here—this references the current mosaic.
2.  Navigate to your DTM mosaic here and select it.
3.  This is the place to select Minus as the mathematical operator— leave the other defaults and click OK.

The Function Editor now looks something like this. Click the Save button to confirm:

Once clicking Save, the properties will appear again. Just click OK.[Edit OK?]
As a result, the name of the function template should have changed to the one you gave it:
 

Don't worry: There is no dataset connected to the raster input yet. Once closed/reopened, it will display.

If there is no error message, the function template is okay and can actually be used (this is the simple version described above). But now you'll improve it with the model setting that avoids values below 0.

Open the Raster Functions pane again, and first search for "Less", then for "Con", and add both to the Function Editor:

Once they are added, you will need to make the necessary connections. Start with connecting the output of the Arithmetic function with the Less Than function. You will want the output of the Arithmetic function to be the first raster in the Less Than function:

As Raster2, the second raster, is the one to check against with the Less Than function, you need a raster that has only 0 values.
You get this by creating it in the Function Editor, using the Add Constant button:

As this is by default 0, you only have to connect this 0 raster to Raster2 of the Less Than function:

Now, you'll have to connect the three inputs that the Con function needs. These are called Raster, True Raster, and False Raster—and those three have to be connected like this:

• Raster is the output of the Less Than function, which is Boolean (either true or false). 
• True Raster is what needs to be done when Less Than has returned True, so this needs to be connected to the 0 raster (again).
• False Raster is what needs to be added when Less Than has returned False. The output of the Arithmetic function needs to be connected here (again).

Here's how it looks once you are done connecting:

You can clean it up using the Auto Layout button, then it should look like this:

Save and close and reopen once: Now you'll also see the reference (shown in green) to the DSM mosaic itself, as mentioned above:

You can now use this template in the UI of ArcGIS Pro and ArcMap or through services:

Use case: One MD

From the perspective of an MD, this is the more advanced and interesting one, and it is applicable very often as map sheets or common tiling schemes or similar processing is applied to elevation data. While the Raster function as such is not different from the first use case, the way to specify the data is different and shows the flexibility of the mosaic dataset. The DSM and DTM datasets have to be filtered and then assigned as input. You'll start like in the first use case and see the result of a query on the map and table of the MD:

In this dataset, for any selected tile in the map, two records will be found. In the attribute table

• Name will show the property the datasets share, here, the tile (could also be the date).
• GroupName in this example uses the built-in field to specify whether this is the DEM or the DSM dataset. In other workflows, a separate field called DEM Type is used for this.

Important: What needs to be fulfilled here is that every request for the common item (here, Name equals Tile) returns just one item of the specified group (here, GroupName equals DSM or DEM). They do not necessarily need to overlap 100 percent. Every request can contain more than one of the (here) tiles—like in the screenshot above to the left, nine tiles would be returned!

This is the needed data and table structure to use the second version of a Raster function template, which will use the records within the MD to do the nDSM calculation.

Before going through the workflow – another hint: To just use DEM or DSM from this dataset and base whatever function template on those is very easy, just use a definition query:

In this example, the definition query (highlighted) would filter out only the DSM datasets for use with this Raster function.

But back to the workflow for your use case One MD and nDSM:
This dataset with DSM and DTM in it looks like this in the Catalog window:

Right-click this one and select the Manage Processing Templates entry:

If there are no processing templates yet, or to create the one you need, add a new template again:

The Function Editor will open with an empty template called None. Open the properties first:

This will display the Edit Properties form—see the important settings highlighted that differ from the first example:

Choose a different name and description (and thumbnail) and make sure to adjust these important settings (noted in red):

1. Change the Type parameter to work on Item Group now (instead of Mosaic previously).
2. Type the field name here that reflects the common group identifier (in this example, the tile name).
3. Type the field name here that contains the dataset type identifier (in this example, DSM and DTM). 

Then click OK to confirm.

The (still) empty Function Editor reappears, like in use case 1 above.

As previously, open the Raster Functions pane:

In the Search box of the Raster Functions pane, enter "Arith" again to quickly find the Arithmetic function. Drag it into the Function Editor:

It will now look like this:

Double-click the Arithmetic function to open the properties, and fill in differently this time:

The Minus operator in the Parameters section is going to be similar. But now you will switch to the Variables section:

This is where it all happens now!

Remember, in the properties of the Raster Functions pane, you had specified in the Name field where the dataset type was to be found. Now, here you have to enter the values in this field that are to be used:

1. In the Name column for the parameter Raster, specify DSM.
2. In the Name column for the parameter Raster2, specify DEM
   Then click OK to confirm.

After these settings to the Arithmetic function, the remaining procedure is identical to the function chain above. So the following steps repeated here are just for your convenience—no change. ...

Open the Raster Functions pane again, and first search for "Less", then for "Con", add both to the Function Editor:

Once they are in, you need to make the necessary connections. Start with connecting the output of the Arithmetic function with the Less Than function. You want the output of the Arithmetic function to be the first Raster in the Less Than function:

As Raster2, the second raster, is the one to check against with the Less Than function, you need a raster that has only 0 values.

You get this by creating it in the Function Editor, using the Add Constant button:

As this is by default 0, you only have to connect this 0 raster to Raster2 of the Less Than function:

Now, you have to connect the three inputs that the Con function needs. These are called Raster, True Raster, and False Raster, and these three have to be connected like this:

• Raster is the output of the Less Than function,  which is Boolean (either true or false). 
• True Raster is what needs to be done when Less Than has returned True, so this needs to be connected to the 0 raster (again).
• False Raster is what needs to be added when Less Than has returned False. The output of the Arithmetic function needs  to be connected here (again).

Here's how it looks once you are done connecting:

You can clean it up by using the Auto Layout button, then it should look like this:

Finally, save this function chain.

You can now use this template in the UI of ArcGIS Pro and ArcMap or through services:

The two resultant RFT templates have been exported and are attached to this blog article so you can look at them in detail. Please do not forget to adjust according to your MD field names/data structure!