Skip navigation
All People > Dan_Patterson > Py... blog > 2016 > August
2016

This post was inspired by the GeoNet thread... https://geonet.esri.com/thread/116880

 

I'm a student and I need a python script that i can use for ArcMap

 

I usually suggest that my students use Modelbuilder to build workflows, export to a python script, then modify the script for general use with the existing, or other, data sets.  I personally don't use Modelbuilder, but I have used one of two methods to generate the needed workflow .... Method 1 will be presented in this post...method 2 follows.

 

Method 1

 

Do it once...get the script...modify and reuse


Because of the imbedded images...please open the *.pdf file to view the complete discussion...

 

Regards

Dan

This is part 2 which follows up on my previous blog post.  In this example, the assignment restrictions have changed and one must now develop a script from what they have read about Python and the tools that are used in everyday ArcMap workflows.

 

Details are given in the attached pdf of the same name.


Regards
Dan

 

Homework... make this into a script tool for use in arctoolbox

This blog is inspired by this thread https://geonet.esri.com/docs/DOC-2436#comment-9031 by Steve Offermann (Esri).  He suggested a very simple way to extend the capabilities of tool results and how to parse arguments for them.  I recommended the use of the Results window outputs in a previous blog.  Hats off to Steve.

 

I am only going to scratch the surface by presenting a fairly simple script...which could easily be turned into a tool.

In this example, a simple shapefile of hexagons, (presented in another blog post) was processed to yield:

 

  • an extent file, giving the bounds of each of the input hexagons,
  • the hexagon corners as points and sent to a shapefile, and,
  • the centroids of each hexagon was treated in a similar fashion

 

The whole trick is to parse your processes down into parameters that can be shared amongst tools.  In this case, tools that can be categorized as:

 

  • one parameter tools like:  AddXY_management and CopyFeatures_management
  • two parameter tools like:
    • FeatureEnvelopeToPolygon_management,
    • FeatureToPoint_management and
    • FeatureVerticesToPoints_management

 

This can then be amended by, or supplemented with, information on the input/output shape geometries.  I demonstrate this by calculating the X,Y coordinates for the point files.

 

So you are saying ... I don't do that stuff ... well remember, I don't do that webby stuff either.   Everyone has a different workflow and if my students are reading this, just think how you could batch project a bunch of files whilst simultaneously renaming them etc etc.  The imagination is only limited by its owner...

 

First the output....

 

Hexagon_outputs.png

 

And now the script....

"""
Script:   run_tools_demo.py
Author:   Dan.Patterson@carleton.ca

Source:   Stefan Offermann
Thread:   https://geonet.esri.com/docs/DOC-2436

Purpose:  Results window on steroids
  - take a polygon shapefile, determine its envelop,
  - convert the feature to centroids,
  - convert to feature points
  - calculate X,Y for all of the above
  - then make a back of everything
Requires:
  - a source file
  - an output folder
  - a list of tools to run
"""
import os
import arcpy
arcpy.env.overwriteOutput = True
in_FC = "c:/!BlogPosts/Runtools_Demo/Shapefiles/pointy_hex.shp"
path,in_File = os.path.split(in_FC)
path += "/"
backup = "c:/temp/shapefiles/"    # some output folder
# file endings
end = ["_env","_fp","_vert"]      # envelop, feature to point, feature vertices
# two argument tools
two_arg = ["FeatureEnvelopeToPolygon_management",
           "FeatureToPoint_management",
           "FeatureVerticesToPoints_management"
          ]
# one argument tools
one_arg =["AddXY_management", "CopyFeatures_management"]
#
outputs = [in_FC.replace(".shp", end[i]+".shp") for i in range(len(end))]
backups =  [outputs[i].replace(path, backup) for i in range(len(end))]
#
polygons = []
points = []
for i in range(len(two_arg)):                  # run the two argument tools
    args = [in_FC, outputs[i]]                 # select the output file
    result = getattr(arcpy, two_arg[i])(*args) # run the tool...and pray
    frmt = '\Processing tool: {}\n  Input: {}\n  Output: {}'
    print(frmt.format(tools[i], args[0], args[1]))
#
for i in range(len(outputs)):
    args = [outputs[i]]
    print(outputs[i], arcpy.Describe(outputs[i]).shapeType)
    if arcpy.Describe(outputs[i]).shapeType == 'Point':
        result = getattr(arcpy, one_arg[0])(*args) # calculate XY
    result_bak = getattr(arcpy, one_arg[1])(result, backups[i]) # backup
    print('Calculate XY for: {}'.format(result))
    print('Create Backups: {}\n  Output: {}'.format(result,result_bak))

Enjoy and experiment with your workflows.

UPDATE:   take the poll first before you read on How do you write Python path strings?

 

I am sure everyone is sick of hearing ... check your filenames and paths and make sure there is no X or Y.  Well, this is going to be a work in progress which demonstrates where things go wrong while maintaining the identity of the guilty.

Think about it
>>> import arcpy
>>> aoi = "f:\test\a"
>>> arcpy.env.workspace = aoi
>>> print(arcpy.env.workspace)
f: est
>>>

 

>>> print(os.path.abspath(arcpy.env.workspace))
F:\ est
>>> print(os.path.exists(arcpy.env.workspace))
False
>>> print(arcpy.Exists(arcpy.env.workspace))
False
>>>
>>> print("{!r:}".format(arcpy.env.workspace))
'f:\test\x07'
>>>

 

>>> os.listdir(aoi)
Traceback (most recent call last):
  File "<interactive input>", line 1, in <module>
OSError: [WinError 123] The filename, directory name,
or volume label syntax is incorrect: 'f:\test\x07'
>>>

 

>>> arcpy.ListWorkspaces("*","Folder")
>>>
>>> "!r:{}".format(arcpy.ListWorkspaces("*","Folder"))
'!r:None'
>>>

 

 

Examples... Rules broken and potential fixes

Total garbage... as well as way too long.  Time to buy an extra drive.

>>> x ="c:\somepath\aSubfolder\very_long\no_good\nix\this"
>>> print(x)                  # str notation
c:\somepath Subfolder ery_long
o_good
ix his
>>> print("{!r:}".format(x))  # repr notation
'c:\\somepath\x07Subfolder\x0bery_long\no_good\nix\this'
>>>
  • No r in front of the path.
  • \a \b \n \t \v are all escape characters... check the result
  • Notice the difference between plain str and repr notation

--------------------------------------------------------------------------------------------------------------------------

Solution 1... raw format

>>> x = r"c:\somepath\aSubfolder\very_long\no_good\nix\this"

>>> print(x)                  # str notation
c:\somepath\aSubfolder\very_long\no_good\nix\this

>>> print("{!r:}".format(x))  # repr notation
'c:\\somepath\\aSubfolder\\very_long\\no_good\\nix\\this'
>>>
  • Use raw formatting, the little r in front goes a long way.

--------------------------------------------------------------------------------------------------------------------------

Solution 2... double backslashes

>>> x ="c:\\somepath\\aSubfolder\\very_long\\no_good\\nix\\this"
>>> print(x)                  # str notation
c:\somepath\aSubfolder\very_long\no_good\nix\this

>>> print("{!r:}".format(x))  # repr notation
'c:\\somepath\\aSubfolder\\very_long\\no_good\\nix\\this'
>>>
  • Yes! I cleverly used raw formatting and everything should be fine but notice the difference between str and repr.

--------------------------------------------------------------------------------------------------------------------------

Solution 3... forward slashes

>>> x ="c:/somepath/aSubfolder/very_long/no_good/nix/this"
>>> print(x)                  # str notation
c:/somepath/aSubfolder/very_long/no_good/nix/this
>>> print("{!r:}".format(x))  # repr notation
'c:/somepath/aSubfolder/very_long/no_good/nix/this'
>>>

 

--------------------------------------------------------------------------------------------------------------------------

Solution 4... os.path functions

There are very useful functions and properties in os.path.  The reader is recommended to examine the contents after importing the os module (ie dir(os.path)  and help(os.path)

 

>>> x = r"F:\Writing_Projects\Before_I_Forget\Scripts\timeit_examples.py"
>>> base_name = os.path.basename(x)
>>> dir_name = os.path.dirname(x)
>>> os.path.split(joined)  # see splitdrive, splitext, splitunc
('F:\\Writing_Projects\\Before_I_Forget\\Scripts', 'timeit_examples.py')
>>> joined = os.path.join(dir_name,base_name)
>>> joined
'F:\\Writing_Projects\\Before_I_Forget\\Scripts\\timeit_examples.py'
>>>
>>> os.path.exists(joined)
True
>>> os.path.isdir(dir_name)
True
>>> os.path.isdir(joined)
False
>>>

ad nauseum

 

--------------------------------------------------------------------------------------------------------------------------

Gotcha's

Fixes often suggest the following ... what can go wrong, if you failed to check.

(1)

>>> x = "c:\somepath\aSubfolder\very_long\no_good\nix\this"
>>> new_folder = x.replace("\\","/")
>>> print(x)                  # str notation
c:\somepath Subfolder ery_long
o_good
ix his
>>> print("{!r:}".format(x))  # repr notation
'c:\\somepath\x07Subfolder\x0bery_long\no_good\nix\this'
>>>

 

(2)

>>> x = r"c:\new_project\aSubfolder\"
  File "<string>", line 1
    x = r"c:\new_project\aSubfolder\"
                                    ^
SyntaxError: EOL while scanning string literal

 

(3)

>>> x = "c:\new_project\New_Data"
>>> y = "new_grid"
>>> out = x + "\\" + y
>>> print(out)
c:
ew_project\New_Data\new_grid

 

(4)

>>> x = r"c:\new_project\New_Data"
>>> z = "\new_grid"
>>> out = x + z
>>> print(out)
c:\new_project\New_Data
ew_grid

 

(5)  This isn't going to happen again!

>>> x = r"c:\new_project\New_Data"
>>> z = r"\new_grid"
>>> out = x + y
>>> print(out)
c:\new_project\New_Datanew_grid

 

(6)  Last try

>>> x = r"c:\new_project\New_Data"
>>> z = r"new_grid"
>>> please = x + "\\" + z
>>> print(please)
c:\new_project\New_Data\new_grid

 

Well this isn't good!   Lesson?  Get it right the first time. Remember the next time someone says...

Have you checked your file paths...?????   Remember these examples.

 

Curtis pointed out this helpful link...I will include it here as well

Paths explained: Absolute, relative, UNC, and URL—Help | ArcGIS for Desktop

 

That's all for now.

I will deal with spaces in filenames in an update.  I am not even to go to UNC paths.

Code formatting tips

 

Updated - 2017/06/27  Added another reference and some editing.

 

This topic has been covered by others as well...

 

We all agree the Geonet code editor is horrible... but it has been updated.

Here are some other tips.

 

To begin... introduction or review

  • don't try to post code while you are responding to a thread in your inbox
  • access the More button, from the main thread's title... to do this:
    • click on the main thread's title
    • now you should see it... you may proceed

 

Step 1... select ... More ... then ... Syntax highlighter
  • Go to the question and press Reply ...
  • Select the Advanced editor if needed (or ...),  then select

If you can't see it, you didn't select it

 

 More...Syntax highlighter ,

 

Your code can now be pasted in and highlighted with the language of your choice .........

Your code should be highlighted something like this ............

--- Python -----------------------------

import numpy as np
a = np.arange(5)
print("Sample results:... {}".format(a))

--------------------------------------------

Now the above example gives you the language syntax highlighting that you are familiar with..

Alternatives include just using the HTML/XML option

-----HTML/XML ---------------------

# just use the HTML/XML option.. syntax colors will be removed
import numpy as np
a = np.arange(5)
print("simple format style {}".format(a))
simple format style [0 1 2 3 4]

--------------------------------------------

 

NOTE:   you can only edit code within this box and not from outside!

 

 

 

Script editing tipscont'd

HTML editing tips:....

  • You can get into the html editor for fine-tuning, but it can get ugly for the uninitiated.
  • Comments get deleted ... this not a full editor under your control
  • If you have lots of text to enter, do it first then enter and format code
  • If editing refresh is slow, use the HTML editor or you will have retired before it completes.
  • The editor seems to edit each character as you type and it gets painfully slower as the post gets bigger.
  • You can improve comments and code output by using tables like this and in the example below.

 

Here is a simple script with code and results published in columns (2 columns * 1 row).  If the contents are wider than the screen, the scroll-bar will be located at the end of the document rather than attached to each table (except for iThingys, then just use swipe).

 

Sample script using a plain format... 1920x1080px screen sizeResult 2
>>> import numpy as np
>>> a = np.arange(5)
>>> print("Sample results:... {}".format(a))
>>> # really long comment 30 |------40 | -----50 | -----60 | -----70 | ---- 80|
Sample results:... [0 1 2 3 4]
>>> # really long comment 30 |------40 | -----50 | -----60 | -----70 | ---- 80|

 

Leave space after a table so you can continue editing after the initial code insertion.

It is often hard to select the whitespace before or after a table and you may need to go to the html editor < > just above the More toggle

 

Larger script sample...

Before code tip:  try to keep your line length <70 characters

# -*- coding: UTF-8 -*-
"""
:Script:   demo.py
:Author:   Dan.Patterson@carleton.ca
:Modified: 2016-08-14
:Purpose:  none
:Functions:  help(<function name>) for help
:----------------------------
: _demo  -  This function ...
:Notes:
:References
:
"""

#---- imports, formats, constants ----

import sys
import numpy as np
from textwrap import dedent

ft = {'bool':lambda x: repr(x.astype('int32')),
      'float': '{: 0.3f}'.format}
np.set_printoptions(edgeitems=10, linewidth=80, precision=2, suppress=True,
                    threshold=100, formatter=ft)
script = sys.argv[0]

#---- functions ----

def _demo():
    """  
    :Requires:
    :--------
    :
    :Returns:
    :-------
    :
    """

    return None

#----------------------
if __name__ == "__main__":
    """Main section...   """
    #print("Script... {}".format(script))
    _demo()

 

Some space for editing after should be left since positioning the cursor is difficult after the fact.

 

Output options

 

  • You can paste text and graphics with a table column.
  • You can format a column to a maximum pixel size.

 

Sample output with a graph

Option 0: 1000 points
[[ 2. 2.]
[ 3. 3.]] extents
.... snip
Time results: 1.280e-05 s, for 1000 repeats

point_in_polygon.png

point_in_polygon.png

 

So there has been some improvement. 

Again...

You just have to remember that to edit code...

you have to go back to the syntax highlighter.

You can't edit directly on the page.

Reclassifying raster data can be a bit of a challenge, particularly if there are nodata values in the raster.  This is a simple example of how to perform classifications using a sample array. (background 6,000+ rasters the follow-up  )

 

An array will be used since it is simple to bring in raster data to numpy using arcpy's:

  •   RasterToNumPyArray
    •   RasterToNumPyArray (in_raster, {lower_left_corner}, {ncols}, {nrows}, {nodata_to_value})

and sending the result back out using:

  • NumPyArrayToRaster
    • NumPyArrayToRaster (in_array, {lower_left_corner}, {x_cell_size}, {y_cell_size}, {value_to_nodata})

On with the demo...

 

Raster with full data

old                                new

[ 0  1  2  3  4  5  6  7  8  9]    [1 1 1 1 1 2 2 2 2 2]

[10 11 12 13 14 15 16 17 18 19]    [3 3 3 3 3 4 4 4 4 4]

[20 21 22 23 24 25 26 27 28 29]    [5 5 5 5 5 6 6 6 6 6]

[30 31 32 33 34 35 36 37 38 39]    [7 7 7 7 7 7 7 7 7 7]

[40 41 42 43 44 45 46 47 48 49]    [7 7 7 7 7 7 7 7 7 7]

[50 51 52 53 54 55 56 57 58 59]    [7 7 7 7 7 7 7 7 7 7]

# basic structure
a = np.arange(60).reshape((6, 10))
a_rc = np.zeros_like(a)
bins = [0, 5, 10, 15, 20, 25, 30, 60, 100]
new_bins = [1, 2, 3, 4, 5, 6, 7, 8]
new_classes = zip(bins[:-1], bins[1:], new_bins)
for rc in new_classes:
    q1 = (a >= rc[0])
    q2 = (a < rc[1])
    z = np.where(q1 & q2, rc[2], 0)
    a_rc = a_rc + z
return a_rc
# result returned

 

Lines 2, 3, 4 and 5 describe the array/raster, the classes that are to be used in reclassifying the raster and the new classes to assign to each class.  Line 5 simply zips the bins and new_bins into a new_classes arrangement which will subsequently be used to query the array, locate the appropriate values and perform the assignment (lines 6-10 )

 

Line 3 is simply the array that the results will be placed.  The np.zeros_like function essentially creates an array with the same structure and data type as the input array.  There are other options that could be used to create containment or result arrays, but reclassification is going to be a simple addition process...

 

  • locate the old classes
  • reclass those cells to a new value
  • add the results to the containment raster/array

 

Simple but effective... just ensure that your new classes are inclusive by adding one class value outside the possible range of the data.

 

Line 10 contains the np.where statement which cleverly allows you to put in a query and assign an output value where the condition is met and where it is not met.  You could be foolish and try to build the big wonking query that handles everything in one line... but you would soon forget when you revisit the resultant the next day.  So to alleviate this possibility, the little tiny for loop does the reclassification one grouping at a time and adds the resultant to the destination array.  When the process is complete, the final array is returned.

 

Now on to the arrays/rasters that have nodata values.  The assignment of nodata values is handled by RasterToNumPyArray so you should be aware of what is assigned to it.

 

Raster with nodata values

old                                  new

[--  1  2  3  4  5  6 --  8  9]      [--  1  1  1  1  2  2 --  2  2]

[10 11 12 13 -- 15 16 17 18 19]      [ 3  3  3  3 --  4  4  4  4  4]

[20 -- 22 23 24 25 26 27 -- 29]      [ 5 --  5  5  5  6  6  6 --  6]

[30 31 32 33 34 -- 36 37 38 39]      [ 7  7  7  7  7 --  7  7  7  7]

[40 41 -- 43 44 45 46 47 48 --]      [ 7  7 --  7  7  7  7  7  7 --]

[50 51 52 53 54 55 -- 57 58 59]]     [ 7  7  7  7  7  7 --  7  7  7]

Make a mask (aka ... nodata values) where the numbers are divisible by 7 and the remainder is 0.

Perform the reclassification using the previous conditions.

 

# mask the values
a_mask = np.ma.masked_where(a%7==0, a)
a_mask.set_fill_value(-1)
# set the nodata value

 

The attached sample script prints out the test with the following information:

 

Input array ... type ndarray

...snip...

Reclassification using

:  from [0, 5, 10, 15, 20, 25, 30, 60, 100]

:  to   [1, 2, 3, 4, 5, 6, 7, 8]

:  mask is False value is None

Reclassed array

...snip...

 

Input array ... type MaskedArray

...snip

Reclassification using

:  from [0, 5, 10, 15, 20, 25, 30, 60, 100]

:  to   [1, 2, 3, 4, 5, 6, 7, 8]

:  mask is True value is -1

Reclassed array

...snip....

-------------------------------------

That's about all for now.  Check the documentation on masked arrays and their functions.  Most functions and properties that apply to ndarrays also apply to masked arrays... it's like learning a new language just by altering the pronounciation of what you already know.