18 Replies Latest reply on Dec 6, 2012 11:28 AM by Caleb1987

    Simple Merge of Shapefiles in a single folder

    tjonest3
      Hello,

          I am very new to Python scripting and apparently not very good at it. I have a single folder with many shapefiles that I need to merge into a single shapefile. They are in the same projection and will not hit the 2gb limit so I assume this would be the easiest type of script to write. I have been trying to execute from PythonWin since that is the IDE the ESRI training uses but at this point I don't care where I execute it just that it works. I assume the workflow to be something like: import arcpy, set the environment to that particular folders location, apply overwrite to true, create a list of the shapefiles in the folder (i assume by telling it to look for all files with the ending ".shp"), the using the Merge_management to execute the merge.
          So with that said, could anyone offer some help. I have found a few other scripts online but they have some extra arguements for parsing and what not and I could not get them to work. All help appreciated
        • Re: Simple Merge of Shapefiles in a single folder
          Caleb1987
          Try this:

          import arcpy, os
          
          out = r'Output\folder\location' # change this
          arcpy.env.workspace = r'Some\Path\Name'  #change this
          shplist =  arcpy.ListFeatureClasses('*.shp')
          arcpy.Merge_management(shplist, os.path.join(out, 'Merged_fc.shp'))
          print 'done'
          
          
          • Re: Simple Merge of Shapefiles in a single folder
            tjonest3
            I'm getting an error for invalid input. This seems to be the same problem I have had with the other scripts. Can't find any extra info on the input parameters in the help section.
            • Re: Simple Merge of Shapefiles in a single folder
              Caleb1987
              Weird, this works for me.  The input parameters need to be supplied as a list (in square brackets []).  Can you add a print statement to show the list? Perhaps you can find the problem there.  Here is a script converting coverages to shapefiles and merging at the end.  I have a similar merge operation below:

              import arcpy, os, sys, traceback
              arcpy.env.overwriteOutput = True
              
              # Workspace for interchange files
              ws = r'C:\IGIC\Beginner\Unit_1\DATA\EffigyMounds\InterchangeFiles'
              arcpy.env.workspace = ws
              
              # Local Variables
              out = r'C:\IGIC\Beginner\Unit_1\DATA\EffigyMounds\Fire'
              
              try:
                  # Creates a folder if it doesn't already exist
                  if not os.path.exists(out):
                      os.makedirs(out)
              
                  # List all ArcInfo Interchange files (.e00)
                  for cov in arcpy.ListFiles('*.e00'):
                      arcpy.ImportFromE00_conversion(cov, ws, cov.split('.')[0])
                      print 'Converted %s' %cov
              
                  # Exclude Interchange files and info folder
                  for cov in arcpy.ListFiles():
                      if not '.e00' in cov and cov != 'info':
                          arcpy.env.workspace = os.path.join(ws,cov)
                          for fc in arcpy.ListFeatureClasses('*polygon'):
                              arcpy.FeatureClassToFeatureClass_conversion(fc, out, '%s.shp' %cov)
                              shp = os.path.join(out, '%s.shp' %cov)
                              print 'Converted Coverage to %s' %shp
              
                  # List shapefiles in folder to create input list for merge
                  arcpy.env.workspace = out
                  shplist =  arcpy.ListFeatureClasses()
                  arcpy.Merge_management(shplist, os.path.join(out, 'All_Fires.shp'))
                  print 'done'
                  
              except:
                  
                  # Get the traceback object
                  tb = sys.exc_info()[2]
                  tbinfo = traceback.format_tb(tb)[0]
                  pymsg = "PYTHON ERRORS:\nTraceback info:\n" + tbinfo + \
                          "\nError Info:\n" + str(sys.exc_info()[1])
                  msgs = "ArcPy ERRORS:\n" + arcpy.GetMessages(2) + "\n"
                  arcpy.AddError(pymsg)
                  arcpy.AddError(msgs)
              
                  # Print Python error messages for use in Python / Python Window
                  print pymsg + "\n"
                  print msgs
              
              
              • Re: Simple Merge of Shapefiles in a single folder
                rdharles
                This is the ultra simplified version.  Merge all fcs in the given directory into a shapefile called "Merged.shp"

                import arcpy
                arcpy.env.workspace = "c:/junk/mergeMe"
                fcs = arcpy.ListFeatureClasses()
                arcpy.Merge_management(fcs, "Merged.shp")
                
                • Re: Simple Merge of Shapefiles in a single folder
                  mvolz47
                  Tyler:

                  What exactly is the error message that you are receiving?

                  Also, can you post the code that you are trying to run?
                  • Re: Simple Merge of Shapefiles in a single folder
                    tjonest3
                    So this is what my code looks like (copied from the python window). I am sure I have some simple error but it is hard to fix when your still at the starting gate. Is there some way that I can actually see what the Listing looks like? As in have it create a simple .txt or something similar so I can see if the ListFeatureClasses command is even finding what it is supposed to find? I tried to simply "print Listing" but all I get are "[]" as a result which makes me think it is not seeing the file in the first place for some reason. This file has about 15 shapefiles (and their associated files) generated by the eCognition image processing software and I am using it for testing because I have much bigger ones with thousands of very small shapefiles that I am really gonna need Python for. Anyway, my error message below is talking about input parameters. Really wish they offered a python course here at my university. Again I appreciate the help from you guys.


                    import arcpy
                    >>> arcpy.env.workspace = "D:eCognition Results/Landcover/Buildings"
                    >>> Listing = arcpy.ListFeatureClasses()
                    >>> arcpy.Merge_management(Listing,"MergedFile.shp")
                    Runtime error <class 'arcgisscripting.ExecuteError'>: Failed to execute. Parameters are not valid. ERROR 000735: Input Datasets: Value is required Failed to execute (Merge).
                    • Re: Simple Merge of Shapefiles in a single folder
                      Caleb1987
                      It appears that you are getting an empty list.  I would double check your path to make sure it is correct.   Also, not sure if this is the issue, but I would try putting an underscore between Cognition and Results (Cognition_Results) for that folder.  ArcGIS does not like spaces in paths.  I would do that then use your print statement (your print statement is correct).
                      • Re: Simple Merge of Shapefiles in a single folder
                        mvolz47
                        Tyler:

                        It looks like this line might show the problem.

                        arcpy.env.workspace = "D:eCognition Results/Landcover/Buildings"

                        Should it read something like this instead:

                        arcpy.env.workspace = "D:/eCognition Results/Landcover/Buildings"

                        where you are currently missing a slash.
                        • Re: Simple Merge of Shapefiles in a single folder
                          tjonest3
                          You are right. Im missing a forward slash after "D:". Now I just feel silly. So I was able to print the Listing file but now at the mergeManagment I get the same error but with the actual filenames I was looking for at least.
                          arcpy.Merge_management(Listing,"MergedFile.shp")
                          Runtime error <class 'arcgisscripting.ExecuteError'>: Failed to execute. Parameters are not valid. ERROR 000732: Input Datasets: Dataset campus_ortho.tiles.tile00.v5.shp;campus_ortho.tiles.tile01.v8.shp;campus_ortho.tiles.tile02.v5.shp;campus_ortho.tiles.tile03.v6.shp;campus_ortho.tiles.tile04.v5.shp;campus_ortho.tiles.tile05.v9.shp;campus_ortho.tiles.tile06.v9.shp;campus_ortho.tiles.tile07.v9.shp;campus_ortho.tiles.tile08.v10.shp;campus_ortho.tiles.tile09.v8.shp;campus_ortho.tiles.tile10.v8.shp;campus_ortho.tiles.tile11.v8.shp;campus_ortho.tiles.tile12.v8.shp;campus_ortho.tiles.tile13.v9.shp;campus_ortho.tiles.tile14.v9.shp;campus_ortho.tiles.tile15.v8.shp;campus_ortho.tiles.tile17.v8.shp;campus_ortho.tiles.tile18.v9.shp does not exist or is not supported Failed to execute (Merge).
                          >>>

                          An example of the shapefile names would be "campus_ortho.tiles.tile00.v5.shp" whereas each tile from my project was assigned a unique name upon export. Does this name have anything to do with the error I am recieving? I am not sure what characters are reserved in filenames when it comes to this.
                          • Re: Simple Merge of Shapefiles in a single folder
                            Caleb1987
                            It may have something to do with the periods before the .shp extension on the shapefiles.  I would rename them by replacing the periods with underscores.  Try this:

                            import arcpy, os
                            arcpy.env.workspace = "D:/eCognition Results/Landcover/Buildings"
                            for file in arcpy.ListFiles('*.shp'):
                                os.rename(os.path.join(ws,file), os.path.join(ws,file.replace('.','_')[:-4] + '.shp'))
                            Listing = arcpy.ListFeatureClasses()
                            arcpy.Merge_management(Listing,"MergedFile.shp")
                            


                            If this doesn't work, I'm stumped.
                            • Re: Simple Merge of Shapefiles in a single folder
                              mvolz47
                              Are these raster datasets?  The merge_management documentation says that this data type is not supported.
                              • Re: Simple Merge of Shapefiles in a single folder
                                tjonest3
                                Caleb,

                                   I went and renamed the files by hand and it worked like a charm. Thanks for the help. Now I just have to get the rename code to work so I don't have to do them all by hand.
                                • Re: Simple Merge of Shapefiles in a single folder
                                  Caleb1987
                                  Caleb,

                                     I went and renamed the files by hand and it worked like a charm. Thanks for the help. Now I just have to get the rename code to work so I don't have to do them all by hand.


                                  I guess I should have mentioned that code was not thoroughly tested.  I copied several of the names of your shapefiles and renamed them as text files and ran this code.  It did successfully convert those txt files to shp, but I do not know how that would work with real shapefiles.  It may work with taking out the wild card that I had in the first time. I also changed it to ListFeatureClasses() which I do not know why I didn't do that the first time.

                                  import arcpy, os
                                  arcpy.env.workspace = "D:/eCognition Results/Landcover/Buildings"
                                  for file in arcpy.ListFeatureClasses():
                                      os.rename(os.path.join(ws,file), os.path.join(ws,file.replace('.','_')[:-4] + '.shp'))
                                  Listing = arcpy.ListFeatureClasses()
                                  arcpy.Merge_management(Listing,"MergedFile.shp")
                                  
                                  • Re: Simple Merge of Shapefiles in a single folder
                                    Caleb1987
                                    If THAT doesn't work (renaming shapefiles may not have been a good idea to begin with), here is my last idea:

                                    import arcpy, os
                                    ws = "D:/eCognition Results/Landcover/Buildings"
                                    arcpy.env.workspace = ws
                                    out = "D:/eCognition Results/Landcover/Buildings/Outputs"
                                    for file in arcpy.ListFeatureClasses():
                                        arcpy.CopyFeatures_management(file, os.path.join(out,file.replace('.','_')[:-4] + '.shp'))
                                    arcpy.env.workspace = out   
                                    Listing = arcpy.ListFeatureClasses()
                                    arcpy.Merge_management(Listing,"MergedFile.shp")
                                    


                                    If this does not work, Python: 1,  Me: 0
                                    • Re: Simple Merge of Shapefiles in a single folder
                                      tjonest3
                                      I tried the first one and the logic seems to make sense to me but I get an 'exeptions.NameError'>: name 'ws' is not defined' So I am looking over the rename function to see what those parameters are all about. Can't be far from it now. Again man thanks for all the help.
                                      • Re: Simple Merge of Shapefiles in a single folder
                                        Caleb1987
                                        I tried the first one and the logic seems to make sense to me but I get an   'exeptions.NameError'>: name 'ws' is not defined' So I am looking over the rename function to see what those parameters are all about. Can't be far from it now. Again man thanks for all the help.


                                        Ah! That is my fault! I had a ws variable in my test script on my end. Try this:

                                        import arcpy, os
                                        ws = "D:/eCognition Results/Landcover/Buildings"
                                        arcpy.env.workspace = ws
                                        for file in arcpy.ListFeatureClasses():
                                            os.rename(os.path.join(ws,file), os.path.join(ws,file.replace('.','_')[:-4] + '.shp'))
                                        Listing = arcpy.ListFeatureClasses()
                                        arcpy.Merge_management(Listing,"MergedFile.shp")
                                        


                                        The problem was I had a variable set for the workspace and used that to append the paths and forgot to do that in yours. If you copied and pasted what I had, that was probably the problem the whole time! Sorry!
                                        • Re: Simple Merge of Shapefiles in a single folder
                                          tjonest3
                                          Works!
                                          Only thing is that it only renames the .shp files and none of the associated files (.dbf,.prj, ect.) so the merge gives an error saying it cannot be completed probably because it cannot see the associated filetypes. However, I assume that I can just rewrite this for each of the filetypes and once they are all renamed the merge should work as expected? Fingers crossed.
                                          • Re: Simple Merge of Shapefiles in a single folder
                                            Caleb1987
                                            Works!
                                            Only thing is that it only renames the .shp files and none of the associated files (.dbf,.prj, ect.) so the merge gives an error saying it cannot be completed probably because it cannot see the associated filetypes. However, I assume that I can just rewrite this for each of the filetypes and once they are all renamed the merge should work as expected? Fingers crossed.


                                            I would just copy the shapefiles with a different name into another folder, much safer:

                                            import arcpy, os
                                            ws = "D:/eCognition Results/Landcover/Buildings"
                                            arcpy.env.workspace = ws
                                            out = "D:/eCognition Results/Landcover/Buildings/Outputs"
                                            for file in arcpy.ListFeatureClasses():
                                                arcpy.CopyFeatures_management(file, os.path.join(out,file.replace('.','_')[:-4] + '.shp'))
                                            arcpy.env.workspace = out   
                                            Listing = arcpy.ListFeatureClasses()
                                            arcpy.Merge_management(Listing,"MergedFile.shp")
                                            


                                            This *should work.