46 Replies Latest reply on Jan 15, 2014 6:31 AM by hcopeland

    ArcGIS 10 Pre-Release finding duplicates

    haydena
      Basically, the info in the article http://resources.arcgis.com/content/kbase?fa=articleShow&d=31854 has worked for me in 9.3 and prior.  In pre-release 10 I get:

      General error executing calculator.
      ERROR 999999: Error executing function.
      Expected statement
      Failed to execute (CalculateField).
      Failed at Thu May 06 10:46:18 2010 (Elapsed Time: 1.00 seconds)

      Something with CreateObject("Scripting.Dictionary") perhaps?  Any other thoughts?
        • Re: ArcGIS 10 Pre-Release finding duplicates
          kmarten-esristaff
          Hi Haydena,

          In ArcGIS 10, the Field Calculator has been enhanced to work with Python scripting. You can no longer use VBA code blocks; instead, both VB Script and Python are fully supported.  The code sample you are trying to use will not work because it uses VBA.  There is not a VBScript alternative for this VBA sample because you cannot declare Static variables.

          You may be interested in learning more about using Python with the field calculator.  Here is a Help Topic to get you started: http://help.arcgis.com/en/arcgisdesktop/10.0/help/index.html#/Calculate_Field_examples/005s0000002m000000/

          I'll try and find some more information for you regarding this specific workflow and using Python.

          Kent M.
          esri
          • Re: ArcGIS 10 Pre-Release finding duplicates
            kmarten-esristaff
            Hi Haydena,

            Using the Field Calculator in ArcGIS 10.  You can use Python to easily find duplicate values in a given field.  In Python you can declare a list outside of the function so that it can retain added values.  Here is a code sample:

            uniqueList = []
            def isDuplicate(inValue):
              if inValue in uniqueList:
                return 1
              else:
                uniqueList.append(inValue)
                return 0


            At the bottom of the Field Calculator, you will have to enter in the function and field name like this:
            isDuplicate(!FIELD_NAME!)


            Thanks for your post.

            Kent M.
            esri
            • Re: ArcGIS 10 Pre-Release finding duplicates
              haydena
              Works perfect.

              Thanks for the response Kent.
              • Re: ArcGIS 10 Pre-Release finding duplicates
                coltersikora
                Hello!

                I am looking to find duplicate attributes in my shapefile as shown in this forum.  However, when I try the Python code example I get the following error message: "Error 000539: Error running expression: uniqueList = Error running expression: uniqueList = [] "

                Here is the code that I entered:

                uniqueList = [ !Duplicate! ]
                def isDuplicate(inValue):
                if inValue in uniqueList:
                  return 1
                else:
                  uniqueList.append(inValue)
                  return 0

                isDuplicate(!Calculate!)


                ---

                "Duplicate" is the field I want the results entered into, "Calculate" is the field that I want duplicates selected from.

                Hopefully this is enough code to explain my situation. 

                Thank you for your help in advance!

                Sincerely,

                Colter
                • Re: ArcGIS 10 Pre-Release finding duplicates
                  jcmolina
                  Hi All,

                  First, thanks Kent M. for making this script available, it works great.

                  Colter, this is how I made it work.

                  1) Right click on the column you wish to populate > field calculator>check in "Show Codeblock" and select "Python" at the top let corner of the calculator
                  2) Copy the first piece of Kent's code in the upper box of the calculator (Do not write anything on that part, do not add the column name, just leave it as it is)
                  3) Copy the second part of the code (---isDuplicate(!Calculate!)-----) in the lower box and replace the "Calculate" word with the column name that you are exploring for duplicates, leave the exclamation marks in.

                  I hope that works for you all.

                  JCMolina
                  • Re: ArcGIS 10 Pre-Release finding duplicates
                    leftyb
                    Hi there, this works great, but I was wondering - is there a way to modify the script so it shows the first record of the duplicate as a 1 also?
                    • Re: ArcGIS 10 Pre-Release finding duplicates
                      mse2k11
                      I'm getting the following error when I run this script:

                      "There was a failure during processing, check the Geoprocessing Results window for details."

                      However, no Geoprocessing Results window is evident.

                      Any suggestions?
                      • Re: ArcGIS 10 Pre-Release finding duplicates
                        tguthrie
                        You can also use the frequency analysis geoprocessing tool to find all duplicates, including the first one. Anything resulting in a frequency > 1 for a given field would be a duplicate.

                        http://help.arcgis.com/en/arcgisdesktop/10.0/help/index.html#/Frequency/00080000001w000000/
                        • Re: ArcGIS 10 Pre-Release finding duplicates
                          davidbrett
                          Thanks for the code, works great.

                          Any insight on modifying it to mark the duplicates with an incremental number per occurrence?

                          Like what EzCalc did in VBA:
                          field_Mark_Duplicates_2.cal Calculates a unique number for each occurence of a value in a field. Long
                          Example:
                          a==>1
                          b==>1
                          c==>1
                          b==>2
                          b==>3
                          a==>2
                          • Re: ArcGIS 10 Pre-Release finding duplicates
                            Jamal432@gmail.com
                            Hi Haydena,

                            In ArcGIS 10, the Field Calculator has been enhanced to work with Python scripting. You can no longer use VBA code blocks; instead, both VB Script and Python are fully supported.  The code sample you are trying to use will not work because it uses VBA.  There is not a VBScript alternative for this VBA sample because you cannot declare Static variables.

                            You may be interested in learning more about using Python with the field calculator.  Here is a Help Topic to get you started: http://help.arcgis.com/en/arcgisdesktop/10.0/help/index.html#/Calculate_Field_examples/005s0000002m000000/

                            I'll try and find some more information for you regarding this specific workflow and using Python.

                            Kent M.
                            esri


                            Hi Kent,

                            i couldn't manage how to rewrite the code below (if statement) in VBscript or python.
                            could you please help me how to do so?

                            and in case we have so many codes in VBA, are we required to switch them to VBscript or python? is that reasonable?

                            thank you

                            the code:
                            -------------------------
                            if [FC_mm] <=50 then
                            op = 0

                            else
                            if [FC_mm]>50 and [FC_mm]<=90 then
                            op = 50

                            else
                            if [FC_mm] >90 and [FC_mm] <= 140 then
                            op = 125

                            else
                            if [FC_mm] >140 and [FC_mm] < = 200 then
                            op = 250

                            else
                            if [FC_mm] > 200 and [FC_mm] < = 250 then
                            op = 500

                            else
                            if [FC_mm] > 250 then
                            op = 750

                            end if
                            end if
                            end if
                            end if
                            end if
                            end if 
                            -----------------------------------------------------
                            • Re: ArcGIS 10 Pre-Release finding duplicates
                              davidbrett
                              In the field calculator, check 'show code block', then paste the following in the 'Pre-Logic Script Code' section:

                              def myVal(FC_mm):
                                rec = 0
                                if FC_mm>50 and FC_mm<=90:
                                  rec = 50
                                if FC_mm >90 and FC_mm <= 140:
                                  rec = 125
                                if FC_mm >140 and FC_mm <= 200:
                                  rec = 250
                                if FC_mm > 200 and FC_mm <= 250:
                                  rec = 500
                                if FC_mm > 250:
                                  rec = 750
                                return rec
                              


                              In the box below, paste:
                              myVal( !FC_mm! )
                              
                              • Re: ArcGIS 10 Pre-Release finding duplicates
                                hami1981
                                #Here's a script that flags duplicates as "1" including the first occurence:
                                #It works fine for strings and integers, but doesn't work for float values, to overcome the float
                                #problem convert the values to strings.
                                #Paste the following code in the Field Calculator:

                                import arcpy
                                inFeatures = "C:\\Folder\\Subfolder\\yourshapefile.shp"

                                inField = "YOUR_FIELD" #The field you want to analyse
                                rows = arcpy.SearchCursor(inFeatures)
                                row = rows.next()
                                ValueList = list()
                                while row:
                                    currentValue = row.getValue(inField)
                                    ValueList.append(currentValue)
                                    row = rows.next()

                                def isDuplicate(inValue):
                                    if ValueList.count(inValue) > 1:
                                        return 1
                                    else:
                                        return 0


                                ##In the box below, paste:
                                ## isDuplicate(!YOUR_FIELD!)
                                • Re: ArcGIS 10 Pre-Release finding duplicates
                                  olau
                                  Hami1981,

                                  I was unable to process the alternative duplicate script that can label the first occurrence as "1"

                                  import arcpy
                                  inFeatures = "C:\\Folder\\Subfolder\\yourshapefile.shp"

                                  inField = "!AnalysisField!"
                                  rows = arcpy.SearchCursor(inFeatures)
                                  row = rows.next()
                                  ValueList = list()
                                  while row:
                                  currentValue = row.getValue(inField)
                                  ValueList.append(currentValue)
                                  row = rows.next()

                                  def isDuplicate(inValue):
                                  if ValueList.count(inValue) > 1:
                                  return 1
                                  else:
                                  return 0

                                  The error I received was "Failed to execute. Parameters are not valid"
                                  Error 000989: Python syntax error: Parsing error <type 'exceptions.SyntaxError'>: invalid syntax (line 9)
                                  Failed to execute (CalculateField)

                                  Any thoughts?
                                  • Re: ArcGIS 10 Pre-Release finding duplicates
                                    hami1981
                                    Hi Olivia,

                                    First, remember python is tab sensitive, so a four-space tab indent width is the prefered coding style for Python. In your code i'm going to repace space/tabs with "_". But do replace it again with spaces in your code!

                                    Also, don't forget to set the right path to your shapefile and keep the double \\ (because \ is an scape character)

                                    I think what your problem is that you put inField = "!AnalysisField!", it should be:
                                    inField = "AnalysisField" without the !.


                                    import arcpy
                                    inFeatures = "C:\\Folder\\Subfolder\\yourshapefile.shp" #the path to your shapefile

                                    inField = "!AnalysisField!"
                                    rows = arcpy.SearchCursor(inFeatures)
                                    row = rows.next()
                                    ValueList = list()
                                    while row:
                                    ____currentValue = row.getValue(inField)
                                    ____ValueList.append(currentValue)
                                    ____row = rows.next()

                                    def isDuplicate(inValue):
                                    ____if ValueList.count(inValue) > 1:
                                    ________return 1
                                    ____else:
                                    ________return 0

                                    In the codeblock below write:

                                    inValue(!AnalysisField!)

                                    Hope it works now!

                                    Hugo
                                    • Re: ArcGIS 10 Pre-Release finding duplicates
                                      jsnider@wwfcanada.org
                                      For finding duplicate records using ArcGIS 10, this Python code might be slightly faster than those codes listed above, at least for large datasets, since it only passes through the list once to complete the value count rather than looping to count for each item in the list...

                                      import arcpy 
                                      from collections import defaultdict
                                      
                                      inft = r"C:\GIS\yourshape.shp"
                                      inField = "Your_field"
                                      
                                      value_list = []
                                      rows = arcpy.SearchCursor(inft)
                                      for row in rows:
                                      [INDENT]value = row.getValue(inField)[/INDENT]
                                      [INDENT]value_list.append(value)[/INDENT]
                                      
                                      duplicate_list = []
                                      dict1 = defaultdict(int)
                                      for value in value_list:
                                      [INDENT]dict1(value) +=1[/INDENT]
                                      
                                      for key, val in dict1.iteritems():
                                      [INDENT]if val > 1:[/INDENT]
                                      [INDENT][INDENT] dup_list.append(key)[/INDENT][/INDENT] 
                                      
                                      print "duplicate list length = ", len(duplicate_list)
                                      
                                      
                                      • Re: ArcGIS 10 Pre-Release finding duplicates
                                        mike.weaver
                                        For finding duplicate records using ArcGIS 10, this Python code might be slightly faster than those codes listed above, at least for large datasets, since it only passes through the list once to complete the value count rather than looping to count for each item in the list... 

                                        How do I get this to work using the Calculate field tool.  I'm kinda new to python and running these scripts.  Is there something i need to put in the in the box below the pre-logic code box?

                                        I want to use the script you posted  because I am working with large datasets and need it run faster.

                                        Thanks in advance.
                                        • Re: ArcGIS 10 Pre-Release finding duplicates
                                          jsnider@wwfcanada.org
                                          Hi Mike -- I'm not positive that the Calculate Value tool is the best solution here, but I that will depend on what exactly you are trying achieve -- for instance, are you trying to calculate a value for a field based on the number of times that value occurs?

                                          The Python code I've posted above it designed to be run either in the Python window within ArcGIS 10 or from an IDLE GUI interface for Python.

                                          The output of the script, as it is currently written, is a list of values that are duplicates. So, in order for this to be useful for a calculation, you would likely have to refer to the items within the list.

                                          Maybe if you provide a better sense of your task, perhaps then I, or someone else here, can provide a more constructive reply.

                                          Thanks,
                                          James
                                          • Re: ArcGIS 10 Pre-Release finding duplicates
                                            mike.weaver
                                            Thanks for the quick reply.  What I'm trying to do is remove duplicate telephone records in a database.  So if I get a database with 600k records about 20% of those are duplicates.  I can de-dup this database in MS access easily and quickly.  But I'm trying to incorporate this de-dup process in a model.  I've tried your script in the Python window in ArcGIS 10 and keep getting an error:  Runtime error <type 'exceptions.SyntaxError'>: illegal expression for augmented assignment (<string>, line 13)

                                            I've used other code that will flag the duplicates, but have proven to be very time consuming comparative to the append query method in MS Access.
                                            • Re: ArcGIS 10 Pre-Release finding duplicates
                                              jsnider@wwfcanada.org
                                              Mike, if it were me, I would proceed by using the code as written, using the Python window in Arc 10.

                                              From there, I would use an updateCursor to delete the duplicate row from the table.

                                              More on updateCursors, including the `deleteRow`expression here:
                                              http://help.arcgis.com/en/arcgisdesktop/10.0/help/index.html#//002z0000001q000000

                                              However in doing so, you need to be careful not to delete the value entirely -- i.e. make sure to retain one instance of each value in the duplicate list.

                                              Unfortunately, I`m not much help in terms of integrating this into a modelbuilder-based approach.
                                              • Re: ArcGIS 10 Pre-Release finding duplicates
                                                erink

                                                Any insight on modifying it to mark the duplicates with an incremental number per occurrence?



                                                DavidBrett,

                                                Python code block:
                                                UniqueDict = {}
                                                def isDuplicateIndex(inValue):
                                                    UniqueDict.setdefault(inValue,0)
                                                    UniqueDict[inValue] += 1
                                                    return UniqueDict[inValue]


                                                Expression:
                                                isDuplicateIndex( !YOUR_FIELD! )


                                                Hope that helps, Erin.
                                                • Re: ArcGIS 10 Pre-Release finding duplicates
                                                  davidbrett
                                                  Thanks Erin! This is exactly what I need.
                                                  • Re: ArcGIS 10 Pre-Release finding duplicates
                                                    mbd7878
                                                    This is great!  Can you help me put this in a stand alone python script?  Everything runs great until the Calculate Field management...thats where it fails.  Can anyone see where I am going wrong?

                                                    fclayer = arcpy.mapping.Layer(C:/data/BaseData.gbd/myLayer)
                                                    fieldName = "Duplicate"
                                                    expression2 = "isDuplicate(!soil_desc!)"
                                                    uniqueList = []
                                                    codeblock = """def isDuplicate(inValue):
                                                             (INDENT)if inValue in uniqueList:
                                                                 (INDENTx2)return 1
                                                             (INDENT)else:
                                                                 (INDENTx2)uniqueList.append(inValue)
                                                                 (INDENTx2)return 0"""

                                                    arcpy.AddField_management(myLayer,fieldName,"SHORT")

                                                    arcpy.CalculateField_management(myLayer,fieldName,expression2,"PYTHON",codeblock)
                                                    • Re: ArcGIS 10 Pre-Release finding duplicates
                                                      davidbrett
                                                      The attached will increment duplicates in a user specified column.  It is configured to run from ArcToolbox.  You can hard code the variable if necessary.
                                                      • Re: ArcGIS 10 Pre-Release finding duplicates
                                                        mbd7878
                                                        I get the following error.

                                                        <type 'exceptions.TypeError'>: 'Layer' object is unsubscriptable
                                                        Failed to execute (Script)

                                                        any ideas why?

                                                        Got it figured out.   Here is worked for me.

                                                        dupField = "soil_desc"
                                                        markField = "Duplicate"

                                                        arcpy.AddField_management(soilssortlayer,markField,"FLOAT")

                                                        rows = arcpy.UpdateCursor(soilssortlayer,"","","",dupField)

                                                        mylist = []

                                                        i = -1
                                                        for row in rows:
                                                            if i == -1:
                                                                value = row.getValue(dupField)
                                                                mylist.append(value)
                                                            i += 1
                                                            if row.getValue(dupField) != value:
                                                                value = row.getValue(dupField)
                                                                mylist.append(value)
                                                                i = 0
                                                            row.Duplicate = i
                                                            rows.updateRow(row)
                                                        • Re: ArcGIS 10 Pre-Release finding duplicates
                                                          biniamts
                                                          Hi there,
                                                          I am wandering for your help. I would like to copy selected rows of an attribute table based on the values of a single field. and duplicate (paste) the copied rows in  the attribute table itself with some editing afterwards on the object id for example. can some one provide me information(idea ) on how to put it in python script. 
                                                          thanx
                                                          B. Weldmeichael
                                                          • Re: ArcGIS 10 Pre-Release finding duplicates
                                                            mbd7878
                                                            feature = "C://Data//Feature"
                                                            newField = "NEWField"

                                                            # Create the new Field
                                                            newField = arcpy.AddField_management(feature,newField,"TEXT","50")

                                                            # Do your Select by Attribute normally
                                                            expression = "your select by attribute expression"
                                                            arcpy.SelectLayerByAttribute_management(feature,"NEW_SELECTION",expression)

                                                            # Calculate the new field
                                                            arcpy.CalculateField_management(feature,newField,"!OLDField!","PYTHON")


                                                            I am new to these Scripts, so I am not sure if when you calculate the new field it will honor those just selected or if it will update all the rows, but I think it will work.  Depending on how complex the "some editing afterwards" is, but if it is simple you can add onto your expression on the Calculation (the !OLDField! part).
                                                            This has not been tested, and like I said I am a beginner, but that should give you a place to start.
                                                            • Re: ArcGIS 10 Pre-Release finding duplicates
                                                              CJCarsley
                                                              Hello!

                                                              I am looking to find duplicate attributes in my shapefile as shown in this forum.  However, when I try the Python code example I get the following error message: "Error 000539: Error running expression: uniqueList = Error running expression: uniqueList = [] "

                                                              Here is the code that I entered:

                                                              uniqueList = [ !Duplicate! ]
                                                              def isDuplicate(inValue):
                                                              if inValue in uniqueList:
                                                                return 1
                                                              else:
                                                                uniqueList.append(inValue)
                                                                return 0

                                                              isDuplicate(!Calculate!)


                                                              ---

                                                              "Duplicate" is the field I want the results entered into, "Calculate" is the field that I want duplicates selected from.

                                                              Hopefully this is enough code to explain my situation. 

                                                              Thank you for your help in advance!

                                                              Sincerely,

                                                              Colter


                                                              I used this script to identify duplicates once, last week, and it worked perfectly.  For some reason, I'm trying to do it again this week, and I'm being consistently presented with a syntax error for line 1.  The only contrast between my two sets of data is size (some 43,000 records vs. 233,000).  Could this difference in size be enough to cause the script to fail?  I've noticed the process gets about 70%-80% complete before it 'fails to execute'.  I still get some results, but a large portion of the "Dup" field is populated with <null> values. 

                                                              Any guidance would be greatly appreciated!
                                                              • Re: ArcGIS 10 Pre-Release finding duplicates
                                                                pppbpnst
                                                                thank s to Mark its work..... and for JCMolina thanks for the step of the script...

                                                                this for Phyton, if anyone know script for the VBA please ...
                                                                • Re: ArcGIS 10 Pre-Release finding duplicates
                                                                  olcoil
                                                                  Hi Guys, I didn't want to start a new thread, instead I'll just post my problem here. Sorry if it seems like I am not understanding it all, our department is just starting to learn Python on ArcMap 10!

                                                                  I have 3 fields:
                                                                  1. "isDuplicate" (I want this field flag as 1 for duplicates and 0 for unique values)
                                                                  2. x (coordinate calculated by Field Calculator)
                                                                  3. y (coordinate calculated by Field Calculator)

                                                                  I want the script do something like this:

                                                                  For All_Objects
                                                                       If x is unique AND y is unique,
                                                                       isDuplicate = 0
                                                                       Else
                                                                       isDuplicate = 1
                                                                  End

                                                                  But I know that doesn't make much sense. Basically this script should identify all points that have the same X/Y coordinates and label the fields as such. There may be other easier ways to do this, but there are other functions that I will need to add to this script anyways.

                                                                  I just very new with Python sorry :(
                                                                  • Re: ArcGIS 10 Pre-Release finding duplicates
                                                                    mbd7878
                                                                    I think this will work, but I have not tested it.  Also there might be an easier way.  But what I did was create a new field that combines the values from the x & y fields (you can delete this new field at the end if you want).  Then search that one field for duplicates and add the values to the isDuplicate field.
                                                                    You also might need to make sure when you created isDuplicate it is a FLOAT field.



                                                                    ObjectLayer = "Name of Layer to be flagged"
                                                                    
                                                                    arcpy.AddField_management(ObjectLayer,"Working","TEXT","50")
                                                                    
                                                                    expression = "x + y"
                                                                    arcpy.CalculateField_management(ObjectLayer,"Working",expression,"PYTHON")
                                                                    
                                                                    dupField = "Working"
                                                                    Duplicate = "isDuplicate" 
                                                                    
                                                                    rows = arcpy.UpdateCursor(ObjectLayer,"","","",dupField)
                                                                    
                                                                    mylist = []
                                                                    
                                                                    i = -1
                                                                    for row in rows:
                                                                        if i == -1:
                                                                            value = row.getValue(dupField)
                                                                            mylist.append(value)
                                                                        i += 1
                                                                        if row.getValue(dupField) != value:
                                                                            value = row.getValue(dupField)
                                                                            mylist.append(value)
                                                                            i = 0
                                                                        row.isDuplicate = i
                                                                        rows.updateRow(row)
                                                                    
                                                                    • Re: ArcGIS 10 Pre-Release finding duplicates
                                                                      mbd7878
                                                                      ObjectLayer = "Name of Layer to be flagged"

                                                                      arcpy.AddField_management(ObjectLayer,"Working","TEXT","50")

                                                                      expression = "x + y"
                                                                      arcpy.CalculateField_management(ObjectLayer,"Working",expression,"PYTHON")

                                                                      dupField = "Working"
                                                                      Duplicate = "isDuplicate"

                                                                      rows = arcpy.UpdateCursor(ObjectLayer,"","","",dupField)

                                                                      mylist = []

                                                                      i = -1
                                                                      for row in rows:
                                                                          if i == -1:
                                                                              value = row.getValue(dupField)
                                                                              mylist.append(value)
                                                                          i += 1
                                                                          if row.getValue(dupField) != value:
                                                                              value = row.getValue(dupField)
                                                                              mylist.append(value)
                                                                              i = 0
                                                                          row.isDuplicate = i
                                                                          rows.updateRow(row)
                                                                      • Re: ArcGIS 10 Pre-Release finding duplicates
                                                                        davidbrett
                                                                        You can use the following in the field calculator

                                                                        Parser  = Python
                                                                        check 'Show Codeblock'

                                                                        In the Pre-Logic Script Code:
                                                                        uniqueList = []
                                                                        def isDuplicate(inValue):
                                                                          if inValue in uniqueList:
                                                                            return 1
                                                                          else:
                                                                            uniqueList.append(inValue)
                                                                            return 0


                                                                        in the Box below where it says FIELD_NAME =

                                                                        isDuplicate ( !FieldName1! + !FieldName2! )


                                                                        tested and works.
                                                                        • Re: ArcGIS 10 Pre-Release finding duplicates
                                                                          angelagotch
                                                                          Juan, that worked perfectly.  Thanks!  ~Angela
                                                                          • Re: ArcGIS 10 Pre-Release finding duplicates
                                                                            cgraves-esriaustralia-com-au-esridist
                                                                            Hi Guys,

                                                                            Further to this thread I would like select duplicate records in a field and then make these unique using the Field Calculator. For example I have parcel IDs (e.g. F555 A30) which might occur 3 times. I want to select these and change the name of these with either 'F555 A30 B' or 'F555 A30 1'. Also I would like the change to occur in the field the original records are in and not create a new field.

                                                                            Thanks in advance,

                                                                            Chris
                                                                            • Re: ArcGIS 10 Pre-Release finding duplicates
                                                                              pputri
                                                                              You can use the following in the field calculator

                                                                              Parser  = Python
                                                                              check 'Show Codeblock'

                                                                              In the Pre-Logic Script Code:
                                                                              uniqueList = []
                                                                              def isDuplicate(inValue):
                                                                                if inValue in uniqueList:
                                                                                  return 1
                                                                                else:
                                                                                  uniqueList.append(inValue)
                                                                                  return 0


                                                                              in the Box below where it says FIELD_NAME =

                                                                              isDuplicate ( !FieldName1! + !FieldName2! )


                                                                              tested and works.



                                                                              Hi David!

                                                                              I have the same issue with Peter and use your code. It works great actually when there are duplicate XY in the data. However, when there are no such data, the script's output highlighting records that has duplicate X value AND/OR duplicate Y value (screenshot attached).

                                                                              How to make a script that find only the absolute duplicate value of both X AND Y?

                                                                              Thank you. :o

                                                                              Prima.
                                                                              • Re: ArcGIS 10 Pre-Release finding duplicates
                                                                                davidbrett
                                                                                Hi David!

                                                                                I have the same issue with Peter and use your code. It works great actually when there are duplicate XY in the data. However, when there are no such data, the script's output highlighting records that has duplicate X value AND/OR duplicate Y value (screenshot attached).

                                                                                How to make a script that find only the absolute duplicate value of both X AND Y?

                                                                                Thank you. :o

                                                                                Prima.



                                                                                Hi,

                                                                                Let's try another way
                                                                                uniqueList = []
                                                                                i = 0
                                                                                def isDuplicate(inValue):
                                                                                    global i
                                                                                    if inValue in uniqueList:
                                                                                        i += 1
                                                                                        return i
                                                                                    else:
                                                                                        uniqueList.append(inValue)
                                                                                    return 0
                                                                                


                                                                                This will increment the dup field per the matching duplicates. You can test how this works by using one inValue field at a time, and then both.

                                                                                Example 1:
                                                                                dup =
                                                                                isDuplicate( !x!)

                                                                                Example 2:
                                                                                dup =
                                                                                isDuplicate( !y!)

                                                                                Example 3:
                                                                                dup =
                                                                                isDuplicate( !x! + !y!)
                                                                                • Re: ArcGIS 10 Pre-Release finding duplicates
                                                                                  davidbrett
                                                                                  Hi Guys,

                                                                                  Further to this thread I would like select duplicate records in a field and then make these unique using the Field Calculator. For example I have parcel IDs (e.g. F555 A30) which might occur 3 times. I want to select these and change the name of these with either 'F555 A30 B' or 'F555 A30 1'. Also I would like the change to occur in the field the original records are in and not create a new field.

                                                                                  Thanks in advance,

                                                                                  Chris


                                                                                  Hi Chris,

                                                                                  The safest way to do this is to create a new field and not update the field you are calculating during the calculation...logic: don't  change the data your analyzing while analyzing the data.

                                                                                  You can use the code I just posted to increment the dup field, then calculate the new id based on the old ID + dup value.
                                                                                  • Re: ArcGIS 10 Pre-Release finding duplicates
                                                                                    rblash
                                                                                    Has anyone figured out how to simply select the duplicates without creating a new field or using summarize?

                                                                                    Thanks,

                                                                                    Rob
                                                                                    • Re: ArcGIS 10 Pre-Release finding duplicates
                                                                                      rblash
                                                                                      Take a look at this add-in. Someone recycled the tool we were using in 9.X, seems to be reliable so far.

                                                                                      http://www.arcgis.com/home/item.html?id=2f8c0a2477db465b8ca2a9f7f519eb5a
                                                                                      • Re: ArcGIS 10 Pre-Release finding duplicates
                                                                                        random_child
                                                                                        Good Morning!  Is there a way this can this be done using the field calculator with a geodatabase feature class?  It isn't working on my end...  Thanks.

                                                                                        #Here's a script that flags duplicates as "1" including the first occurence:
                                                                                        #It works fine for strings and integers, but doesn't work for float values, to overcome the float
                                                                                        #problem convert the values to strings.
                                                                                        #Paste the following code in the Field Calculator:

                                                                                        import arcpy
                                                                                        inFeatures = "C:\\Folder\\Subfolder\\yourshapefile.shp"

                                                                                        inField = "YOUR_FIELD" #The field you want to analyse
                                                                                        rows = arcpy.SearchCursor(inFeatures)
                                                                                        row = rows.next()
                                                                                        ValueList = list()
                                                                                        while row:
                                                                                            currentValue = row.getValue(inField)
                                                                                            ValueList.append(currentValue)
                                                                                            row = rows.next()

                                                                                        def isDuplicate(inValue):
                                                                                            if ValueList.count(inValue) > 1:
                                                                                                return 1
                                                                                            else:
                                                                                                return 0


                                                                                        ##In the box below, paste:
                                                                                        ## isDuplicate(!YOUR_FIELD!)
                                                                                        • Re: ArcGIS 10 Pre-Release finding duplicates
                                                                                          drewoakley
                                                                                          Is it possible to give weight to a feature with this code? Basically I have two sets of data that I have combined.  One comes from GPS points and the other comes from geocoded points.  I have a field named 'info_src" that distinguishes this for me.  I want to erase the duplicate geocoded points NOT the GPS points.  The script I have been using is pasted below.  I want the duplicates marked but I want the geocoded point to be marked as the duplicate instead of the GPS point and just the opposite is occuring.  Help!! :)

                                                                                          uniqueList = []
                                                                                          def isDuplicate(inValue):
                                                                                            if inValue in uniqueList:
                                                                                              return 1
                                                                                            else:
                                                                                              uniqueList.append(inValue)

                                                                                          isDuplicate( !Account! )
                                                                                          • Re: ArcGIS 10 Pre-Release finding duplicates
                                                                                            iraida94
                                                                                            I'm a Python beginner and there a couple things I don't understand in the script below. 

                                                                                            1. What does invalue mean?
                                                                                            2. What is .setdefault(invalue,0)?
                                                                                            3. What does += mean?


                                                                                            Thanks


                                                                                            DavidBrett,

                                                                                            Python code block:
                                                                                            UniqueDict = {}
                                                                                            def isDuplicateIndex(inValue):
                                                                                                UniqueDict.setdefault(inValue,0)
                                                                                                UniqueDict[inValue] += 1
                                                                                                return UniqueDict[inValue]


                                                                                            Expression:
                                                                                            isDuplicateIndex( !YOUR_FIELD! )


                                                                                            Hope that helps, Erin.
                                                                                            • Re: ArcGIS 10 Pre-Release finding duplicates
                                                                                              davidbrett
                                                                                              There's a function named isDuplicateIndex and it takes one parameter. The name of that parameter = InValue.

                                                                                              Using Field Calculator, Expression:
                                                                                              Code:

                                                                                              isDuplicateIndex( !YOUR_FIELD! )

                                                                                              isDuplicateIndex is the name of function. InValue is a variable and is set to whatever is in the parenthesis: !YOUR_FIELD!

                                                                                              the .setdefault method ...set's the default value of the dictionary. You can read about that here http://docs.python.org/2/library/stdtypes.html

                                                                                              setdefault(key[, default])If key is in the dictionary, return its value. If not, insert key with a value of default and return default. default defaults to None.

                                                                                              += is an increment:  x = x + 1 is the same as x += 1.
                                                                                              • Re: ArcGIS 10 Pre-Release finding duplicates
                                                                                                wipark93
                                                                                                I have a question and would like some input on how to go on with my concept

                                                                                                So I used the Python code provided above and it works well.
                                                                                                What I am curious as to doing is:
                                                                                                --  Is there a way to create a script / something that is continuous to check for duplicates
                                                                                                -- Have some kind of "validation rule" that will give you an error/warning  message if you enter a "duplicate value" that is already within the database. (or attributes table feature list)

                                                                                                Example


                                                                                                DC_ID : 1
                                                                                                DC_ID : 2
                                                                                                DC_ID : 3
                                                                                                DC_ID : 4
                                                                                                DC_ID : null <---- It will warn you if you put any value for 1-4 within this field.
                                                                                                • Re: ArcGIS 10 Pre-Release finding duplicates
                                                                                                  hcopeland
                                                                                                  Hi Kent,
                                                                                                  The code below is what I'm looking for, with the exception that I want to find duplicates across 4 fields... in my case, Latitude, Longitude, BirdID and DateTime. Can you give me pointers for how to make that modification? Thanks much, Holly

                                                                                                  ====================
                                                                                                  Using the Field Calculator in ArcGIS 10.  You can use Python to easily find duplicate values in a given field.  In Python you can declare a list outside of the function so that it can retain added values.  Here is a code sample:

                                                                                                  uniqueList = []
                                                                                                  def isDuplicate(inValue):
                                                                                                    if inValue in uniqueList:
                                                                                                      return 1
                                                                                                    else:
                                                                                                      uniqueList.append(inValue)
                                                                                                      return 0


                                                                                                  At the bottom of the Field Calculator, you will have to enter in the function and field name like this:
                                                                                                  isDuplicate(!FIELD_NAME!)


                                                                                                  Thanks for your post.

                                                                                                  Kent M.
                                                                                                  esri[/QUOTE]