4 Replies Latest reply on Dec 5, 2012 9:00 AM by kevinmorrisgis

    Descripencies between Attribute Table (arcmap) datetime and the value read by arcpy.

      I have recently observed descripencies in time between what is shown in the attribute table in arcmap and what is obtained from reading the date field in python. In particular approximately 1 out of 10 time elements differ by one second. This difference prevents use of the datetime field for identification of duplicates.

      Flow: JSON serialized datetime (1) -> python datetime obj (2) -> written to feature class as obj (3) -> read from feature class (type = date) as obj(4).

      In the process the time element is consistent from step 1 to 3, however, the read time in (4) may be one second less than observed in the attribute table.

      Code to match and/or convert JSON to ESRI 10.0(python) datetime (and others):

      def MatchDataType(ESRI_Cursor_Object, JSON_Data_Structure):
          DType = ESRI_Cursor_Object.type
          DName = ESRI_Cursor_Object.name
          JData = JSON_Data_Structure[DName]

          if DType == 'SmallInteger':
              return int(JData)
          if DType == 'Integer':
              return int(JData)
          if DType == 'SHORT':
              return int(JData)
          if DType == 'LONG':
              if long(Jdata) < 2147483647:  # Had to add this due to GeoPro 16 digit Integer range greater than ESRI accomodates with LONG
                  return long(JData)
                  msgr('Conversion to long above ESRI limit for long integer')
          if DType == 'Single':
              return int(JData)
          if DType == 'Double':
              return long(JData)
          if DType == 'String':
              return str(JData)
          if DType == 'Date':
              if JData.find('Date('):     # Java JSON date is expressed as milliseconds since IBM inception Date(XXXXXXXXXXXXX-YYYY) X = milliseconds Y=offset (positive or negative?)
                                          # eg. /Date(1348860772000-0400)\  Note that if you do a find on '/Date(' it can fail! Some of my tests were successful others not.
                  try:                    # poor man's error checker in case the date stamp is not correct. In this case return nothing
                      index = int(JData.find('Date('))
                      JData_milliseconds = int(JData[index + 5:index + 5 + 13])       # rip out the milliseconds in the proper placement starting from the location of finding the match
                      JData_hourshift = int(JData[index + 5 + 13:index + 5 + 13 + 5])
                      dt = datetime.datetime.fromtimestamp(round(JData_milliseconds/1000))  # need to get rid of the tailing bits of real info due to conversion to ESRI date that round this information differently.
                      dtn = dt - datetime.timedelta(hours=round(JData_hourshift/100))        # not sure if I should be subtracting/adding this hour shift or what exactly this means. Only want the hour not minutes.                      
                      return dtn                                                     # may need to only convert the -04 to integer, need to verify dates against the GeoPro webpage.

      The code writing the datetime element uses a regular insert cursor and converts the JSON data (Gmsg) as a python dictionary list into a matching field in the feature class based upon the feature class field type (predefined).
      # Create the ESRI point geometry
                                  inPoint = arcpy.Point(SEKI_Location['Longitude'],SEKI_Location['Latitude'])
                                  newIncident = rowInserter.newRow()

                                  # Populate Shape Attributes
                                  newIncident.SHAPE = inPoint
                                  # Populate matching fields.
                                  for field in fieldList:         # loop through the Feature class attribute names
                                      if field.name in Gmsg.keys():
                                          print field.name
                                          newIncident.setValue(field.name, MatchDataType(field, Gmsg))
                                  # add attribute stuff here
                                  # newIncident.setValue(descriptionField, inDescription)

      It can then be read lader and matched, however the times may be off by one second.
      This is observed as a discripency in the print statement. Approximately one out of 10 times will be off by one second

      def IsDuplicateEntry(GeoPro_fc, ComparisonList,FieldList, GeoProMessageObject):
          # SCursor=SearchCursor.reset()      unsupported                                        # reset the search cursor in the feature class for checking the next entry

          Duplicates = 0
          SCursor = arcpy.SearchCursor(GeoPro_fc)

              for row in SCursor:                                                      # Iterate through all the rows to find a match if there is one
                  Duplicates = 0
                  for ComparisonName in ComparisonList:
                      if ComparisonName == "EventDate":
                          print row.getValue(ComparisonName), ' --- ', MatchDataType(GetESRIFieldObject(ComparisonName, FieldList), GeoProMessageObject)                              
                      if not(row.getValue(ComparisonName) == MatchDataType(GetESRIFieldObject(ComparisonName, FieldList), GeoProMessageObject)):
                          continue                                                    # if there is any non match we don't need to keep checking
                          Duplicates = Duplicates + 1
                  if Duplicates == len(ComparisonList):                                # if all the comparison fields have duplicates then this is a duplicate record and pass a True message
                      #del row
                      #del SCursor
                      return True                                                     # This record is a duplicate - no need to keep searching through the feature class.

              #del row
              #del Scursor
              return False
              #del row
              #del SCursor
              return                                                            # If there is an unusual error just pass the duplicate found flag.