Poll an ArcGIS Server for Features: Methods used to identify incremental updates

This article provides additional information and examples for the Poll an ArcGIS Server for Features
input's Method to Identify Incremental Updates parameter. Please refer to the input's on-line help topic
for usage notes, a description of the input's other parameters, as well as considerations and limitations
of this input.

See Also:
- Polling feature services for "Incremental Updates" (Updated August 2023)
- Question regarding "Incremental Update" workarounds, custom components?
- Using a partial GeoEvent Definition to update feature records

The Method to Identify Incremental Updates parameter has four options as described in the Parameters table in the input's on-line help topic:

ObjectID – GeoEvent Server will cache the greatest object identifier from the feature record set returned from a map/feature service poll. Only features whose object identifier is greater than the value cached from the last poll will be included in the next poll.
Timestamp since last-received to newest-feature timestamp – GeoEvent Server will cache the greatest date/time value from the feature record set returned from a map/feature service poll. Only feature records whose timestamp is greater than the value cached from the last poll will be included in the next poll.
Timestamp interval between last query time until now – GeoEvent Server will construct a temporal query with a lower-bound and upper-bound. The lower-bound will be the date/time the last query was executed. The upper-bound will be the date/time “now”. Only feature records with a timestamp within the temporal query’s range will be included in the next poll. The feature record timestamp is taken from a specified attribute field.
Timestamp interval between last query time with overlap until now – GeoEvent Server will construct a temporal query with a lower-bound equal to a specified number of seconds before the last query was executed and an upper-bound equal to the date/time “now”. Only feature records whose timestamp is within the temporal query’s range will be included in the next poll.

When using the Poll an ArcGIS Server for Features input’s Get Incremental Updates capability, if you configure GeoEvent Server logging to include DEBUG messages from the feature service inbound transport, you will see detailed messages which identify the query expression and/or temporal query GeoEvent Server constructs to limit which feature records are polled.

(Option) - Timestamp since last-received to newest-feature timestamp

(A) 1=1 and last_updated > timestamp '1969-12-31 23:59:59'

(B) 1=1 and last_updated > timestamp '2023-08-07 13:29:00'

The logged message (A) indicates the input has no cached timestamp value. In addition to honoring the input’s default Query Definition 1=1, the input has constructed a query to include any feature record with a epoch date/time greater than “the beginning of time” defined as January 1st 1970 (Midnight UTC). This is expected to retrieve any feature records which a valid (non-null) date/time stamp.

The logged message (B) indicates that, of the feature records returned in the previous poll, the records with the greatest timestamp were those whose date/time attribute held a value “Aug 07 2023 13:29:00 UTC”. Only feature records greater than this value are included when polling using the cached timestamp.

(Option) - Timestamp interval between last query time until now

(A) 1=1 and last_updated >= timestamp '1969-12-31 23:59:59' and
last_updated < timestamp '2023-08-08 01:19:05'

(B) 1=1 and last_updated >= timestamp '2023-08-08 01:19:05' and
last_updated < timestamp '2023-08-08 01:20:06'

(C) 1=1 and last_updated >= timestamp '2023-08-08 01:20:06' and
last_updated < timestamp '2023-08-08 01:21:06'

The logged message (A) indicates that the input has no cached timestamp value. In addition to honoring the input’s default Query Definition 1=1, the input has constructed a query to include any feature record with a epoch date/time greater than “the beginning of time” defined as January 1st 1970 (Midnight UTC), but less than the current time “now” when the query was executed.

The logged message (B) indicates that the input last polled for feature records “Aug 08 2023 01:19:05 (UTC)” and has therefore assigned that as the lower-bound for the constructed temporal query. The upper-bound is set to the current time “now”. The expectation is that any feature records with a valid (non-null) date/time between the lower-bound and upper-bound will be retrieved.

The logged message (C) indicates that the input last polled for feature records “Aug 08 2023 01:20:06” and has therefore assigned that as the lower-bound for the constructed temporal query. The upper-bound is set to the current time “now”. The expectation is that any feature records with a valid (non-null) date/time between the lower-bound and upper-bound will be retrieved.

Reviewing these logged messages we recognize that the input is polling for input every 60 seconds. We recognize that the ArcGIS Server’s managed geodatabase, in this case, is rounding date/time values to the nearest second -- otherwise the logged messages would include millisecond values.

Also note that the upper-bound in the logged messages (A) and (B) differ by one second. There is inherent latency (some number of milliseconds) between the time GeoEvent Server determines the current time “now” (setting the temporal query’s upper-bound) and when a result is returned from the database. The input's polling is conducted approximately “every 60 seconds” depending on operational latency within GeoEvent Server (as data records are ingest, adapted, and processed) as well as between solution components (e.g. ArcGIS Server and ArcGIS Data Store).

(Option) - Timestamp interval between last query time with overlap until now

(A) 1=1 and last_updated >= timestamp '1969-12-31 23:59:59' and
last_updated < timestamp '2023-08-08 02:21:15'

(B) 1=1 and last_updated >= timestamp '2023-08-08 02:21:05' and
last_updated < timestamp '2023-08-08 02:22:45'

(C) 1=1 and last_updated >= timestamp '2023-08-08 02:22:35' and
last_updated < timestamp '2023-08-08 02:24:15'

The logged message (A) indicates that the input has no cached timestamp value. In addition to honoring the input’s default Query Definition 1=1, the input has constructed a query to include any feature record with a epoch date/time greater than “the beginning of time” defined as January 1st 1970 (Midnight UTC), but less than the current time “now” when the query was executed.

The logged message (B) indicates that the input last polled for feature records “Aug 08 2023 02:21:15 (UTC)”. The input, in this case, was configured with an additional offset parameter Timestamp overlap duration in seconds set to 10 seconds, so the input has constructed a temporal query with a lower-bound 10 seconds earlier than its last query. The computed lower-bound is “Aug 08 2023 02:21:05 (UTC)”. The constructed temporal query’s upper-bound is set to the current time “now”. Any feature records with a valid (non-null) date/time between the lower-bound (with its offset) and upper-bound will be retrieved using this query.

The logged message (C) indicates that the input last polled for feature records “Aug 08 2023 02:22:45 (UTC)” and set a lower-bound offset by the 10 seconds specified by the input’s Timestamp overlap duration in seconds parameter. The computed lower-bound is “Aug 08 2023 02:22:35 (UTC)”. The constructed temporal query’s upper-bound is set to the current time “now”. Any feature records with a valid (non-null) date/time between the lower-bound (with its offset) and upper-bound will be retrieved using this query.

Reviewing the logged messages in this third example we recognize that the input is polling for input approximately every 90 seconds -- there is a 90 second difference between the upper-bound values in each of the constructed temporal queries.

You can use the following GeoEvent Server component logger to configure logging to include DEBUG level messages which include the information illustrated above for the Poll an ArcGIS Server for Features input’s Get Incremental Updates capability:

com.esri.ges.transport.featureService.FeatureServiceInboundTransport