Most of the time in GeoEvent we are trying to get the events out the door as fast as possible. But in a few rare cases, we need to pause for a moment. This blog details a couple of event processing options that delay events for a specific benefit.
Note: The custom processors mentioned here were created by Esri staff, but are considered custom components and not supported officially by the GeoEvent product team. If you like these processors, and want to see them included either in the GeoEvent gallery or directly in the GeoEvent product, please create an Idea to vote on and leave a comment about it on this article. If you find an bug or want to request an enhancement to either of these custom processors, please create an issue on the respective GitHub page.
Delay Processor
As I said above, most of the time we want events out the door as fast as possible. But sometimes, it helps to delay things a bit. For example, a public facing map of street sweepers or snow plows will show anyone that is interested exactly where your vehicles are. But for various reasons, it may not be a good idea to display their exact location. Instead, you might want to show the public where those vehicles were 15 minutes ago. This strategy balances safety of your drivers and assets with the need for letting the public know the current status of a situation.
For this scenario, I would use the Delay Processor published here:
https://github.com/mshareghi/geoevent-delay-processor/releases/tag/v1.0
This processor takes a delay time (in practically any time unit you might need) and adds it to the received time of each event. The events are then put into a delay queue that will release them from the processor once they've waited out their delay. In our example above, if you introduce the Delay Processor just before writing your data out to that public Feature Layer on ArcGIS Online, you can set it to delay each event by 15 minutes.
Temporal Ordering Processor
In this scenario, you might be polling a data provider that returns location events out of temporal order. The most common example of this is:
Delayed/Batch reporting - A vehicle may go offline for many hours (maybe it is visiting remote facilities outside the range of cell coverage). During this offline time, the device in the vehicle may cache location data until it re-connects to the cellular network. The slug of location events provided by the AVL API may be in any order.
In this situation, it is impossible to do any reasonable stateful analysis on the data (e.g. "What time did the vehicle enter/exit a facility?"). Wouldn't it be great if we could tell GeoEvent to temporally order these events and then release them in the correct order and time space?
An updated Delay processor created by me can do just that:
https://github.com/eironside/geoevent-delay-processor/releases/tag/1.4
In this case, the delay must be set to the longest period of time your data might be out of order by. In the example case above, the vehicles might have been offline for as much as 8 hours. So the delay time would be set to 8 hours (or maybe 9 just to be sure we've got everything before proceeding with processing). However in other cases, your data may only be out of order by a few minutes, so in those cases you can definitely reduce the delay down to something like 3 or 4 minutes. Hopefully, an accurate analysis of data is more important than the introduced delay.
Note: For this scenario, the other properties included with this processor should be set as follows: Allow Duplicates? Yes, Include TRACK_ID? No
Unique Temporal Ordering Processor
Finally, you may have a data provider that is giving a batch of data that covers a specified time span (e.g. last 60 seconds of data). In most of these cases, there's no guarantee the data will be in the correct temporal order or even unique (you may end up getting the same event over and over again). An example of this might be:
A data provider API only allows you to request data for the last 5 minutes - You can request the data more frequently, but it will always return the last 5 minutes of location information for each vehicle. If each vehicle is reporting every 30 seconds, there may be as many as 10 events per vehicle in each response (or there may be none if the vehicle is off). The returned data from each request may or may not be temporally ordered correctly.
In this example, we see a similar problem to the Temporal Ordering Processor above with an added twist. Each batch of data could have as many as 9 repeat events. In this case, it would be good to be able to identify the repeat events and discard them since they are already in our delay queue.
Use the updated Delay processor created by me but tell it to discard duplicate events:
https://github.com/eironside/geoevent-delay-processor/releases/tag/1.4
For this example, you would set the delay to be something slightly larger than your polling rate (e.g. if you are polling every 90 seconds, set the delay to 120 seconds). You would then set Delay Time Field to TIME_START, Allow Duplicates? to No and Use TRACK_ID? to Yes. Each event record's unique key would then become "<timestamp>_<trackid>". Any duplicate events with a key that matches an event already processed would be dropped (namely the same TRACK_ID and TIME_START).
GeoEventServer-DelayProcessor-r1.4.zip
