What time is it? Well That Depends...
When someone asks you, "What time is it?", you are probably assuming he or she wants to know the local time where the two of you are right now. As I write this, the time now is Tuesday, March 12, 2019 at about 2:25 PM in Redlands, California, USA.
Typically, we do not qualify our answers so explicitly. We say "It's 2 o'clock" and assume it's understood that this is the time right now in Redlands, California. But that is sort of like answering a query about length or distance by simply saying "36". Is that feet, meters, miles, or kilometers?
Last weekend, here in California, we set our clocks ahead one hour to honor daylight savings time (DST). California is now observing Pacific Daylight Time (PDT) which is equal to UTC-7:00 hours. When we specify the time at which an event was observed, we should include the time zone in which the observation is made as well as whether or not the time reflects a local convention honoring daylight savings time.
When ArcGIS GeoEvent Server receives data for processing, event records usually include a date/time value with each observation. Often the date/time value is expressed as a string and does not specify the time zone in which the date/time is expressed or whether the value reflects a daylight savings time offset. These are sort of like the "units" (e.g. feet, meters, miles, or kilometers) which qualify a date/time value.
The intent of this blog is to identify when GeoEvent Server assumes a date/time value is expressed in Coordinated Universal Time (UTC) versus when it is assumed that a date/time expresses a value consistent with the system's locale. We'll explore a couple situations where this might be important and the steps you can take to configure how date/time values are handled and displayed.
Event data ingest should generally assume date/time values are expressed as UTC values
There are several reasons for this. In the interest of brevity, I'll simply note that GeoEvent Server is running in a "server" context. The assumption is that the server machine is not necessarily located in the same time zone as the sensors from which it is receiving data and that clients interested in visualizing the data are likewise not necessarily in the same time zone as the server or the sensors. UTC is the time standard commonly used around the world. The world's timing centers have agreed to synchronize, or coordinate, their date/time values -- hence the name Coordinated Universal Time.(1)
If you have ever used the ArcGIS REST Services Directory to examine the JSON representation of feature records which include a date/time field whose data type is esriFieldTypeDate, you have probably noticed that the value is not a string, it is a number; an epoch long integer representing the number of milliseconds since the UNIX Epoch (January 1, 1970, midnight). The default is to express the value in UTC.(2)(3)
When does GeoEvent Server assume the date/time values it receives are UTC values?
Out-of-the-box, GeoEvent Server supports the ISO 8601 standard for representing date/time values.(4)
It is unusual, however, to find sensor data which expresses the date/time value "March 12, 2019, 2:25:30 pm PDT" as 2019-03-12T14:25:30-07:00. So when a GeoEvent Definition specifies that a particular attribute should be handled as a Date, inbound adapters used by GeoEvent Server inputs will compare received string values to see if they match one of a few commonly used date/time patterns.
For example, GeoEvent Server, out-of-the-box, will recognize the following date/time values as Date values:
- Tue Mar 12 14:25:30 PDT 2019
- 03/12/2019 02:25:30 PM
- 03/12/2019 14:25:30
- 1552400730000
When one of the above date/time values is handled, and the input's Expected Date Format parameter does not specify a Java SimpleDateFormat expression / pattern, GeoEvent Server will assume the date/time value represents a Coordinated Universal Time (UTC) value.
When will GeoEvent Server assume a date/time value is expressed in the server machine's locale?
When a GeoEvent Server input is configured with a Java SimpleDateFormat expression / pattern the assumption is the input should convert date/time values it receives into an epoch long integer, but treat the value as a local time, not a UTC value.
For example, if your event data represents its date/time values as "Mar 12 2019 14:25:30" and you configure a new Receive JSON on a REST Endpoint input to use the pattern matching expression MMM dd yyyy HH:mm:ss as its Expected Date Format property, then GeoEvent Server will assume the event record's date/time expresses a value consistent with the system's locale and will convert the date/time to the long integer value 1552425930000.
You can use the EpochConverter online utility to show equivalent date/time string values for this long integer value. Notice in the illustration below that the value 1552425930000 (expressed in epoch milliseconds) is equivalent to both the 12th of March, 2019, at 9:25 PM Greenwich Mean Time (GMT) and 2:25 PM Pacific Daylight Time (PDT):
.png)
The utility's conversion notes that clocks in my time zone are currently seven hours behind GMT and that daylight savings time is currently being observed. You should note that while GMT and UTC are often used interchangeably, they are not the same.(5)
What if I have to use a SimpleDateFormat expression, because my date/time values are not in a commonly recognized format, but my client applications expect date/time values will be expressed as UTC values?
You have a couple of options. First, if you have the ability to work with your data provider, you could request that the date/time values sent to you specify a time zone as well as the month, day, year, hour, minute, second (etc.).
For example, suppose the event data you want to process could be changed to specify "Mar 12 2019 14:25:30 GMT". This would enable you to configure a Receive JSON on a REST Endpoint input to use the pattern matching expression MMM dd yyyy HH:mm:ss zzz as its Expected Date Format property since information on the time zone is now included in the date/time string. The input will convert the date/time string to 1552400730000 which is a long integer equivalent of the received date/time string value.
Using the EpochConverter online utility to show the equivalent date/time string values for this long integer value, you can see that the Date value GeoEvent Server is using is a GMT/UTC value:
.png)
If the data feed from your data provider cannot be modified you can use GeoEvent Server to compute the proper UTC offset for the ingested "local" date/time value within a GeoEvent Service.
Because GeoEvent Server handles Date attribute values as long integers, in epoch milliseconds, you can use a Field Calculator to add (or subtract) a number of milliseconds equal to the number of hours you need to offset a date/time value to change its representation from "local" time to UTC.
The problem, for a long time, was that you had to use a hard-coded constant value in your Field Calculator's expression which rendered your GeoEvent Service vulnerable twice a year to time changes if your community started and later stopped observing daylight savings time. Beginning with the ArcGIS GeoEvent Server 10.5.1, the Field Calculator supports a new wrapper function that helps address this: currentOffsetUTC()
A Field Calculator, running within a GeoEvent Service on my local server, evaluates currentOffsetUTC() and returns the value -25200000, the millisecond difference between my local system's current date/time and UTC. Currently, here in California, we are observing Pacific Daylight Time (PDT) which is equal to UTC-7:00.
Even though GeoEvent Server assumes date/time values such as "Mar 12 2019 14:25:30" (received without any time zone "units") represent local time values -- because a pattern matching expression MMM dd yyyy HH:mm:ss must be used to interpret the received date/time string values -- I was able to calculate a new date/time value using a dynamic offset and output a value which represents the received date/time as a UTC value. All I had to do was route the event record, with its attribute value ReportedDT (data type: Date) through a Field Calculator configured with the expression: ReportedDT + currentOffsetUTC()
How do I configure a web map to display local time rather than UTC time values
When recommending that date/time values should generally be expressed as UTC values, a frequent complaint when feature records updated by GeoEvent Server are visualized on a web map, is that the web map's pop-up values show the date/time values in UTC rather than local time.
It is true that, generally, we do not want to assume that a server machine and sensor network are both located in the same time zone as the localized client applications querying the feature record data. That does not mean that folks in different time zones want to perform the mental arithmetic needed to convert a date/time value displayed by a web map's pop-up from UTC to their local time.
In the past I have recommended data administrators work around this issue using a Field Calculator to offset the date/time, as I've shown above, by a number of hours to "falsely" represent date/time values in their database as local time values. I say "falsely" because most map/feature services are not configured to use a specified time zone. For a long time it wasn't even possible to change the time zone a map/feature service used to represent its temporal data values. There are web pages in the ArcGIS REST API which still specify that feature services return date/time values only as epoch long integers whose UTC values represent the number of milliseconds since the UNIX Epoch (January 1, 1970, midnight). So even if a map/feature service is configured to use a specific time zone, we should not expect all client applications to honor the service's specification.
For now, let's assume our published feature service's JSON specification follows the default and client apps expect UTC values returned when they query the map/feature service. If we use GeoEvent Server to falsely offset the date/time values to local time, the data values in our geodatabase are effectively a lie. Sure, it is easy to say that all client applications have been localized, and assume all server machines, client applications, and reporting sensors are all in one time zone; all we are trying to do is get a web map to stop displaying date/time values in UTC.
But there is a better way to handle this problem. Testing the latest public release (10.6.1) of the Enterprise portal web map and ArcGIS Online web map I found that pop-ups can be configured with custom expressions which dynamically calculate new values from existing feature record attributes. These new values can then be selected as the attributes to show in a web map's pop-up rather than the "raw" values from the feature service.
Below are the basic steps necessary to accomplish this:
- In your web map, from the Content tab, expand the feature layer's context menu and click Configure Pop-up.
- On the lower portion of the Configure Pop-up panel, beneath Attribute Expressions, click Add.
- Search the available functions for date functions and build an expression like the one illustrated below.
Assign the new custom attribute a descriptive name (e.g. localDateTime) and save the attribute calculation. You should now be able to select the dynamic attribute to display along with any other "raw" attributes from the feature layer.
References:
(1) UTC – Coordinated Universal Time
(2) ArcGIS for Developers | ArcGIS REST API
(3) ArcGIS for Developers | Common Data Types | Feature object
(4) World Wide Web Consortium | Date and Time Formats
(5) timeanddate.com - The Difference Between GMT and UTC
(6) ArcGIS for Developers | ArcGIS REST API | Enterprise Administration | Server | Service Types