Do spatiotemporal data store layers have a spatial reference

[AndrewRudin1]

We just deployed GeoEvent, GeoAnalytics, and the Spatiotemporal Big Data Store (SBDS) deployed and I ran into an anomaly when while trying them out.  

In GeoEvent I'm reading in CSV data that has geometry in lat/long format, and then I want to store the results in a SBDS layer for use in analysis later.  On my first attempt, I chose the defaults when creating the SBDS layer in GeoEvent Manager, which means it is created as WGS84.  Once I got some sample data loaded, I tried to use GeoAnalytics to create space time cubes out of it, but I got the warning that the layer needs to be in a projected coordinate system. 

So, back to the drawing board.   I figured I needed to recreate the SBDS layer with a projected coordinate system (at my office we use one of the US State Plane formats). When I went back and tried to create a new SpatioTemporal Data Source, I saw there are options to define the coord system of the Map & Feature services attached to the data source, but no option to define the coord system of the underlying data.  I went ahead and tried this option to have the services in State Plane, but the result were services that were still in WGS 84, and when I look at the JSON results of the web service I see the coordinates are still lat/long.

So here are my questions?

1) Am I right in thinking I should store data in the SBDS in the same coordinate system as the rest of our organization to avoid "projection on the fly" processing overhead during analysis.

2) If so, how do I define the coord system of the spatiotemporal layer?  And then how do I get the lat/long feed to go into the data source properly?

Thanks for any help you can provide,

Andrew

[SarahAmbrose · Accepted solution]

Hi Andrew Rudin‌ - thanks for the question!

Currently - you're only able to store data in the spatiotemporal data store in WGS84 (wkid 4326). Tools in GeoAnalytics that require binning (Create Space Time Cube, Aggregate Points) require that your data be projected for those analyses. To do this we recommend that you project on the fly (like you mentioned). Compared to the rest of your big analysis, we haven't noticed that projecting adds to much time. 

I'm guessing you are at version 10.5? At 10.5.1 or greater, we will automatically project data being binned for you to World Equal Area projection if it's in a geographic coordinate system (we don't overwrite the projection choice if you supply one!). Depending on how you are running your analysis (Pro, Rest, Portal) you can specify the processing SR.

For example, if you were using the NAD_1927_StatePlane_Texas_ Central_FIPS_4203 projection, you would apply the WKID 32039 (I like to use this link to find the wkids, or look at the spatial reference options in Pro - outlined below).

To do this for each UI:

• In ArcGIS Pro, set the GP Environment settings by going to Analysis ribbon > Environments > Output Coordinate System > and paste in that WKID or browse for it. Then hit save. 
• In the portal Map Viewer, when you open up a tool, hit the "gear" icon and set the processing spatial reference there. 
• In REST you would set the context parameter like so {"processSR":{"wkid":32039}}

The results of your tool will either be written to:

• the spatiotemporal data store and saved in WGS84 (projected back)
• the relational data store and saved in the spatial ref you processed in
• a cube (which you download, not store it in a data store) in the spatial reference your processed in. 

Please let me know if you have any follow up questions or I missed anything,

Sarah

Product Engineer, GeoAnalytics Team

[AndrewRudin1]

Thank you Sarah,  That helps to know the projection I'm aiming for so I can design GeoEvent processes to do as little as possible.  Would hate to project to State Plane only to have the data auto-reprojected back.

I'm using fresh 10.6.1 servers and Pro 2.2.3.  I've now redeployed a default GeoEvent SBDS output layer and have some sample data populated.  When I try to create a space time cube from it using the GeoAnalytics toolbar in Pro, I see that it was just giving me a warning about the coordinate system and that the tool will apply the "World Cylindrical Equal Area" projection.  However, right after that it bombed out because the input layer is time interval based, and that is what caused it to error out. I pasted the results window text below.  

The data I'm using for my sample are city 311 service request tickets.  There are 500,000 records covering requests for all the various City services and we have history for the last 5 years, so it seemed like a good candidate to try spatiotemporal hosting on, I used Geoevent to create it since I can see us loading in tickets as they come in during every day.  The data has a few date fields in them that would be useful in various spatiotemporal analysis operations depending on what an analyst wants to look at. In my case those fields are created_date, last_status_update_date,  and close_date. When I defined the GeoEvent Definition for this data, I tagged the created_data as the START_TIME tag, and the closed_date as the END_TIME tag.  Therefore, I guess when the overlying feature service got created , it was then time-enabled as interval based with those two fields hard coded as start-end time values.  The space-time cube wants a dataset that is instant time...meaning only one field like created_date is used to time-enable.  

So as an analyst, if I stumble upon this amazing GeoEvent output layer in my Portal and I want to do spatiotemporal analysis on it, then how would I go about it?

For example, maybe I want to investigate "how many tickets were created in various parts of the city over space and time the last 5 years?" and then "is there a spatiotemporal variance in how long tickets are open in parts of the city". With how it's setup now I don't see how I can do it without extracting the SBDS data to my desktop or another hosted copy where I can redefine the time-enablement on a Pro layer or a separate nearly-identical feature service.  If I load the SBDS feature service into Pro, the time-enablement section of the properties is grayed-out so I can't redefine it there before creating the space-time cube.  I also don't see a way to alter the properties of the feature service in GeoEvent or Portal to tinker with the time-enable properties.  I think hosted views aren't an option for this kind of data either.

I really wish users could alter/override the time-enablement of web layers inside of applications (Pro, Portal map layers), rather than it being hard coded into service definitions. This would give analysts freedom to explore lots of different attributes from a single dataset, rather than being limited to whatever the data admin that initially created the web service picked.

Here is the output from the processing.

Parameters

Point Layer AustinOpenData_311UnifiedData\AustinOpenData_311UnifiedData
Output Name junkspacetimecube.nc
Distance Interval 1000 Feet
Time Interval 1 Months
Time Interval Alignment REFERENCE_TIME
Reference Time 1/1/2013
Summary Fields
Output File

Messages
Start Time: Thursday, October 11, 2018 5:04:48 PM
Running script Create Space Time Cube...
Submitted.
Executing...
Executing (CreateSpaceTimeCube): CreateSpaceTimeCube "Feature Set" 1000 Feet 1 Months ReferenceTime 1/1/2013 [] junkspacetimecube.nc #
Start Time: Thu Oct 11 17:04:51 2018
Using URL based GPRecordSet param: https://coagisentd1.coacd.org/server/rest/services/Hosted/AustinOpenData_311UnifiedData/FeatureServe...
WARNING 120094: Bin generation and analysis requires a projected coordinate system and a default projection of World Cylindrical Equal Area has been applied.
ERROR 120040: Wrong time type for 'Input Features'. Expected 'instant', got 'interval'.
Failed to execute (CreateSpaceTimeCube).
Failed at Thu Oct 11 17:04:55 2018 (Elapsed Time: 3.91 seconds)
Failed.
ERROR 000582: Error occurred during execution.
Completed script Create Space Time Cube...
Failed to execute (CreateSpaceTimeCube).
Failed at Thursday, October 11, 2018 5:04:55 PM (Elapsed Time: 6.66 seconds)

-Andrew

[SarahAmbrose]

Hey Andrew Rudin‌,

Thanks for the use case, it really helps me understand the workflow and the problem. So I really appreciate it. 

Can you expand on this (what you said above):

For example, maybe I want to investigate "how many tickets were created in various parts of the city over space and time the last 5 years?" and then "is there a spatiotemporal variance in how long tickets are open in parts of the city". With how it's setup now I don't see how I can do it without extracting the SBDS data to my desktop or another hosted copy where I can redefine the time-enablement on a Pro layer or a separate nearly-identical feature service.  If I load the SBDS feature service into Pro, the time-enablement section of the properties is grayed-out so I can't redefine it there before creating the space-time cube.  I also don't see a way to alter the properties of the feature service in GeoEvent or Portal to tinker with the time-enable properties.  I think hosted views aren't an option for this kind of data either.

Here is the solution I was planning on proposing:

I think there are two ways you can solve this:

1. Modify the time settings on the layer:

You can change the time settings of the layer in portal. To do this - open the item in portal (the item page), you'll see the layers listed. From there, there should be a button called "time settings", and you can change the time on the service that way. This does change the service definition though. 

2. Create a view of the layer, and modify the time settings on that. 

You can make a view of the layer (again, from the portal item page), and modify the time settings on that layer (in the portal item page again). Here is an overview of views to get you started: Create hosted feature layer views—Portal for ArcGIS | ArcGIS Enterprise

So I'd like to know:

• Why can't you create a view on this layer? In the portal item page, does "create view layer" show up?
• In portal - you should be able to change the time settings on the item page. Are you the owner of the layer?

Thanks!

[AndrewRudin1]

Okay, I think I see the issue.  I do not get those options to create a view or alter time settings on the SBDS layer I created in GeoEvent manager. To be specific, in GeoEvent Manager go to Site->Spatiotemporal Big Dat Stores->Create Data Source.   I guess GeoEvent created SBDS layers are not considered hosted layers. 

To test I created a second SBDS layer a different way...by going into Pro and running GeoAnalytics->Copy to Data Store on the sample data housed on my PC.   That creates a portal item labeled as hosted and does provide the "time settings" and "create view" options.

Perhaps if I create a hosted SBDS layer first via Pro, I can get GeoEvent to write to it.  Something to try later.  Regardless, given this little discrepancy, sounds like we'll have to put some forethought into how we want to use any data from GeoEvent for analysis before we put any GeoEvent services into production.

Thank for walking through this example with me.  This really helps me understand how these servers fit into solutions for our business needs.

-Andrew