Latest Contributions by RyanClancy

In GeoEvent Manager under Site -> Spatiotemporal Big Data Stores we have the ability to create and configure data sources. As far as I can tell there is no way to alter service properties programatically, such as through a REST API. Is this correct? Is there a plan to eventually have a REST API for GeoEvent / SBDS service configuration? My particular use case involves altering the Refresh Interval parameter on my data source. The help pop-up for that setting says, "..... Setting the value to '-1' disables the data indexing, for a period of time, until the value is set back to a positive number. Dynamically changing the value to '-1' and back for a period of time is useful for cases where there is a big burst of events, allowing for a more performant bulk-indexing action." . We have large quantities of data to load periodically, like once a day for example, in addition to real-time data. We'd like to schedule the loading of those big bursts to occur outside of business hours. In these scenarios we'd be scripting in Python to fetch the bulk data from another system at some scheduled time, like midnight, feed it into GeoEven via websocket, and have the data available for use the next morning. It would be useful if our Python script could switch the Refresh Interval to -1 prior to loading and then flip it back to 1 once loading is complete. It isn't practical for us to do this manually given that we want to schedule this outside of business hours. Failing programmatic access to that parameter, can anyone make recommendations as to how it should be set? How resource (CPU, RAM) intensive is the indexing process? Is it likely to cause users to notice performance differences when querying or viewing the data? Cheers, Ryan ... View more

I posted this in the ArcGIS API for Python space yesterday: https://community.esri.com/message/804065-spatiotemporal-bds-rest-query-performance and I'm raising a slightly different version of the issue here because I'm not sure if the performance I'm experiencing is due to the way I'm querying the data (Python API), the way I set up the data source in the big data store, or maybe both. Or maybe some other factor. I created the output data source in the STBDS using GeoEvent Manager. In a nutshell, I've got unprojected point data (GPS-like data in sr 4326), two spatial indexes (the default geohash plus flat hexagons), and a datetime index. My spatiotemporal big datastore is a 3 node cluster where each node has 16 cores and 32GB RAM with ArcGIS Enterprise 10.6.1 running on Linux. I loaded 450 million records via GeoEvent server. The first query I need to do gives me a list of distinct values and their counts for each 24 hr period. This works and I can get the query results I wanted, it's just a lot slower than I expected. I'm using the ArcGIS Python API to query the data via an arcgis.features.FeatureLayer.query() and each query takes over an hour to return. By comparison, the same query via cx_Oracle directly to the same data in an Oracle db takes 38 seconds. The exact queries I'm doing against Oracle and via the Feature Layer are included at the link I posted above. Am I doing something wrong, either in way I've configured the STBDS data source, or in the way I'm querying? My assumption is that the time_filter parameter in the FeatureLayer.query() method leverages the datetime index on my STBDS data source. Is that true? Is there a way to leverage the indexes that isn't exposed via the Python API? What type of query response time is reasonable to expect in this situation? I was shocked that the query against Oracle took 38 seconds while the one against data in the STBDS took over an hour. ... View more

Using the ArcGIS API for Python I'm comparing query performance against data in Oracle vs a Hosted Feature Layer where the data is in the Spatiotemporal Big Datastore. We've got ArcGIS Enteprise (portal, hosted server, geoevent server, datastore) at version 10.6.1 running on Linux; each component is on it's own machine that meets the recommended specs. Our spatiotemporal big data store has 3 nodes each with 16 cores, 32GB RAM, and 1TB storage. I'm using ArcGIS API for Python v1.4.2. I used GeoEvent server to load 450 million rows into the Spatiotemporal Big Data Store. I made a table view of the same data in Oracle. For one day at a time, my query gets all of the unique values in a field (myField) and counts up how many records there are for each unique value. Using a cx_Oracle database connection I query my Oracle data like so: select = """SELECT COUNT(myField) as CNT, myField FROM MY_TABLE                   WHERE ((DATETIME >= TO_DATE('2018-08-01 00:00', 'YYYY-MM-DD HH24:MI')) AND                   (DATETIME < TO_DATE('2018-08-02 00:00', 'YYYY-MM-DD HH24:MI')))                   GROUP BY myField                   ORDER BY CNT desc""" unique_values = pandas.read_sql(select, <database connection>)‍‍‍‍‍‍‍‍‍‍‍‍ That query returns in 38 seconds and tells me that I have 3.8 million unique values for that day in myField. When I put that in a for loop and do that query for each day in a month the average query time is 31 seconds. I expected my queries against the spatiotemporal big datastore to be faster but instead they are WAY slower. Doing that same query takes 1 hr 12 min. Iterating over a month one day at a time yields an average query time of 1 hr+. I'm doing the query like this: fl = arcgis.features.FeatureLayer('<url>/MapServer/0', gis) time1 = datetime.datetime(2018,8,1) time2 = time1 + timedelta(days=1) unique_values = fl.query(                          where='1=1',                          time_filter=[time1, time2],                          out_fields='myField',                          return_distinct_values=True,                          return_geometry=False                         )‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍ The query returns exactly what I expect it to return, which is the same thing that the Oracle query returns. This is good. But it's waaaaaaay slower at about 1 hr 15 min on average (again, it takes 30 seconds via Oracle). How can I be sure I've optimized this query? Does Elasticsearch have some kind of query overhead that makes queries take a long time (in other words, should I only expect Elasticsearch to outperform Oracle when the scale of my data is huge?). Or should I not expect Elasticsearch to outperform Oracle at all for this type of query? Incidentally, what does the dynamic_layer parameter in the FeatureLayer constructor do? arcgis.features module — arcgis 1.5.0 documentation  The built-in docstring viewed via ctrl+tab in my Jupyter Notebook says, "Optional dictionary. If the layer is given a dynamic layer definition, this will be added to the functions.". What does that mean? There are no examples and I can't figure out what this is for. I was hoping I could use it like a definition expression to limit the data in my feature layer to the time range in question, thereby making my query faster. ... View more

We're experimenting with replacing a legacy system which uses Python scripts for processing and an Oracle DB for storage with GeoEvent Server for processing and the Spatiotemporal Big Data Store for storage. I've been able to replicate our processing (filtering, data validation, geometry creation, etc) in GeoEvent 10.6 with one exception: detecting and removing duplicates. In Python we load data into a Pandas.Dataframe and removing duplicates is as simple as using the .drop_duplicates() method. I haven't found a way to do this in GeoEvent yet. Is it possible? Or can I somehow delete duplicate records from the Big Data Store after the events are pushed in? ... View more

In JS API v3.19 there appears to be a bug which requires the JSON style file for a Mapbox GL style to include a 'layout' property, even if it is empty. The absence of a 'layout' causes the map to crash. We've only seen this issue when the layer type is 'line' but it may also occur for other layer types. Manually adding ("layout: {}") to root.json fixes the problem but this is tedious when there are many layers. Will this be corrected in future versions of the API? We have not tested with v3.20 directly, as we encountered the problem when adding a VectorTileLayer to Portal v10.5.0 which uses JS API v3.19 and our testing has been limited to Portal so far. ... View more

I haven't been able to find anything online to indicate how esri/request should work via https. Should this work? I'm using js api v3.13. require[(      "esri/request", ... ], function(request, ...) {      var printInfo = new request({           url: "https://my.server.url:6443/arcgis/rest/services/Geoprocess/ExportWebMap/GPServer/Export%20Web%20Map",           //url: "http://my.server.url:6080/arcgis/rest/services/Geoprocess/ExportWebMap/GPServer/Export%20Web%20Map",           content: {"f":"json"}      });      printInfo.then(           function(response){                console.log("Response: ");                console.dir(response);      }, function(error){           console.dir(error);      }); When I use the https url to my print task the esri/request response object is null but when I use the http endpoint I get a response like I would expect. I never get an error via https, just a null response. Do I have to handle https differently? I've tested this with print services on two internal (i.e. corporate intranet) servers and get the same result. Servers are at AGS v10.1. One server has a self-signed certificate (this is on Windows), the other has a certificate authorized by our CA (this is on Linux). I've tried this via proxy and directly (via CORS) and results are the same. I've tried accessing my application via https and http and results are the same. Any tips or suggestions Kelly Hutchins​? ... View more