geofence - limitations - count and memory

3039
8
12-04-2016 04:44 AM
DavidHoy
Esri Contributor

Hi team,

this is my first foray into GEX and it's a step at a time.

In our site, we wish to run a simple GeoTag to track less than a dozen vehicles.

We have a set of polygons representing "chunks" of a limited part of the street network. Each event that falls within one of the roadway chunks to be tagged with the name of the chunk.

In my understanding this is a simple case of the use of a GeoFence.

I have the RoadChunks available in a Feature Service and I can load and use a subset of these as Geofence and capture these in a test case.

The problem is - I cant use all my polygons (And there are >36,000) - When loading the Geofence, I get an out of memory message from GEX.

I understand that there is a 4GB memory usage limit. (#Our server actually has 16GB available) and I can see the memory warning message comes up at around the 4GB limit (observed in Task Manager). So it sort of makes sense, but....

My RoadChunk polygons are quite simple (6-8 vertices each) and when in a file geodatabase take up a few 10s of MB.

1. Why do I run out of memory at around 10,000 loaded "chunks"?

2. Could I increase the memory limit to (say) 12 GB ?

3. any other tricks?

4. Am I being unrealistic wanting to use such a large number of Geofence polygons?

5. Can anyone suggest another strategy?

8 Replies
ChrisBeyett
Occasional Contributor

HI David,

As you've noticed working with geofences within the GeoEvent Extension can get tricky depending on the polygons your're working with. I'll be answering the questions you posed as well as outlining the steps you'll need to take in order to ensure all polygons can be consumed within GeoEvent. Currently, when importing geofences in GeoEvent it stores the geometry information inside the system RAM. This is shown as an increase in consumed RAM utilization by the java.exe process (as you've seen in the Task Manager). The answers to your questions can be found below:

1. Since geofence geometry is stored in memory and can be constantly read and interpreted, more complex polygons will consume more system RAM until the 4 GB limit is reached. In many situations, the finite limit of polygons is actually variable, and I've seen the default max be as low as 100 polygons (very complex geometry) or as high as 20,000 (simplistic small geometry).

2. There is a way to increase the java.exe process heap size limit which will allow more geofences to be imported. However, doing so is at your own risk as any increased RAM usage could cause unintended consequences on the machine. That being said, I would start with increasing the limit to 8192 MB (8 GB) and assess the performance of the machine at that point. While you certainly could set the limit to 12 GB, it would be best to start small and continually test and assess in line with your organization's IT policy. Below are the steps needed to change the heap size from the 4 GB default to an 8 GB testing size:.

a. Log in to the GeoEvent server machine, as we'll need to modify a configuration file in the installation directory
b. Stop the ArcGIS GeoEvent Windows Service
c. Navigate to the following directory:
C:\Program Files\ArcGIS\Server\GeoEvent\etc
d. Edit the "ArcGISGeoEvent.cfg" file, which is typically the second file in the directory

e. About halfway down the file, you'll find a listing of Java parameters that GeoEvent uses. The parameter you want to change is commented with "# Maximum Java Heap Size" and is typically Java parameter 15.
f. Change the value from -Xmx4096m to -Xmx8192m, save and exit the file
g. Start the ArcGIS GeoEvent Windows Service
h. Attempt to re-import the same block of geofences that you received the failure on before. These should now import successfully

3. Along with changing the Java Heap Size, another good trick for ensuring that geofences are imported successfully is to simplify the geometry as best as you can. Limiting the amount of vertices in each polygon will greatly improve the changes of it being imported into GeoEvent.

4. You are absolutely realistic in your logic pertaining to geofences. The 4 GB limit is in place by default and intended to work with the minimum system requirements for ArcGIS Server and the GeoEvent Extension. This may not be enough for some testing and production environments depending on the scope of the project you're working on. The maximum amount of unique geofences that I've seen in a production environment is just shy of 70,000. They consumed over 16 GB of system RAM and were incorporated into 22 different GeoEvent Services. That's not to say that it's the upper limit, just the most I've personally encountered.

5. Start with increasing the Java Heap Size as well as being mindful of the polygon complexity over time. Since you have more than 36,000 total polygons, you may not have enough system RAM installed to accomplish importing them all at the same time. This is why small batch testing is important, as your testing results can be used as a justification if the installed machine resources need to be increased over time. 

- Chris

DavidHoy
Esri Contributor

Thanks Chris,

i will certainly try your suggestion.

we have already simplified our polygons down to between 4-8 vertices each so I have my fingers crossed I will get all 36000+ loaded in the memory I do have available.

i assume it really makes no difference if I load in batches or all in one go. Once loaded, GEX still needs to bring the whole lot up into memory each time the service is started.

i am still surprised so much memory is consumed - as I said - within a fgdb -  these records are only dozens of MB - I guess the structure used within the cache is not a binary geometry - but still?

i will let all know how we go

David H

DavidHoy
Esri Contributor

A bit of an update.

I have done all I can to simplify the input polygons  and also looked at tweaking the parameters in the Settings tab.

i.e. Set "Geofence Acceleration Degree" to 0 and

" SetSpatial Index Maximum Level of Details" to 8 (rather than default = 16)

None of these changes have made the slightest difference.

I have increased available Memory limit to 14 GB - and I can load all the - just - within this limit.

The amount of memory consumed iby the java.exe is always ~13.2 GB regardless of changes to the above settings.

I also notice that when I delete all the cached GeoFences - the memory consumed DOES NOT CHANGE.

It appears to me that there is a memory leak in the caching process. As each polygon is loaded, it grabs some memory and does not release it prior to loading the next feature.

I can think of no other reason that a Feature Class that takes 22 MB on disk in a FGDB consumes 13 GB when "cached" - For 36 000 features (with 4-12 vertices) - that equates to something like 360 KB per feature!!

MarkBramer
Occasional Contributor II

dhoyesriaustralia-com-au-esridist‌,

Would you be able to send me a file geodatabase of your road chunk layer?  I could try and see what I get on my end...

Mark

0 Kudos
DavidHoy
Esri Contributor

Mark – sorry I have taken so long to get back to you – I have been on leave.

I have copied the road chunk layer in an fgdb to our ftp site

ftp://clients@ftp.esriaustralia.com.au/Clients/For_Esri_MarkBramer/Chunking.gdb.zip

 

The polygon layer that I load to the geofence is the RMS_RoadLinks_chunkSimplifiedBuffer

I will send the login details to your esri email account

MarkBramer
Occasional Contributor II

David,

I downloaded your data and see the same thing you do.  I didn't really have any expectation I wouldn't, but it's always nice to have multiple eyes on a problem just in case.  I tried multiple little things here and there as long shots (using attributes as categories, i.e. subnet_ran) and breaking out multiple feature services (i.e. just the subnet_ran=4 ones) but no change.  I did not think these would work since GeoEvent's geofence implementation has no overlap with the geodatabase (i.e. using its spatial indexing) or ArcGIS Server, but again, it never hurts to try.  

Unfortunately, the only solution is what's already been discussed/done -  simplify the geometry, and throw more RAM at the problem. 

Sorry I don't have any more for you.

Mark 

DavidHoy
Esri Contributor

Thanks Mark,

disappointing, but not unexpected. Did you also experience the fact that memory is not released by the java.exe even after the geofence is deleted?

To me, that's evidence of a memory leak in the caching process. Something that should go for investigation as a BUG - not just as known behaviour.

DavidHoy
Esri Contributor

A bit of an update.

We have upgraded to 10.5 and very pleasantly surprised to find that GeoFence memory management is MUCH better.

It is now about 45 KB per polygon (4-12 vertices each) which still seems large to me – but much better than the 5MB per feature we were seeing at 10.4.1.

We noted that even though significantly less memory was consumed by the GeoFences – this memory was still NOT released when the GeoFences were deleted.

The memory is not returned to the general pool until the entire GeoEvent Server windows services is stopped and restarted.

David H.