Simulator Settings | GEP Settings | GEP Monitor | DB records | LOSS |
2 simulators, 5 msgs/sec | tcp->filter->processor->fs output 1 sec insert | 2070 features | 2065 | 5 |
2 simulators, 5 msgs/sec | tcp->filter->processor->fs output 1 sec insert | 2690 features | 2678 | 12 |
2 simulators, 5 msgs/sec | tcp->filter->processor->fs output 1 sec insert | 3395 | 3386 | 9 |
2 simulators, 5 msgs/sec | tcp->filter->processor->fs output 3 sec insert | 2125 | 2123 | 2 |
2 simulators, 5 msgs/sec | tcp->filter->processor->fs output 3 sec insert | 2655 | 2651 | 4 |
2 simulators, 5 msgs/sec | tcp->filter->processor->fs output 3 sec insert | 3475 | 3469 | 6 |
2 simulators, 5 msgs/sec | tcp->filter->processor->fs output 6 sec insert | 2045 | 2040 | 5 |
2 simulators, 25 msgs/sec | tcp->filter->fs output 6 sec insert | 2225 | 2220 | 5 |
2 simulators, 5 msgs/sec | tcp->filter->fs output 6 sec insert | 2690 | 2688 | 2 |
2 simulators, 25 msgs/sec | tcp->filter->fs output 12 sec insert | 2250 | 2235 | 15 |
2 simulators, 25 msgs/sec | tcp->filter->fs output 12 sec insert | 2425 | 2419 | 6 |
2 simulators, 25 msgs/sec | tcp->filter->fs output 3 sec insert | 2300 | 2293 | 7 |
1 sumulator, 25 msgs/sec | tcp->filter->fs output 1 sec insert | 2075 | 2065 | 10 |
Is there a definitive way to confirm GEP is successfully receiving/processing/outputting everything, other than what Monitor is reporting?The monitor counts and the log files are the only way for you to get information from the GEP system at 10.2. The only exception to this is if you develop your own Adapter/Transport/Processor, you can connect a debugger and step through your own code. Note that setting the log level to DEBUG for specific components within the GEP is possible.
Or is Monitor a very trustworthy tool for determining that GEP has received/processed/output everything sent to it?I would say that the Monitor page is very useful for determining if a GeoEvent got dropped by the Input, Service, or Output. In your case, it seems pretty clear that the messages are being dropped in the Output or in the transaction between the Output and the Feature service. But the Monitor page doesn't give you details on what takes place inside the Output. The problem with getting much more from the Monitor here is that the Monitor is generic to all Outputs and you need details on the inner working of a specific Adapter & Transport.
What ways exist to track down where the message "dropping" is occurring (especially given there are no errors in GEP or AGS logs)?There's 4 possible culprits that I can identify from a GEP perspective: Adapter, Transport, Output, Feature Service. I'm lumping everything downstream from the Feature service (like the SQL Server) as part of the Feature Service. As mentioned, there's no definitive way for you to determine if messages get passed successfully from Adapter to Transport, but you have a way to intercept the data that goes from the Transport to the Feature Service. All of this data is transferred through http, and can be intercepted by wireshark. However, this would take a lot of patience as you would have to figure out what message got dropped, and it sounds like you are only losing about 0.1% of your data. That's a lot of data to sift through looking for the lost geoevent. You can also set the log level of your Transport to DEBUG and see the content of all features sent to the feature service, and the response from the server for each of those insert statements. But that's still a lot of data to sift through.
Is it possible that GEP is successfully passing on everything, but ArcGIS Server is not successfully writing all records?That is possible. I would look at the memory and cpu usage of the GEP and the ArcGIS Server processes on your machine to determine if one of them looks to be behaving abnormally. The ArcGIS Server logs may contain something useful as well.
Could this be a SQL Server thing?This seems possible as well, but I wouldn't know how to systematically diagnose an error here.
What other places in the entire cosmos of this data flow (not just within GEP) are potential suspects?That's difficult to say from the perspective of GEP. There may be settings use when publishing the feature service that affect its performance as well as a plethora of database settings, network settings, etc.
Is there any flaw in our methodology that could explain the discrepancy between what GEP reports and what's written to the DB?The only possible logical error I can think of would be if you were using the "Update Feature Service" connector. This connector will only update each track ID once per buffer interval. For example, if you set it to buffer the geoevents for 3 seconds before sending an update to the feature service, then send in two GeoEvents with the same track ID, only one of those GeoEvents will get written to the Feature Service.
Simulator Settings | GEP Settings | GEP Monitor | DB Records | LOSS |
1 simulator, 25 msgs/sec | tcp->filter->fs output 1 sec insert | 2075 | 2068 | 7 |
2 simulators, 25 msgs/sec | tcp->filter->fs output 3 sec insert | 2300 | 2277 | 23 |
2 simulators, 25 msgs/sec | tcp->filter->fs output 12 sec insert | 2400 | 2386 | 14 |
2 simulators, 25 msgs/sec | tcp->filter->fs output 6 sec insert | 4700 | 4664 | 36 |
How many events per second is the GeoEvent Simulator you are using sending to GeoEvent Processor?
Are you using the Update a Feature or the Add a feature outbound connector to update the feature service?
Updating features can be expensive if the target feature service contains many thousands of features.
There are a couple of things you might try to help narrow the scope of the problem:
- Reduce the number of events being sent to see if the problem is with data rate.
- Switch to use the Add a feature outbound connector to see the problem is with adding records vs. updating records.
- Specify a smaller number (50, 10, or even 1) for the outbound connector's Maximum Features Per Transaction parameter to see if updating the feature service with smaller batches helps.
- Check to make sure that there are no errors with the event data, such as a string which exceeds the maximum length allowed by the feature service or an integer value being sent which exceeds the range of the feature service's target field.
You might also use the GeoEvent Processor's log to see how many milliseconds (or seconds) elaspse between GeoEvent Processor's POST to retrieve information and its POST to update the features. If there are any errors in these transactions, you might also find them in the log.
Hope this information is helpful -RJ