GeoEvent-Poll an External Website for JSON
I have configured API in the Input connector, but the response of API is not correctly mapped when there is an array or nested JSON.It is only considering the parent JSON and thus all the content is not fetched properly.
Can someone please guide me if I am missing something, or this is how it behaves and there is some other approach to handle it.
API : api.waqi.info/feed/lucknow/?token=fa73462a52dbbb8848d20a0554fb72424af90975
Incorrect JSON:
Try putting "data" in the JSON object name parameter of the input connector.
This might be worth a read, hopefully it helps:
https://community.esri.com/t5/arcgis-geoevent-server-blog/json-data-structures-working-with-hierarch...
Also, I have a JSON feed with multi-cardinality structure, albeit a simple one, but in the input definition the property with multi-cardinality looks like that below. Maybe you can adjust your definition similarly. It's the rental uris.
Hi @Gene_Sipes ,
I have validated the multi-cardinality structure, but not sure why nested json properties are still not correctly mapped.
Consider the "attributions" (for example ) in the api response, it does not even map url ,name and logo.I tried multiple ways to map it out but no luck
api.waqi.info/feed/lucknow/?token=fa73462a52dbbb8848d20a0554fb72424af90975
Hello @RipaliBatra --
Given the rich hierarchical structure of the JSON you are receiving from the https://waqi.info web service you really cannot rely on the GeoEvent Sampler for a good representation of the data. The sampler struggles when given complex hierarchical JSON.
A better approach is to create one or more Write to a JSON File outputs and use them to log the event records emitted from different processors along your configured event processing workflow.
I have a GeoEvent Server 11.1 deployment that I used to test your feed. I was able to allow a Receive JSON on a REST Endpoint input to generate a GeoEvent Definition for me. A sample of the data I sent to my GeoEvent Server input and the generated GeoEvent Definition are shown below {Fig 1] and [Fig 2]. Note that I specified the input use data as the root of the JSON structure when adapting the received JSON.
Note: Be careful when relying on auto-generated GeoEvent Definitions. An input will make its "best guess" as to what the GeoEvent Definition ought to be based on the first data record it receives. But the generated event definition will often use Double for data values received as epoch long integer values (for example). You have to review the generated GeoEvent Definition and verify that each array and element will be adapted properly for the data you expect to receive.
I was able to configure a Field Mapper to pull specific values out of the hierarchical JSON:
Note the expressions being used to access the data:
- Attributions is an array, so we have to provide the index of the element we want to access from that array. attributions[1].name accesses the second element in the array, the one with the named string "World Air Quality Index Project".
- City is the name of an element, so we do not access it with ordinal values like we do when accessing the array. city.name is sufficient to retrieve the string "B R Ambedkar University, Lucknow, India".
- The latitude and longitude coordinates in the city element, however, are in an array nested within an element. When pulling these as separate values we have to specify their ordinal positions in the array:
- The latitude is city.geo[0]
- The longitude is city.geo[1]
- The iaqi values are grouped within an element. Accessing them is fairly straightforward:
- iaqi.co.v
- iaqi.no2.v
- iaqi.pm10.v
In my Field Mapper illustration I only pulled the string for the "day", but if we wanted to be a little more creative we could build a string from available values for a given day. The string value "Day: 2024-05-01 (Avg/Min/Max: 29.0 / 14.0 / 50.0)" could be build using the following expression to pull individual values and append them together -- taking care to use the toString( ) function to explicitly cast Double values to String values when appending them to literal strings:
'Day: ' + toString(forecast.daily.o3[2].day) + ' (Avg/Min/Max: ' + toString(forecast.daily.o3[2].avg) + ' / ' + toString(forecast.daily.o3[2].min) + ' / ' + toString(forecast.daily.o3[2].max) +')'
Because the daily forecast values for "o3", "pm10", and "pm25" all have the same elemental structure you will not be able to use a Multicardinal Field Splitter processor to collapse or flatten these three arrays. It looks like these arrays are allowed to contain a variable number of items. The array o3 has 8 elements whereas the arrays pm10 and pm25 both have 9 elements. There is no iterator or looping mechanism available in any of GeoEvent Server's processors, so you will very likely have to stick with extracting essential information from the JSON using field calculation expressions like I show above.
Hope this information helps, and a special Thank You to @Gene_Sipes for jumping in to help with this complicated hierarchical JSON.
-- RJ
