Building a Data Expression: Populating Fields & Attributes

[jcarlson]

If you know me, you know I love a good Data Expression in a Dashboard.

But if you know Data Expressions, you know they can be a bit tedious to build, defining all those fields and feature attributes. I wanted to share a couple of shortcuts you can use in your next Data Expression.

Populating Fields: Schema and Splice

Maybe you've been in this situation: you want to use a Data Expression to create one or two new fields, but the output needs to include a lot of the fields from the input FeatureSet. You've got better things to do than re-write the schema!

If you haven't tried it out yet, take a look at the Schema function, and what it returns:

Let's look closer at the fields object:

Isn't that what we want? Yes, it is! But we need to add our new fields to it. Enter the Splice function, which takes two arrays and just smooshes them together. Next time you run into this situation, try something like this:

var fields = Schema(fs)['fields'];

var new_fields = [
    {name: 'next_due', type: 'esriFieldTypeDate'},
    {name: 'window_start', type: 'esriFieldTypeDate'},
    {name: 'overdue', type: 'esriFieldTypeString'},
    {name: 'current_ok', type: 'esriFieldTypeString'},
]

var out_dict = {
    fields: Splice(fields, new_fields),
    geometryType: '',
    features: []
}

You've still got to define those new fields, there's no way around it. But you can just copy the schema for the others, the ones I call "pass-through" attributes. In just a few lines, I can tee up my full schema for adding features. If my input FeatureSet had, say, 30 fields I wanted included, it would take twice as many lines just to define those, to say nothing of the new ones.

Bonus: using Schema grabs the alias and domains for your fields, too!

Populating Attributes

So now we're in our for (var f in fs) { ... } loop, populating the output features array. If you've got lots of attributes in the output, this can get pretty tedious, too. Unfortunately, a Feature object is immutable, so we can't just add or change attributes to the features.

Consider this, though. Look what happens when you loop through a Feature:

It's actually looping through the attributes dictionary keys! So check this out: we create a new dictionary, then push the feature attributes into it.

* record screech sound effect*

What about dates, though? If you've read this far and you're still interested, you probably know that Date values tend to break Data Expressions. You need to do Number(date_value) for it to work. If you iterate over your attributes, can you still do that? Yes! We just use the function TypeOf to check if it's a date, then adjust.

var attrs = {}

for (var attr in f) {
    attrs[attr] = Iif(TypeOf(f[attr]) == 'Date', Number(f[attr]), f[attr])
}

But, like the Schema function, this only helps us with the attributes that already existed in incoming FeatureSet, not our new fields. There's no Splice for dictionaries, but since we can create new values simply by calling dictionary['new_key'] = new_value, it's pretty simple:

attrs['some_new_field'] = some_value
// and so on

So, back to the real example I'm working with, I've got my 4 new fields I want to add to the output, and half a dozen "pass-through" attributes. I will actually define those new fields at the same time as creating the dictionary, then do the attribute loop.

Remember, this is all in my for (var f in fs) loop, so at the end, I'm pushing the feature into the features array in my output dict.

    // 4 new fields in dict
    var attrs = {
        next_due: Number(next_due),
        overdue: Iif(lr < prev_due && !IsEmpty(lr), 'OVERDUE', ''),
        current_ok: Iif(lr > window_start && !IsEmpty(lr), 'OK', ''),
        window_start: Number(window_start)
    }

    // add pass-through fields
    for (var attr in f) {
        attrs[attr] = Iif(TypeOf(f[attr]) == 'Date', Number(f[attr]), f[attr])
    }
    
    // Add feature to output dict
    Push(out_dict['features'], {attributes: attrs})

Again, I managed this all in just a handful of lines, where this could ordinarily take a great deal more if I have a lot of fields from the input FeatureSet. As the number of "pass-through" fields increases the benefits do, too, as this method can apply to any number of input fields without any changes to the code.

Bonus: the attrs dictionary is not a feature, so its values are mutable. Suppose I had a field in the input that I wanted to change slightly. As long as I do it before the Push function, I can, and without needing to creating any new variables.

attrs['establishment_name'] = Proper(attrs['establishment_name'])

 I hope some of this helps you out the next time you're working on a Data Expression!

[JohannesLindner]

Josh, this is very helpful stuff, thanks for the excellent writeup!

Alas, I have to address my Arcade pet peeve:

What about dates, though? If you've read this far and you're still interested, you probably know that Date values tend to break Data Expressions.

I honestly doubt that most users know that... I have answered multiple questions about that and I've seen you and others answering even more questions. And while your solution of using TypeOf is nice and concise, it's also mind-boggling that we have to resort to that. Please consider adding your support to this idea that tries to remove that problem:

https://community.esri.com/t5/arcgis-online-ideas/arcade-allow-date-values-in-date-fields/idi-p/1204...

[jcarlson]

@JohannesLindnerI'm surprised I missed that! Well, I've upvoted it now!

[MatejVrtich]

Hi,

thank you for sharing this.

I'd like to add that the Feature coming from a FeatureSet is indeed immutable, you can create a mutable one using a Feature function itself.

dict.features = [];
for (var f in fs) {
  var f_new = Feature(f);
  f_new.PARK_ALL = f.PARK_NA + f.PARK_OA + f.PARK_A + f.PARK_K;
  f_new.KM_NEXT = calculateDistanceToNext(f);
  Push(dict.features, f_new);
}

I also found out, that the Dictionary returned by a Schema function is immutable as well (don't know why). But it is possible to create a mutable copy of it and use it as a skeleton for an output FeatureSet.

var dict = Dictionary(Text(Schema(fs)));

Regards,

Matej

[DavidNyenhuis1]

@jcarlson, thank you for the detailed guide and tips for data expressions. Regarding the nuance with date fields, an enhancement has just been rolled out this week. You no longer have to convert them to EPOCH with the Number() function. You just have to make the following change.

Instead of:

return FeatureSet(Text(dict))

Do this:

return FeatureSet(dict)

Learn more on this blog post. 

NOTE: For Enterprise users, this update is targeted for 11.2