Introduction to jtlc

This document provides a brief description of jtlc (Javascript Template Language Compiler) — an infrastructure for creating declarative templates compiled into Javascript code — as well as JXL (JSON Transformation Language) — a specific template language implemented with jtlc for transforming and re-shaping JSON data. At the moment the two concepts are very much intertwined and may be considered one and the same; the author does, however, expect to see — and come up with — template languages with different syntax and purpose built on jtlc infrastructure.

Usage model

The jtlc is designed around the two-stage model of template execution: first, the template is compiled into a Javascript function with a call to dojox.jtlc.compile(template, language) where language is an object instance encapsulating both the specifics of the template language and the optional setting selected by the user. The returned evaluator function may then be executed as many times as necessary on different inputs. Structure of the input and output data for the template depends on the language; JXL treats both as anonymous (i.e. without a custom constructor) Javascript objects and arrays in keeping with the philosophy of JSON.

Performance

The primary reason behind the two-stage execution model is performance: the code compiled from templates should generally be expected to compare well to handwritten Javascript in the efficiency of execution. Care is taken to avoid or minimize spurious copying, use efficient loop constructs, evaluate complex subexpressions once etc. All data structures maintained during the execution of the template remain under direct control of the template's programmer: the compiler introduces only local variables and no additional arrays or objects. Turning on optional argument checks may lead to introduction of function calls to contexts where default settings produce only inline code: choosing uncompromising performance vs. timely detection of errors in input data is a tradeoff left to the user of the library.

Introduction to JXL

JXL is similar in purpose to XSLT: it provides a concise notation describing the transformation of one JSON structure into another. The language itself is also based on Javascript object notation but it is not JSON compliant: aside from the primitive types and object and array literals it also requires the use of functional tags which expand into non-anonymous — and invisible to the user of the library — Javascript objects.

Execution model

The execution of a JXL template is best described in terms of the dataflow programming model: values are generated by some primitives, are operated upon by others resulting in new values and end up in sinks provided by a third kind of primitives. The behavior of most languages entities depends on whether they appear in a context of a singleton or an iteration:

Singleton context

In singleton context only a single value is processed. All primitives with parameters operate as functions insofar as they have no side effects.

Iterative context

In iterative context the values are produced in sequence by the innermost generator, undergo processing imposed by the primitives enclosing the generator and ultimately end up in a sink which limits the context. Array is the most typical sink for the iterative contexts though it is also possible to populate dictionaries (anonymous Javascript objects) iteratively or aggregate the results to a single value. The generators typically iterate over arrays passed in as template's arguments, produced by executing a query over input data or built by other parts of the same template; it is also possible to iterate over keys in a dictionary.

Usage examples

As JXL was designed to build upon and extend the capabilities provided by dojox.json.query rather than as a replacement for the latter library, the examples in this section concentrate on use cases where dojox.json.query in itself is not sufficient to do the job. The author hopes that simpler patterns of JXL use can be easily derived from these examples by the reader.

Grouping

Note that strings "author", "$[0].author" and "[/author]" are implicitly treated as expressions and a query, respectively, based on the execution context. At the same time, string "title" requires an explicit tag around it since it appears in an iterative context.

Using $[0] in an expression within group body is an easy way to access the key fields as they are equal for the entire group anyway and the group is guaranteed to have at least one element. Simple expressions referencing the properties of the current input do not have "$." in front of the name as it is implicitly assumed by expr().

For optimal performance, only one sorting on a composite key occurs here. The inner group() operates on the current value (sequence of records with identical continent key) of the outer one so it does not need the third argument.

Aggregation

In this example, using an object literal at JXL level to form the output records is not possible since it establishes an execution context of its own. Hiding it inside an expression solves the problem for simple cases. More complex aggregation patterns may require external accumulators implemented using an object or a closure:
Here, each() is used to evaluate the object literal for each input array element (iteration over current input is assumed by each() when it is given only one argument).

Flattening nested arrays

Again, it is much easier to move object construction inside the inline expression than to deal with an object literal with a context of its own.
Note that the second expression does create an intermediate copy: it is necessary to pass multiple properties to the innermost iteration. This copy however is shallow and performance impact of it should be minimal.

Transforming dictionaries

The example above takes a simple record and transforms it into a data structure suitable to populate the names and values for controls in a form template.

Simple joins

At the moment, JXL lacks comprehensive support for SQL-like joins. Nevertheless some simple yet important cases — such as lookup by an unique key — can be implemented in a straightforward manner:
The association of the two tables relies on an intermediate dictionary with the key field of the join serving as the key in the dictionary. The same approach can be extended to compound keys by using replace() to build their string representations.

JXL and jtlc reference

dojox.jtlc.compile

compile( template, language )
Accepts template as its first argument and language description (such as an instance of dojo.jtlc.JXL) as its second argument. Returns the compiled evaluator function for the template or throws an exception in case of errors.

dojox.jtlc.JXL

new JXL( { setting... } )
Instantiates language description for JXL that can be customized by specifying the following settings:
elideNulls
When true, null values are not placed into array or dictionary sinks but are thrown away instead (with their corresponding keys in case of a dictionary).
failOnDuplicateKeys
When true, an attempt to insert a duplicate key into the dictionary sink results in an exception.
singletonQuery.failOnNoResults
Set to true to verify that queries in singleton contexts produce at least one result or throw an exception otherwise.
singletonQuery.failOnManyResults
Set to true to verify that queries in singleton contexts produce no more than one result or throw an exception otherwise.
queryLanguage
By default this option is set to dojox.json.query. You may substitute a query language compiler with compatible API.
replaceLanguage
By defualt this option is set to dojo.replace. You may substitute a formatting function of your own with compatible API.
Note that default settings always favor performance over other concerns; turning on additional checking in the code is likely to make it somewhat more complex and, as a result, slower.

JXL literals

Object literals

{ key0: input0 [, ...keyN: inputN] }
Object literals are sinks producing dictionaries (anonymous Javascript objects). By default, all inputs of an object literal are evaluated in singleton mode and the resulting values inserted into the resulting dictionary with literal key values. In order to populate a dictionary with keys generated from an input sequence, use the setkey() primitive from a sub-template enclosed within many(). In this case the key associated with this sub-template within the literal will be ignored and the computed value will be used instead.

Array literals

[ input0 [, ...inputN] ]
Array literals are sinks producing anonymous Javascript arrays. By default, each input of an array literal is evaluated in an iterative context of its own. In order to evaluate a singleton sub-template and put its value in an array, enclose the sub-template within one().

Note that array literals themselves are not generators and should not appear in an iterative context even though they establish an iterative context of their own inside the brackets. In order to use the value produced by an array literal as a generator, enclose it in from().

String literals

String literals may appear in place of a sub-template anywhere in JXL. They are interpreted differently depending on the context: in the context of a singleton, the string is taken to be an inline expression as if it were enclosed in an expr() tag. In an iterative context, the string is assumed to be a query without additional parameters, which is to say, an equivalent of query( string, current() ).

Numeric and boolean literals

Numeric and boolean literals are always interpreted as if they were enclosed in quote(). Consequently they may not appear in an iterative context.

Functions

Values with the type of function are treated as if they were enclosed in bind().

JXL tags

dojox.jtlc.tags.acc

acc( input )
Ensures that the value of its argument (usually a constant) is stored in a local variable (accumulator). Usually used as part of an aggregation construct such as expr( '$1+=expression', input, acc(0) ). It is not an error to use acc() in an iterative context but the tag will have no effect as the accumulator will be overwritten by the next iteration.

dojox.jtlc.tags.arg

arg( index )
Returns one of the arguments passed into the evaluator of the template. While current() defaults to iterating over arg( 0 ), the two are not equivalent: arg() is not a generator and should be enclosed in from() to produce one. The argument of arg() should be a numeric value (usually a constant).

dojox.jtlc.tags.bind

bind( function [, input0...inputN ] )
Evaluates the input0...inputN arguments, then applies function to them. In the iterative context, input0 is treated as the generator, the rest of the arguments are evaluated only once. If the input arguments are omitted completely, bind() assumes that the function should be applied to current().

dojox.jtlc.tags.current

current()
In a singleton context, returns the value of the current input. Current input defaults to the value of arg(0) but may be temporarily reset to different objects in a sub-template enclosed in primitives such as each(), group() and setkey(). In an iterative context, current() serves as a generator of values from current input.

dojox.jtlc.tags.defined

defined( [ input ] )
Serves as a filter passing only the defined values (that is, values whose typeof is not equal to 'undefined') from the input. Should not be used outside of iterative contexts.

dojox.jtlc.tags.each

each( input0 [, input1...inputN ] )
Performs nested iterations by evaluating its first argument with current input set to the value generated by the second argument and so on. Thus the rightmost argument forms the outermost iteration and the leftmost argument forms the innermost iteration. This primitive can only be used in iterative contexts and evaluates every one of its arguments in an iterative context of its own. Note that each() does not nest sinks, only the iterations. In fact, its usual application is flattening hierarchies formed by nested arrays passed into the template.

dojox.jtlc.tags.expr

expr( expression [, input0...inputN ] )
Evaluates its input0...inputN arguments, then evaluates the inline expression, substituting the values of the arguments for placeholders $ and $0...$9 as follows:
$0...$9
Values of input0 through input9.
$
In singleton context, equivalent to $0. In an iterative context, current value generated by input0.
If the expression contains no placeholder strings and begins with an identifier, expr() assumes it to be a property reference on the current input and prepends "$." to it.

If no input arguments are provided, the expression is evaluated with input0 equal to current input. Note that expr() is entirely similar to bind() except that bind() refers to a function outside of the template's evaluator and expr() injects Javascript code directly into the evaluator's body.

dojox.jtlc.tags.from

from( input )
Evaluates the input argument as a singleton and generates values from the array it returns as its result. Applying from() to current() has no effect because current() already serves as a generator in iterative contexts and from() cannot be used as a singleton.

dojox.jtlc.tags.group

group( keys, body [, input ] )
Provides SQL-like grouping capability by evaluating its body over subsequences from input. The subsequences are established by evaluating the keys (either a single sub-template or an array of subtemplates, typically inline expressions) over each element of the sequence and comparing the results. The longest contiguous subsequence of generated values where keys are equal forms a group. The body is then evaluated with current input set to the entire group. The group() primitive can be used only in iterative contexts; it generates values returned by the body: one value per group.

Note that for the grouping to work properly the input sequence should already be ordered in respect to the keys so usually group() is applied to the results produced by a query(). However, the user is left free to group sorted sequences originated elsewhere. If input argument is omitted, current input is used.

dojox.jtlc.tags.keys

keys( [ input ] )
Generates keys from the input: property names if input is an object, indices if it is an array. If input is omitted, current input is used. Cannot be used outside of an iterative context.

dojox.jtlc.tags.last

last( [ input ] )
Serves as a special type of sink, returning only the last value generated by input. Usually used for aggregation purposes. If input is omitted, current input is used.

dojox.jtlc.tags.many

many( input )
Evaluates its input in an iterative context if none exist already. If many() is used within an externally established iterative context it does nothing.

dojox.jtlc.tags.one

one( input )
Evaluates its input in a singleton context even when used within an externally established iterative context (in which case one() effectively works as a generator returning a single value).

dojox.jtlc.tags.query

query( query [, input0...inputN ] )
Executes a query over the specified inputs (each of the arguments input0...inputN is evaluated as a singleton). By default, the query is pre-compiled with dojox.json.query and the resulting evaluator is associated with the compiled template via a closure; changing the queryLanguage setting allow the user to use a different query language implementation following the same paradigm.

In iterative contexts, query() works as a generator returning individual matches from the query. In singleton contexts query() assumes that one and only one result should be produced, which is returned on its own. If query returns no matches, the result will have an "undefined" value; if query returns more than one match only the first one will be used. This default behavior may be modified with the settings singletonQuery.failOnNoResults and singletonQuery.failOnManyResults.

Using query() without the input arguments applies it to current input.

dojox.jtlc.tags.quote

quote( value )
Treats value as a literal even if it is a Javascript object, array or string.

dojox.jtlc.tags.replace

replace( format [, input ] )
Formats and returns its input by performing substitutions within the format string. By default, replace() uses dojo.replace(); user can change this behavior with replaceLanguage setting. If the input argument is omitted, replace() works with current input.

dojox.jtlc.tags.setkey

setkey( body [, input ] )
Changes the key (property name) under which computed values are stored in a dictionary. The input argument is evaluated first, then body is evaluated as a singleton with current input set to the value returned by input; usually, the body consists of a replace(), possibly in combination with expr(). The result returned by the body is used as the key, the value from the input serves as the result of the setkey() itself. The setkey() primitive can be used both within singletons and iterative contexts, but makes little sense in the former. Without the input argument, setkey() operates on the current input.