Introduction to jtlc
This document provides a brief description of jtlc (Javascript
Template Language Compiler) — an infrastructure
for creating declarative templates compiled into Javascript code — as well as JXL
(JSON Transformation Language) — a specific template language
implemented with jtlc for transforming and re-shaping JSON data. At the moment the two
concepts are very much intertwined and may be considered one and the same; the author does, however, expect
to see — and come up with — template languages with different syntax and purpose built on
jtlc infrastructure.
Usage model
The jtlc is designed around the two-stage model of template execution: first, the template is compiled
into a Javascript function with a call to dojox.jtlc.compile(template,
language) where language is an object instance encapsulating both the specifics
of the template language and the optional setting selected by the user. The returned evaluator function
may then be executed as many times as necessary on different inputs. Structure of the input and output
data for the template depends on the language; JXL treats both as anonymous (i.e. without a custom
constructor) Javascript objects and arrays in keeping with the philosophy of JSON.
Performance
The primary reason behind the two-stage execution model is performance: the code compiled from templates
should generally be expected to compare well to handwritten Javascript in the efficiency of execution.
Care is taken to avoid or minimize spurious copying, use efficient loop constructs, evaluate complex
subexpressions once etc. All data structures maintained during the execution of the template remain
under direct control of the template's programmer: the compiler introduces only local variables and no
additional arrays or objects. Turning on optional argument checks may lead to introduction of function
calls to contexts where default settings produce only inline code: choosing uncompromising performance
vs. timely detection of errors in input data is a tradeoff left to the user of the library.
Introduction to JXL
JXL is similar in purpose to XSLT: it provides a concise notation describing the transformation
of one JSON structure into another. The language itself is also based on Javascript object notation but
it is not JSON compliant: aside from the primitive types and object and array literals it also requires
the use of functional tags which expand into non-anonymous — and invisible
to the user of the library — Javascript objects.
Execution model
The execution of a JXL template is best described in terms of the dataflow programming model: values are
generated by some primitives, are operated upon by others resulting in new values and end up
in sinks provided by a third kind of primitives. The behavior of most languages entities depends
on whether they appear in a context of a singleton or an iteration:
Singleton context
In singleton context only a single value is processed. All primitives with parameters operate as functions
insofar as they have no side effects.
Iterative context
In iterative context the values are produced in sequence by the innermost generator, undergo processing
imposed by the primitives enclosing the generator and ultimately end up in a sink which limits the context.
Array is the most typical sink for the iterative contexts though it is also possible to populate dictionaries
(anonymous Javascript objects) iteratively or aggregate the results to a single value. The generators
typically iterate over arrays passed in as template's arguments, produced by executing a query over input
data or built by other parts of the same template; it is also possible to iterate over keys in a dictionary.
Usage examples
As JXL was designed to build upon and extend the capabilities provided by dojox.json.query
rather than as a replacement for the latter library, the examples in this section concentrate on use cases
where dojox.json.query in itself is not sufficient to do the job. The author hopes that
simpler patterns of JXL use can be easily derived from these examples by the reader.
Grouping
Note that strings "author", "$[0].author" and "[/author]"
are implicitly treated as expressions and a query, respectively, based on the execution context. At the
same time, string "title" requires an explicit tag around it since it appears in an iterative
context. Using $[0] in an expression within group body is an easy way to access the key
fields as they are equal for the entire group anyway and the group is guaranteed to have at least one element.
Simple expressions referencing the properties of the current input do not have "$." in front
of the name as it is implicitly assumed by expr().
For optimal performance, only one sorting on a composite key occurs here. The inner group()
operates on the current value (sequence of records with identical continent key) of the outer
one so it does not need the third argument.
Aggregation
In this example, using an object literal at JXL level to form the output records is not possible since it
establishes an execution context of its own. Hiding it inside an expression solves the problem for simple
cases. More complex aggregation patterns may require external accumulators implemented using an object or
a closure:
Here, each() is used to evaluate the object literal for each input array element (iteration
over current input is assumed by each() when it is given only one argument).
Flattening nested arrays
Again, it is much easier to move object construction inside the inline expression than to deal with an
object literal with a context of its own.
Note that the second expression does create an intermediate copy: it is necessary to pass multiple properties
to the innermost iteration. This copy however is shallow and performance impact of it should be minimal.
Transforming dictionaries
The example above takes a simple record and transforms it into a data structure suitable to populate
the names and values for controls in a form template.
Simple joins
At the moment, JXL lacks comprehensive support for SQL-like joins. Nevertheless some simple yet important
cases — such as lookup by an unique key — can be implemented in a straightforward manner:
The association of the two tables relies on an intermediate dictionary with the key field of the join serving
as the key in the dictionary. The same approach can be extended to compound keys by using replace()
to build their string representations.
JXL and jtlc reference
dojox.jtlc.compile
compile( template, language )
Accepts template as its first argument and language description (such as an instance of dojo.jtlc.JXL) as
its second argument. Returns the compiled evaluator function for the template or throws an exception in
case of errors.
dojox.jtlc.JXL
new JXL( { setting... } )
Instantiates language description for JXL that can be customized by specifying the following settings:
- elideNulls
- When true, null values are not placed into array or dictionary sinks but are thrown
away instead (with their corresponding keys in case of a dictionary).
- failOnDuplicateKeys
- When true, an attempt to insert a duplicate key into the dictionary sink results in
an exception.
- singletonQuery.failOnNoResults
- Set to true to verify that queries in singleton contexts produce at least one result
or throw an exception otherwise.
- singletonQuery.failOnManyResults
- Set to true to verify that queries in singleton contexts produce no more than one result
or throw an exception otherwise.
- queryLanguage
- By default this option is set to dojox.json.query. You may substitute a query language
compiler with compatible API.
- replaceLanguage
- By defualt this option is set to dojo.replace. You may substitute a formatting function
of your own with compatible API.
Note that default settings always favor performance over other concerns; turning on additional checking in the
code is likely to make it somewhat more complex and, as a result, slower.
JXL literals
Object literals
{ key0: input0 [, ...keyN: inputN] }
Object literals are sinks producing dictionaries (anonymous Javascript objects). By default, all inputs
of an object literal are evaluated in singleton mode and the resulting values inserted into the resulting
dictionary with literal key values. In order to populate a dictionary with keys generated from an input
sequence, use the setkey() primitive from a sub-template enclosed within
many(). In this case the key associated with this sub-template within the literal will be
ignored and the computed value will be used instead.
Array literals
[ input0 [, ...inputN] ]
Array literals are sinks producing anonymous Javascript arrays. By default, each input of an array literal
is evaluated in an iterative context of its own. In order to evaluate a singleton sub-template and put its
value in an array, enclose the sub-template within one().
Note that array literals themselves are not generators and should not appear in an iterative context
even though they establish an iterative context of their own inside the brackets. In order to use the value
produced by an array literal as a generator, enclose it in from().
String literals
String literals may appear in place of a sub-template anywhere in JXL. They are interpreted differently
depending on the context: in the context of a singleton, the string is taken to be an inline expression
as if it were enclosed in an expr() tag. In an iterative context, the string is assumed
to be a query without additional parameters, which is to say, an equivalent of query(
string, current() ).
Numeric and boolean literals
Numeric and boolean literals are always interpreted as if they were enclosed in quote().
Consequently they may not appear in an iterative context.
Functions
Values with the type of function are treated as if they were enclosed in bind().
JXL tags
dojox.jtlc.tags.acc
acc( input )
Ensures that the value of its argument (usually a constant) is stored in a local variable (accumulator).
Usually used as part of an aggregation construct such as expr( '$1+=expression',
input, acc(0) ). It is not an error to use acc()
in an iterative context but the tag will have no effect as the accumulator will be overwritten by the next
iteration.
dojox.jtlc.tags.arg
arg( index )
Returns one of the arguments passed into the evaluator of the template. While current()
defaults to iterating over arg( 0 ), the two are not equivalent:
arg() is not a generator and should be enclosed in from() to produce
one. The argument of arg() should be a numeric value (usually a constant).
dojox.jtlc.tags.bind
bind( function [, input0...inputN ] )
Evaluates the input0...inputN arguments, then applies function to them. In
the iterative context, input0 is treated as the generator, the rest of the arguments are
evaluated only once. If the input arguments are omitted completely, bind()
assumes that the function should be applied to current().
dojox.jtlc.tags.current
current()
In a singleton context, returns the value of the current input. Current input defaults to the value of
arg(0) but may be temporarily reset to different objects in a
sub-template enclosed in primitives such as each(), group() and
setkey(). In an iterative context, current() serves as a generator
of values from current input.
dojox.jtlc.tags.defined
defined( [ input ] )
Serves as a filter passing only the defined values (that is, values whose typeof is
not equal to 'undefined') from the input. Should not be used outside of
iterative contexts.
dojox.jtlc.tags.each
each( input0 [, input1...inputN ] )
Performs nested iterations by evaluating its first argument with current input set to the value
generated by the second argument and so on. Thus the rightmost argument forms the outermost
iteration and the leftmost argument forms the innermost iteration. This primitive can only be used
in iterative contexts and evaluates every one of its arguments in an iterative context of its own.
Note that each() does not nest sinks, only the iterations. In fact, its usual
application is flattening hierarchies formed by nested arrays passed into the template.
dojox.jtlc.tags.expr
expr( expression [, input0...inputN ] )
Evaluates its input0...inputN arguments, then evaluates the inline expression,
substituting the values of the arguments for placeholders $ and
$0...$9 as follows:
- $0...$9
- Values of input0 through input9.
- $
- In singleton context, equivalent to $0. In an iterative context, current
value generated by input0.
If the expression contains no placeholder strings and begins with an identifier, expr()
assumes it to be a property reference on the current input and prepends "$." to it.
If no input arguments are provided, the expression is evaluated with input0
equal to current input. Note that expr() is entirely similar to bind() except that
bind() refers to a function outside of the template's evaluator and expr()
injects Javascript code directly into the evaluator's body.
dojox.jtlc.tags.from
from( input )
Evaluates the input argument as a singleton and generates values from the array it returns as its
result. Applying from() to current() has no effect because
current() already serves as a generator in iterative contexts and from()
cannot be used as a singleton.
dojox.jtlc.tags.group
group( keys, body [, input ] )
Provides SQL-like grouping capability by evaluating its body over subsequences from
input. The subsequences are established by evaluating the keys (either a single
sub-template or an array of subtemplates, typically inline expressions) over each element of the
sequence and comparing the results. The longest contiguous subsequence of generated values where
keys are equal forms a group. The body is then evaluated with
current input set to the entire group. The group() primitive can be used only
in iterative contexts; it generates values returned by the body: one value per group.
Note that for the grouping to work properly the input sequence should already be
ordered in respect to the keys so usually group() is applied to the
results produced by a query(). However, the user is left free to group sorted sequences
originated elsewhere. If input argument is omitted, current input is used.
dojox.jtlc.tags.keys
keys( [ input ] )
Generates keys from the input: property names if input is an object, indices
if it is an array. If input is omitted, current input is used. Cannot be used outside
of an iterative context.
dojox.jtlc.tags.last
last( [ input ] )
Serves as a special type of sink, returning only the last value generated by input. Usually
used for aggregation purposes. If input is omitted, current input is used.
dojox.jtlc.tags.many
many( input )
Evaluates its input in an iterative context if none exist already. If many()
is used within an externally established iterative context it does nothing.
dojox.jtlc.tags.one
one( input )
Evaluates its input in a singleton context even when used within an externally established
iterative context (in which case one() effectively works as a generator
returning a single value).
dojox.jtlc.tags.query
query( query [, input0...inputN ] )
Executes a query over the specified inputs (each of the arguments input0...inputN
is evaluated as a singleton). By default, the query is pre-compiled with
dojox.json.query and the resulting evaluator is associated with the compiled template
via a closure; changing the queryLanguage setting allow the user to use a different
query language implementation following the same paradigm.
In iterative contexts, query() works as a generator returning individual
matches from the query. In singleton contexts query() assumes that one
and only one result should be produced, which is returned on its own. If query returns
no matches, the result will have an "undefined" value; if query returns more than one
match only the first one will be used. This default behavior may be modified with the settings
singletonQuery.failOnNoResults and singletonQuery.failOnManyResults.
Using query() without the input arguments applies it to current input.
dojox.jtlc.tags.quote
quote( value )
Treats value as a literal even if it is a Javascript object, array or string.
dojox.jtlc.tags.replace
replace( format [, input ] )
Formats and returns its input by performing substitutions within the format string.
By default, replace() uses dojo.replace(); user can change this
behavior with replaceLanguage setting. If the input argument is omitted,
replace() works with current input.
dojox.jtlc.tags.setkey
setkey( body [, input ] )
Changes the key (property name) under which computed values are stored in a dictionary. The input
argument is evaluated first, then body is evaluated as a singleton with current input set to
the value returned by input; usually, the body consists of a replace(), possibly
in combination with expr(). The result returned by the body is used as the key,
the value from the input serves as the result of the setkey() itself. The
setkey() primitive can be used both within singletons and iterative contexts, but makes
little sense in the former. Without the input argument, setkey() operates
on the current input.