ModMaker.Lua v0.9.2 Alpha
By: ModMaker Productions
Copyright © 2013 ModMaker Productions
Copyright © 2013 ModMaker Productions
ModMaker.Lua is an framework library for .NET that parses, compiles, and runs Lua code. This was written from scratch according to the Lua 5.2 Language Definition. This manages all the work for registering functions, managing the stack, and manipulating user-defined types. This is designed to be easy to use and manipulate. This library has few dependencies and only requires .NET 4.0. Most of the behavior can be modified to suit your needs.
This framework makes a few changes to the Lua language to support new features; however all Lua 5.2 code is compatible with this framework and there are alternates so your code can work with native Lua. This framework allows many of the familiar C# language features like classes (§3.1), method overloads (§3.1), by-reference parameters (§3.1.3), and automatic casting. This also manages user-defined types , registering methods, and controlling visibility (§3.3). The framework requires no special permissions except the default IO and OS libraries may require some when invoked, (§4).
The main managing object is Lua. You can specify a LuaSettings object to modify the default environment. It is suggested that you pass an instance when using the default environment (§5). Once you have an instance of Lua, call Lua.DoFile(string) and pass a file name, in this case "world.lua". This method will load the file and execute it.
program.cs
using ModMaker.Lua; namespace MyProgram { public static class Program { public static void Main(string[] args) { Lua lua = new Lua(); lua.DoFile("world.lua"); } } }
Now we need the Lua code file. The syntax for Lua has not changed (§3) so simply use normal Lua code:
world.lua
print "Hello World"
This framework does all the work when registering types and delegates for use in Lua code. To register one, simply pass it
to Lua.Register. You can also specify a different name. If no name is given, the simple name of the type is used or the name
of the method. After you call this method, the type or delegate is visible to Lua code. Note that this overrides the behavior
of LuaIgnore (§3.3.1). If you do not want to use an explicit cast when registering delegates, include
'using ModMaker.Lua.Extensions'.When a method is invoked from within Lua, overload resolution
(§3.1) is used to determine the version to invoke. If the arguments cannot be converted to any overload,
or if there is two methods that match the given arguments, an exception will be thrown.
lua.Register(typeof(T)); lua.Register((Action<int>)Foo);
There is a difference between a type being visible to Lua code and registering a type. If a method returns an object, it can be used within Lua even if the type is not registered. When you register a type with Lua, you give access to two things: (1) the visible constructors, and (2) the visible static members. An object that is returned from a method is not affected by whether it is registered. By default, all public members of types are visible to Lua code. Also, optionally the framework will respect return types by making only the public members of the returned type visible. There are two ways to change this behavior: LuaIgnoreAttribute (§3.3.1), and LuaUserData(§3.3.2). If there is a conflict, a member will not be visible.
There are several ways to load and execute Lua code. Initially, we used DoFile. This will automatically parse and the given path. There is also a similar method called DoText that will do the same thing except the argument is the loaded plain-text code. These methods will use the Parser and Compiler specified by the current environment (§5). If you want to simply parse the code, you can call Load. This will parse and compile the given code. There are overloads to specify a name, path or stream, and an environment to compile with. If you specify an environment, it will not change the current environment and will only use the given one to load. You can also load code using LoadText if you have the plain-text code.
When the code is loaded, it is returned as an IMethod object. When Invoke is called on this object, the code will execute in the current environment. The object is returned by each of the methods and is also stored in the Lua object. The loaded objects can be retrieved using the indexer. The loaded code can also be executed using Execute. Execute will execute all the loaded code in the order that they were loaded and pass the given arguments to each one. It will then return an array of the returns. You can also specify an index to execute only a single chunk.
There are several types regarding invokable objects in this framework. This section will describe the differences between the types and how they should be used. This only describes how the default environment does it and how it is assumed to work, if you change the environment, it may not be the same. The most likely use of the different types is when you want to accept an argument to a C# defined method and you only want a certain kind of method.
IMethod simply describes an invokable object. This may be a method or delegate, or it can be a user-defined object that supports being invoked. You can implement this interface and it will allow Lua to invoke your object like a method. This interface is the base type of an invokable object. LuaMethod implements this interface, but its use is slightly different. This is intended for an object that supports invoking.
This is the base class of all functions that are used in Lua. There are several different derived classes that are used in different circumstances. This class is meant for an object that is only a function such as a delegate. This should not be used if your objects simply can be invoked. This distinction is not important as the framework simply will call _IMethod.Invoke_; however other users can make this assumption when accepting arguments.
If you want to define your own method, you should select the best object to subclass. If it is simply an object that can be invoked, or if you want another base type, you can implement IMethod; otherwise you should derive from LuaMethod. When you do, you need to pass the current environment and a string name to the constructor. The name can be null, it is only used for error messages. Then you need to implement the abstract member InvokeInternal. This is what does the work of invoking your object. The args and byref will never be null. 'overload' may be negative if an overload is not specified. You do not need to use the overload and will most likely ignore it.
This section describes many of the features that this framework provides in addition to traditional Lua features. Some of these features are optional and may not be supported if using another environment. If you are using the default environment, then all these features are implemented. This also describes the changes to the Lua language. These are changes to the syntax and make the Lua code invalid in other compilers. In many cases, there are alternates to use.
You can register multiple (.NET) delegates to a single name and the runtime will automatically select the best one based on the runtime types of the arguments. This allows for C# style overloading. The overloads only work if they are all .NET delegates and there is not another with the same signature registered with that name.
The behavior of overload resolution is not specified, so it may differ when using a different environment. The default overload resolution works in a similar manner than in C#. First an overload is searched for that has the least number of conversions needed to make the arguments work. If there is ambiguity, it favors more explicit arguments over optional and params arrays. Otherwise it throws an ambiguous match exception.
When there are two methods that both match the given arguments, it may be needed to specify which overload to chose from. You can specify an overload with the function overload. The function accepts a LuaMethod object and a zero-based index of the method to invoke. You can also by adding a grave (`) to the name of the method and then the zero-based index of the method to invoke. If the invoking object was registered with Lua.Register, the indexes are the order that they were added. For user-defined objects, it is the order returned from Type.GetMethods, this is usually the order that they appear in the source file (but not necessarily).
overload(Foo, 2, "Arguments...") Foo`2("Arguments...") -- this also works for member methods obj.ToString`1(12)
When passing arguments to .NET, some methods may accept arguments by-reference (e.g. 'ref' and 'out' in c#). This framework supports passing arguments to these methods and altering Lua values to match. This means that your C# code can modify Lua variables with no alteration of C# code. However, there are some changes that need to be made to Lua code to support by-reference. First, only variables can be passed by-reference. Second, the arguments need to be marked as by-reference. This can be done in two ways: (1) prefixing the variable with '@', or (2) using the new 'ref' keyword. Here is an example:
i = 12 t = { [23] = "Pie" } -- the '@' has the highest precedence and must come first Foo(@i, @t[23]) -- the 'ref' can be specified with optional parentheses Foo(ref i, ref(t[23]))
Note that 'ref' is a keyword and cannot be used as a name. Also note this is not a method, it can only be used in a function call. This is used so by-reference code can be used as valid Lua code using the native compiler. If ref is used anywhere else, it contains more than one argument, or the argument is not a variable, then a syntax exception will be thrown.
An overload that accepts a by-reference parameter is not compatible if the given argument is not passed by-reference. However, if a variable is passed by-reference and the argument is not, then it will have no effect. Overload resolution favors methods with more by-reference arguments over those without. If passing an argument by-reference to a Lua or framework method has no effect also.
This framework allows you to define new types within Lua code. This uses similar syntax to LuaBind. It uses the new 'class' keyword to define the new type. The types defined in Lua can be initialized within Lua or can be loaded into .NET and used there. This framework also supports inheriting from .NET types so you can use static binding.
There is two syntaxes for defining a new class. One is similar to the syntax of LuaBind, and the other is similar to the syntax in C#. Classes are defined as a new object in the globals table so a class can be defined in a function. The class definition must come before any member definitions and there cannot be an object in the globals table with that name already. To add members, simply index a variable with that name (see below). When the object is invoked it will create an instance of the created type; however, it will generate the type and any changes to the type after that point will throw an exception. The type will be generated if CreateInstance is called on the object from .NET.
-- similar to C#, the name of the class and -- optionally a colin followed by the names of the base type or interfaces class MyClass : IDisposable -- similar to LuaBind, the name is in a string and -- optionally a parrentheses and the base type or interfaces class "MyClass" (IDisposable)
The types that are defined in Lua can extend another type or implement interfaces. To do this, the type must be visible according to LuaSettings.ClassAccess. The types are stored as a string and are resolved when the code is executed. Any abstract member or interface method that is not defined when the class is instantiated will be generated and throw a NotImplementedException. Due to .NET, the class may only derive from a single base class. The order of implementations does not matter and the base class does not need to be first.
You can define new fields or properties from within Lua. The field or properties that are defined are defined as Public, unless you are changing a virtual member. The type of the field depends on the type of the first object that it is set to. To define a field or property, you simply set a member of the class to an initial value. If set to function, see below. If set to anything else, it will be defined as a field with that initial value. You can set a field to a type (if it is registered) to define that field as the given type. If the base type defines a member with that name and the type is not compatible, it will throw an exception.
If the base class or interface defines a property with the name of a field you set, the behavior is different. If set to a table, then it must define only a 'get' and 'set' and they must be the get/set functions for the property respectively. If set to a value, then it will create a get method that will always return the given value and (if needed) a set method that will throw an InvalidOperationException.
-- define a field with type of 'int'. MyClass.MyField = int; -- define an initial value for the field. MyClass.MyField = 25; -- define a field of type 'LuaTable'. MyClas.MyProperty = { }
When a function is defined in Lua for the class, the base class and interfaces are searched for a function with the given name (see below). The return values are converted as needed and an exception is thrown if there is an error. Make sure to define methods with a colon ':' otherwise it will mess up the arguments because the first argument will be 'this'. There is also a pseudo-argument called 'base' that is defined for member functions that allows access to the base type the same was as in C#. If you define a method with the name '__ctor' it will be called inside the constructor. Any arguments passed to the constructor will be passed to this method.
If a method is defined in the base class, it must be defined as virtual/abstract; otherwise an exception will be thrown. Any interfaces that define methods with that name are ignored. If it is marked with virtual/abstract but there are multiple overloads, then you must specify an overload (§3.1) or an exception will be thrown. Otherwise the method will be overridden. If an interface defines a method with that name and there is only one overload, then it is implemented implicitly. If there is multiple overloads, an exception is thrown unless the overload is specified. If there is more than one interface that defines method(s) with that name, it they are ignored. You can also define a method explicitly using similar syntax as C#.
-- make sure to use the colon ':' otherwise the 'self' argument will not be defined function MyClass:Foobar() { } -- this will implicitly implement IDisposable.Dispose -- (if IDisposable is implemented and no other conflict). function MyClass:Dispose() { } -- this will explicitly implement IEnumerable.GetEnumerator. function MyClass.IEnumerable:GetEnumerator() { }
LuaIgnoreAttribute is used to control visibility for all variables of a given type. This is an attribute that can be attached to a member or type to alter what members are visible. The exact behavior depends on what the attribute is attached to and what arguments are passed. This attribute is inherited.
When LuaIgnoreAttribute is attached directly to a member, the given member is not visible to Lua code. There is no way to modify this behavior. Any arguments passed to the attribute will be ignored. If this is attached to a member of an interface, than it will only have affect if the type is behaving-as (§3.3.3) the given type.
If the attribute is attached to a type there are several cases that it can be depending on the arguments that are passed. If the class is inherited, this attribute will come with it, however it can be overridden by attaching the attribute to a derived class. If the sole argument is 'true' then it behaves the same as no arguments; if it is 'false' then all members are visible (same as without the attribute). If the sole argument is a Type object, then a variable of this type will behave-as (§3.3.3) the given type.
You can change the behavior beyond these defaults. To start, call the constructor with 'false' as the first argument. Then you can add members either to AccessMembers or IgnoreMembers. If AccessMembers is not null, a member must be in that array to be visible. If IgnoreMembers is not null, then a member must not be in that array to be visible. If DefinedOnly is true, than only members that are explicitly defined in this type are visible. If BehavesAs is not null, than the type will behave-as (§3.3.3) the given type.
LuaUserData is a special variable type that is used to control visibility for a single variable. There are two versions of this type, the generic and non-generic versions. The generic version derives from the non-generic version and only the non-generic version is used internally. The generic version should be used by users to ensure type-safety with arguments. When a variable is passed back to Lua code, it sometimes is stored as a LuaUserData variable. If a variable is passed to Lua through a return value or a by-reference argument, then it is usually stored in a LuaUserData variable to ensure that it is treated as the given type.
When a LuaUserData object is passed as an argument to a function, it is valid to convert it to it's backing type. However, it may be helpful to accept a variable of type LuaUserData to ensure that the visibility rules are not lost or violated. The generic version ensures that a variable of the given type is passed. The non-generic version is the same as accepting a variable of any type (type object).
The behavior of this object is the same as for LuaIgnoreAttribute (§3.1.1), except that it is for a single variable. Also, there is a field called CanPass, This ensures that special variables are not passed back to C# code.
Behaves-as is used in both LuaUserData and in LuaIgnoreAttribute. This means that a variable or type will behave as if it was the given type. This is the same as storing a variable of one type in a variable of a base type or interface. This is used in LuaUserData to ensure that Lua cannot violate accessibility when return types do not match the backing object type. This can also be used with LuaIgnoreAttribute when attached to a type. This means that variables of that type always behave as if they were the given type. This can be used if you define a class as the default implementation of an interface that you want to expose to Lua. The default environment is marked in this way to ensure that Lua cannot access the method on DynamicObject.
When an object is behaving as another type, that other type entirely defines what members are visible to Lua. The members are still invoked virtually, however new members defined in the derived type will not be visible. When a member of an interface is marked with LuaIgnore, it does not effect the visibility of that member unless the type is behaving as the interface type. This is because attributes are not inherited from interfaces unless the member is implemented explicitly and that would mean that the member is not public and then cannot be accessed by Lua.
There were several changes to both the Lua language and the framework methods. Due to the behavior of .NET and how the Lua code is compiled, there were some changes that needed to be made to make it work better. The only language changes were described in Framework Features (§3), this section describes the changes to the framework functions.
This library uses System.String to manipulate strings, so it supports Unicode characters and assumes multi-byte chars. Indexes are the same as in the Language Definition, 1 is the first letter and negative incidies count from the end of the string (i.e. -1 is the last char). Indicies also refer to the char index, not the byte index. If you do not include LuaLibraries.String in LuaSettings, the global string will point to System.String and will allow you to call the static members in it.
This library simply converts the arguments and passes them to the respective function in System.Math. There are no changes to this library, but there is something to be aware of for math.random and math.randomseed. This library uses a static System.Random field to get random values. An exclusive lock is acquired each call to ensure thread safety. Calling math.randomseed will create a new System.Random object with the specified seed, must be an integer. Also, math.huge returns double.PositiveInfinity.
This library requires the respective System.IOPermissions to work. Permissions are Demand'ed by System.IO.File. The stream object returned by the functions here are a table with a key of "Stream" that contains the System.IO.Stream object. All functions will accept a table with that member or a System.IO.Stream object.
The environment defines how this framework will behave. It is set in the Lua object and is passed around by the framework. The environment simply contains a LuaTable for the global variables and an instance of each of the other fundamental types. The table defines the global variables and can never be null. The parser object is in charge of converting input text into a DOM code tree. The compiler then converts that into an IMethod object that can then be executed. The runtime defines the behavior of the executed code.
Each of the fundamental types can be changed to alter its behavior. The environment in the Lua object is used by its methods to parse and compile code. This is the object that is passed to the created objects, unless a different environment is passed. Each object that is created may store an environment. Any time that an object stores the environment it can be changed. If it does not store the environment, then it either does not use it or is passed by another object.
The environment is the main type and defines how the framework will behave. This object simply contains a reference to the global table and one of the other three types. It also contains an IThreadFactory object that will create LuaThread objects. This type does little actual work and simply contains each of the other types. This framework assumes that any property will never return a null value and it will be a valid reference. The default implementation is LuaEnvironment and can be extended to alter its behavior.
The parser is in charge of converting input code into a DOM (Document Object Model) tree. This tree is represented as a tree of IParseItem objects. Each item is defined in ModMaker.Lua.Parser.Items. There is also an IParseItemVisitor that can be used with the visitor pattern to traverse the tree. The only method defined is the one that will parse the code. It can throw an exception if there is a syntax error or if there is an invalid argument. If the method returns, it must return a valid IParseItem tree. This object can also optionally use a cache to reduce the time it takes to parse the same code twice. The default implementation is PlainParser and can be extended to alter its behavior.
The compiler is in charge of converting an IParseItem tree into an invokable object (IMethod). The exact object that is returned is not important; however calling invoke on the object should execute the given code. There is a helper visitor object GetInfoVisitor that will traverse a tree and get information about it and store it in the objects. It is important to use this so labels and goto's are resolved. Also, this will get information about captured variables. This object also creates delegate objects from an IMethod object. This is used for dynamic binding so IMethod objects can be stored as delegates. The default implementation is CodeCompiler and cannot be extended.
The runtime defines how the code executes. The methods in the runtime are invoked by the generated code when certain operations are performed. Most of the complicated logic is performed by the runtime allowing users to alter the behavior of Lua code by changing the runtime object. The default runtime is LuaRuntime and can be extended to alter its behavior.
This library is simply a framework for defining the behavior of Lua code. With the four fundamental types, you can change the entire behavior of this framework. With the default implementations, you can pick and choose which portions of code to use and which to change. This section will describe how to extend this framework to suit your needs. It is highly suggested to read the source code to learn how the default implementations work.
The parser is the first step to creating Lua code. It is in charge of converting plain-text code into a DOM (Document Object Model) of the code. This is separated into two parts: (1) the tokenizer, and (2) the parser. The tokenizer starts with some input (usually from a file) and converts this into a sequence of tokens. The parser takes these tokens and converts these into an IParseItem tree.
The ITokenizer is the interface that defines how the tokenizer behaves. Tokenizer defines the default behavior for a plain-text input. The default tokenizer takes a TextElementEnumerator as input. This allows for full Unicode support. This will automatically keep track of position and line. This can be extended to change how it behaves. It defines the behavior of Peek, Read, and PushBack. It then as needed calls InternalRead that does the actual reading from the input. This method will read a single token from the input.
There are several helper functions that are defined. Each helper function can be altered to change how it works. ReadElement and PeekElement will read and peek a single letter (or two for surrogate pairs) and will return null on the end of the stream. ReadElement will also increase the position and Line as needed. It will also convert any line endings into '\n'. ReadWhitespace will read any whitespace characters in the input using PeekElement and ReadElement. ReadComment will read a comment, it assumes that the first two letters have been read '--' and the input is on the next letter. ReadString will read a string, the input must be on the first letter of the string and 'depth' must specify the depth of the string. -1 for ', -2 for ", and a positive number for that number of equal signs (e.g. 3 means '[===['). ReadNumber will read a number, you must pass the first letter of the number and the input must be on the second one.
The parser will take an ITokenizer input and will convert this into an IParseItem tree. The default PlainParser can be extended to alter its behavior. What may be confusing is the different uses of the Token type. The tokens returned fro the tokenizer are used to determine what to add to the output, but are also used to store debug information. Each IParseItem object contains a Debug property that contains the token that defines it. If it is a compound token (e.g. BlockItem) then it is the entire token. This is done by calling Token.Append which will append the given token onto the end of the current token. If a function accepts a token object, this represents the enclosing token and anything that is read should be appended to the end of that token. This is usually done by holding a global 'debug' Token that defines the item that is currently being read and at the end of the function calling append on the enclosing token with 'debug' as the argument.
There are several Read functions that will read different items. ReadBlock does most of the work and will read a block of code. This can either be the main section or the contains of another item, such as an 'if' block. It is important to note that you should not append the 'end' token to what is read by a block, the 'end' token is read and added by the enclosing function. ReadPrefixExp will read a prefix-expression from the input. A prefix expression is a simple expression such as literal or an indexer. It will read the entire indexer(s) as well as any function calls. This may call ReadBlock for an embedded function or ReadExp for function call arguments. ReadExp will read a single expression from the input. If the 'precedence' argument is -1, then it is the initial call and will keep reading until it gets to something that is not an expression; if it is not -1, then it will keep reading until it reaches the end or an expression with a higher precedence. ReadFunction and ReadTable will read a function and table respectively.
The compiler is in charge of converting the IParseItem tree into an IMethod object that can be invoked later. The default compiler cannot be extended due to its complexity. If you want to write your own compiler, you need to do it from scratch. The compiler need to perform two functions, (1) compile IParseItem trees into IMethod objects, and (2) create delegates that will invoke a given IMethod object. If you want to use the default behavior of the second item, you can call the static method CodeCompiler.CreateDelegate in your method. The default compiler uses System.Reflection.Emit to generate IL according to the IParseItem tree. It uses an internal type CompilerVisitor to visit, the parse tree and it generates the code. There is also a ChunkBuilder that helps generate the code.
There is a public type called GetInfoVisitor that will visit an IParseItem tree and will get information about it. It is highly suggested that you use this in you compilers. This will resolve any 'goto' or 'break' statements. It will also search for captured variables and generate a FuncDefItem.FunctionInfo object for each function. This will help to determine which variables are local, captured, and global. The array inside contains any local variables that are captured by nested functions, all others can be real local variables because they are only used in this function. There is also a field called CapturesParrent that determines if it captures variables from the parent function. It also defines a field called HasNested that determines if this function also has nested functions.
The runtime defines how the Lua code will execute. If you write your own compiler, using the runtime is not needed. However, if using the default compiler, you need to use ILuaRuntime. The runtime defines several functions that are called by the generated code. The default runtime (LuaRuntime) can be extended to change parts of its behavior. It is important to read the section on operator overloads (§3.1) to make sure the operation of the function is the same. You can use OverloadInfo and GetBetterOverload to determine overload information. Also make sure to read the next section on methods to correctly support tail-calls.
The Lua specification says that is supports proper tail-calls. This means that if a Lua-defined function returns a call to another function, it will remove it's stack information so it can support infinite recursion. By default, C# does not ever support tail-calls; however you can use IL to support tail-calls with the OpCodes.TailCall opcode. This means that you need to dynamically generate some types to support proper tail-calls. This is what the framework does by default. If you want to create a LuaRuntime instance, you need to call the static method Create and it will create an instance of the dynamic type. This is the same for each of the LuaMethod types. The call to LuaRuntime.Invoke and LuaMethod.Invoke are generated so the tail-call opcode is injected. If you want to support tail-calls, you need to do the same.
This is the call structure of the default runtime when Lua code executes a Lua-defined function. (1) the backing method calls ILuaRuntime.Invoke passing the arguments and the object to call (defined in the default compiler), (2) the runtime then calls LuaMethod.Invoke (as LuaMethod implements IMethod), then (3) that validates the arguments and calls the abstract method LuaMethod.InvokeInternal, (4) this is handled by LuaDefinedMethod.InvokeInternal and it simply calls the dynamically generated method. In each step above, it needs to contain the tail-call opcode, otherwise it will not support infinite recursion. If you want to define your own LuaMethod object, you can use LuaMethod.AddInvokableImpl which will generate dynamic implementations of IMethod that will call InvokeInternal. NOTE that deriving from either LuaMethod or LuaRuntime will NOT support tail-calls, the code for their respective Invoke methods MUST be dynamically generated to support tail-calls.