Writing Total Commander Plugin in Visual Basic (or C#)

Windows Explorer is not enough. People who have the same feeling as me can choose of several alternative file managers. One of the most popular is Total Commander - shareware file manager for Microsoft Windows. It has classic 2-panes view known from Norton Commander from old DOS times and supports many file operations. One of important features of Total Commander is ability to be extended via plugins. This article describes how to write Total Commander plugin in managed language (Visual Basic or C#) in general and with special focus on File System plugin (wfx).

Plugin architecture

Total Commander itself is written in Delphi (Object Pascal) language (unmanaged). Delphi compiler produces C-style executables and libraries. Plugin interface is designed for plugins written in C/C++. Author provides C header files for plugin authors. There are 4 plugin types:

Content plugin (wdx)
Allows Total Commander to show additional details of files (like ID3 track name or Exif camera model) and use them for search or renaming.
File System plugin (wfx)
Allows Total Commander to access another file systems like Linux-formatted partitions, device file systems or FTP servers.
Lister plugins (wlx)
Allows Total Commander to show content of files of various types (like MP3, HTML etc.).
Packer plugins (wcx)
Allows Total Commander to access - show, extract, pack and manipulate content of various types of archives (like CAB or MSI).

Each plugin is created as a C DLL library which exports defined set of functions. Library has extension wdx/wfx/wlx/wcx instead of dll depending on which type of plugin it represents. Set of functions that plugin must export is defined by Total Commander plugin interface. There are only a few compulsory functions for each type of plugin and then there are plenty optional functions. Compulsory functions provide very basic necessary functionality of plugin. For file system plugin only 4 functions providing initialization and list of files and directories are compulsory. Then there are optional functions for downloading, uploading, deleting, renaming etc. Of course there are no limitations on Total-Commander-unrecognized functions plugin can export. As newer and newer versions of Total Commander are developed set of supported plugin functions grows. So, when using plugin designed for newer version of TC with older one, some functions are never called.

So, writing Total Commander plugin in C++ is easy task. It's an easy task in any language that can export functions in C-like way, like Delphi or PowerBasic. But managed .NET languages (like Visual Basic or C#) cannot export functions in this way. .NET can export COM objects, but not those Win32-API-like functions, even though .NET can import them using DllImportAttribute and Visual Basic has special syntax for importing DLL functions. As far as I know it is even technically impossible to export those functions from managed assembly because .NET does not provide way how to place function on static address - .NET functions got its addresses when they are JIT-compiled. Another limitation, especially for Visual Basic, is lack of support or pointers widely used in TC plugin functions. OK, simple answer to question "Can I write Total Commander plugin in Visual Basic (C#)?" is "No.". As you probably guess, "No" is not the answer I put up with.

There is one a little bit special language in .NET that can combine managed and unmanaged code in one assembly, that can export Win32-style functions, that can define global functions, that can work with pointers - it's C++/CLI. So, my solution how to write managed Total Commander plugin is:

  1. Write interface between Total Commander and managed code in C++/CLI.
  2. Write plugin in any managed language.

The C++/CLI interface is being called by Total Commander, converts all that ugly data types from C++ like char* to nice managed types like String. Managed plugin then do its work and returns what is should return. C++/CLI interface converts managed return value to unmanaged one and passes it back to Total Commander in requested way.

If interface is implemented simply as written above, you must write (copy & paste) the interface again and again for each plugin you'll write. What I wanted was some general purpose solution. So, I've created C++/CLI assembly that contains some support classes and structures for passing data between managed and unmanaged code and above all it contains plugin base class actual plugin implementations are derived from. So, the way hot the plugin is implemented is fully object oriented as it is common in .NET. Basic parts of Total Commander plugin managed framework are:

C++/CLI umnanaged ↔ managed interface (the Tools.TotalCommander assembly)
Plugin base class and support classes and structures are contained in this assembly.
Plugin implementation
The managed assembly (dll) that implements the plugin. It can even implement more than one plugin or plugins of different types. It can be written in any managed language that can derive from plugin base class.
Plugin assembly
Small assembly written in C++/CLI. This assembly represents the plugin from Total Commander point of view. It has the required extension (wdx/wfx/wlx/wcx) and it exports all the necessary functions. It initializes the plugin instance and then it simply calls plugin instance function whenever Total Commander calls plugin function. This assembly is generated by Total Commander Plugin Builder from plugin implementation.
Total Commander Plugin Builder
Command line tool that builds plugin assembly from predefined template using information from plugin implementation. Especially it ensures that right type of plugin is built and that only functions implemented by plugin-implementing class are exported by plugin library.

Interface between Total Commander and managed code

As written above this assembly is written in C++/CLI and performs marshaling between unmanaged (Total Commander) and managed (plugin implementation) code. It contains some support classes and structures, some of them are visible from managed code. It also defines several attributes used to specify way how Plugin Builder builds the plugin assembly. In fact this assembly contains only very little portion of unmanaged code - only definition of unmanaged structures imported from Christian-Ghisler-provided header files (and those header files include a few Windows SDK header files). But code in this assembly deals with those unmanaged types and with pointers and C++-like strings (char*). It's something we avoid in C# and can't do in Visual Basic.

The plugin (abstract) base class simply contains non-virtual (not overridable in VB) functions accepting and returning unmanaged types and then it contains virtual (overridable in VB) functions overriden by actual plugin implementation. Virtual functions accept and return managed and CLS-compliant types. Non-virtual function converts its parameters from unmanaged to managed types and passes them to virtual function. When the virtual function returns its return value (and out parameters' values) is converted from managed to unmanaged types and passed to caller. The caller is actually global function in plugin assembly which passes the values back to Total Commander. Virtual function differs in behavior from non-virtual ones. For example exceptions are used instead of error return codes and return value instead of output parameters (sometimes output parameters cannot be avoided - multiple return values).

In File System plugin the mapping between non-virtual and virtual functions is usually 1:1. For almost each non-virtual function (starting with the FS prefix) corresponding virtual function exists. Virtual functions for compulsory functions have no implementation which effectively makes plugin author to implement them in derived class. Optional functions have default implementation throwing NotSupportedException. It is ensured that Total Commander never calls optional function which is not overriden in plugin class because such function is not exported by plugin assembly (Total Commander Plugin Builder does not generate export for it). From object oriented point of view it can be determined that actual implementation of optional function does nothing but throws NotSupportedException by MethodNotSupportedAttribute applied on the method.

Sometimes the interface provides a little bit higher level of abstraction than unmanaged Total Commander plugin interface. It does not use handles but actual objects - bitmaps and icons.

The plugin assembly

Plugin assembly contains code responsible for creating instance of plugin and it actually exports the functions to unmanaged environments and passes function calls from Total Commander to plugin abstract base class. As the plugin assembly is generated by Total Commander Plugin Builder from template using information from plugin implementation, it is customized for actual plugin it represents - and it always represents only one plugin. In case plugin implementation assembly contains more plugins, more plugin assembles are generated.

The most tricky part of work of plugin assembly is assembly binding. The plugin assembly has references to Tools.TotalCommander and to plugin implementing assembly. Tools.TotalCommander refers to Tools and plugin implementing assembly may refer to any assembly - locally copied or GAC. Total Commander plugins usually resides in subfolders of subfolder plugins of Total Commander installation folder. And now problems arise: Plugin assembly is loaded to the totalcmd.exe process. Totalcmd.exe resides 2 or more folders above plugin assembly. Plugin assembly refers to plugin implementation assembly and Tools.TotalCommander, neither of them is in GAC. By default .NET looks for references in the same directory as the process was stared in. Subfolders are not examined. So, references are not found and plugin crashes. Total Commander can recover from it, but plugin is not loaded and it does not work. The only assembly that is correctly loaded is plugin assembly, because it is loaded as part of something that seems to be unmanaged Win32-API-style DLL. So, the plugin assembly must ensure that references will be searched where they lie. We can solve the issue in several ways:

Place all the assemblies in same directory as totalcmd.exe.
It is not good idea as many, possibly conflicting, files will be in Total Commander installation directory. This may lead to chaos problems.
Place all necessary assemblies in GAC
This is also not very good solution. First it puts additional demands to plugin installation process and requires administrator privileges. It prevents plugin from being carried with Total Commander when it is installed on flash drive. And single-purpose assemblies as Total Commander plugin implementation assembly should not be in GAC.
Intercept assembly resolution
We can handle event AppDomain.AssemblyResolve which is raised when assembly resolution fails. Handler of this event can load the assembly and return it. Problems will arise when multiple managed plugins are used with Total Commander. Plugins can use different versions of Tools.TotalCommander or plugins have not to be based on this framework. This assembly resolution can effectively destroy other managed plugins.
Load each plugin into separate application domain
This is the way I've finally chosen. Only the plugin assembly is loaded to default domain. It creates another domain and sets its base directory to directory it is located in. Then it creates instance of helper class in newly created application domain. The class creates instance of plugin class - it is resolved correctly, because all references are in its base directory. Assemblies loaded by plugin does not interfere with other  plugins because application domains are separated. There is only some overhead because TC calls global function in plugin assembly, it calls function in assembly domain helper, it calls function in plugin helper, it calls non-virtual function in plugin class and it finally calls virtual function in plugin class.

Total Commander Plugin Builder

A command line tool that creates plugin assembly for each plugin class in plugin implementation assembly. It can be invoked from command line or it can be used programmatically. It needs access to C++/CLI compiler vcbuild.exe. It is written in Visual Basic. The best way of using Total Commander Plugin Builder is to have it in post-build event in Visual Studio.

It enumerates all types in plugin implementation assembly and for those that represent Total Commander plugin generates plugin assembly. While generating plugin assembly, the plugin class is examined to determine which plugin functions are implemented (overriden) by plugin class. Methods that are not implemented are not generated in plugin assembly. It is achieved simply by writing several C++ preprocessor #defines to control how the plugin assembly will be compiled. Same way the name of class to create instance of is specified. Reference to plugin implementation assembly is set by the #using C++ directive. Total Commander Plugin Builder also examines certain attributes of plugin implementation assembly and plugin class to set plugin assembly attributes and to refine generation behavior.

The Code

OK, I'm not gonna to re-type all the code here. Lets download attached example. Only a few interesting parts of the code:

Creation of application domain

Following code snippet in C++/CLI shows how the application domain is created:

namespace Tools{namespace TotalCommanderT{
    extern bool RequireInitialize;
    extern gcroot<AppDomainHolder^> holder;
    //PluginInstanceHolder class keeps plugin instance
    PluginInstanceHolder::PluginInstanceHolder(){
        this->instance = TC_WFX; //TC_WFX ide C++ preprocessor macro defined to somethig like gcnew MyPluginClass()
    }
    //AppDomainHolder keeps AppDomain instance
    AppDomainHolder::AppDomainHolder(){
        this->holder = gcnew PluginInstanceHolder();
    }
    //Global function initialize is called by plugin functions FsInit and FsGetDefRootName which can be called as first call to plugin by Total Commander
    void Initialize(){
        if(!RequireInitialize) return;
        RequireInitialize = false;
        PluginSelfAssemblyResolver::Setup();
        AppDomainSetup^ setup = gcnew AppDomainSetup();
        Assembly^ currentAssembly = Assembly::GetExecutingAssembly();
        setup->ApplicationBase = IO::Path::GetDirectoryName(currentAssembly->Location);
        AppDomain^ pluginDomain = AppDomain::CreateDomain(PLUGIN_NAME,nullptr,setup);
        AppDomainHolder^ iholder = (AppDomainHolder^)pluginDomain->CreateInstanceFromAndUnwrap(currentAssembly->CodeBase,AppDomainHolder::typeid->FullName);
        Tools::TotalCommanderT::holder = iholder;
    }
}}

PluginSelfAssemblyResolver is simple helper class that allows resolution of assembly itself when it cannot be found by .NET. It contains only two functions:

namespace Tools{namespace TotalCommanderT{
    Assembly^ PluginSelfAssemblyResolver::OnResolveAssembly(Object^ sender, ResolveEventArgs^ args){
        AssemblyName^ name = gcnew AssemblyName(args->Name);
        if(AssemblyName::ReferenceMatchesDefinition(name,thisAssembly->GetName())) return thisAssembly;
        else return nullptr;
    }
    inline void PluginSelfAssemblyResolver::Setup(){
        AppDomain::CurrentDomain->AssemblyResolve += gcnew ResolveEventHandler( PluginSelfAssemblyResolver::OnResolveAssembly );
    }
}}

Definition of plugin functions

Optional as well as compulsory Total Commander plugin functions are wrapped in #ifdef-#endif blocks. Corresponding #defines for that blocks are writen by Total Commander Plugin Builder to the define.h file. Because, due to the architecture using application domains, those functions are in plugin assembly 3 times with identical signature and similar body, I've extracted them to separate file wfxFunctionCalls.h. This file is included at 3 different places with several C++ preprocessor #defines to control how it is compiled. Each function is defined like this:

#ifdef TC_FS_INIT
    TCPLUGF int FUNC_MODIF FsInit(int PluginNr,tProgressProc pProgressProc, tLogProc pLogProc,tRequestProc pRequestProc){
        return FUNCTION_TARGET->FsInit(PluginNr,pProgressProc,pLogProc,pRequestProc);
    }
#endif

TCPLUGF is defined as empty (not used). Once I thought about using __declspec(dllexport)to export functions. Lately I've switched to seperate Exports.def file.
FUNC_MODIF is either __stdcall or class name (AppDomainHolder:: or PluginInstanceHolder::). __stdcall is used for exported functions, internal calls use managed calling convention.
Finally FUNCTION_TARGET is instance to call function on. It is Tools::TotalCommanderT::holder in exported (global) functions, this->holder in AppDomainHolder and this->instance in PluginInstanceHolder.

WfxFunctionCalls.h is included like this:

#define TCPLUGF
#define FUNC_MODIF AppDomainHolder::
#define FUNCTION_TARGET this->holder
#include "FunctionCalls.h"

I now, it may be more comprehensible typing it 3 times. But so many functions typed 3 times - I'm really lazy and besides when some change is needed it is done only once (in plugin assembly template; then in plugin base class and in common header file and ...).

Marshalling

Marshalling from unmanaged to managed code is quite simple. Total Commander plugin interface uses several structures and constants. For structures I've created managed counterparts and before passing object to managed code structure is converted to managed one. Before structure is returned to unmanaged code it is converted back. Constant values are represented by managed enumeration values and are simply cast. Something very often passed between Total Commander and plugin are strings. Total Commander passes and accepts strings as char* (sometimes char[MAX_PATH]) - always null-terminated. Marshaling those values from unmanaged to managed code is easy, because System.String has constructor that accepts char* (System.SByte*).

Note: In current version Total Commander neither passes to plugins nor accepts from them Unicode strings (wchar_t*) although a few Win32 API structures used by Total Commander are declared as Unicode. Total Commander uses current system encoding. Unicode support will be in next one of future versions of Total Commander. This is not limitation of my framework but of Total Commander itself.

Passing string to unmanaged code is little more tricky. It is possible to enumerate all the characters of string easilly i .NET. But those characters are Unicode code points. They must be converted to default system encoding values. Finally, I've created my own StringCopy functions:

namespace Tools{namespace TotalCommanderT{
    void StringCopy(String^ source, char* target, int maxlen){
        if(source == nullptr)
            target[0]=0;
        else{
            System::Text::Encoding^ enc = System::Text::Encoding::Default;
            cli::array<unsigned char>^ bytes = enc->GetBytes(source);
            for(int i = 0; i < bytes->Length && i < maxlen-1; i++)
                target[i]= bytes[i];
            target[source->Length > maxlen-1 ? maxlen-1 : source->Length] = 0;
        }
    }
    void StringCopy(String^ source, wchar_t* target, int maxlen){
        StringCopy(source,(char*)(void*)target,maxlen);
    }
}}

The second function simply treats wchar_t* as char*, see note above.

Function encodes string using default system encoding and then copies encoded bytes to unmanaged buffer (maximally maxlen - 1 characters). Character after last used character is set to nullchar.

Note: I'm not sure if the default encoding will behave correctly in systems where default encoding is multibyte (e.g. Chinese). I hope for soon implementation of Unicode in TC.

Sample plugin

Sample plugin is IMHO the simplies that can be written. It simply accesses local file system. Christian Ghisler provides such example plugin in C++. Mine is written in Visual Basic. It shows hot to utilize Managed Total Commander Plugin Framework.

Notes

Known issues

License

Plugin framework, plugin builder, sample plugin as well as any other code in this article is released under Open Source license at codeplex.com/Tools.