The current exposition is a detailed explanation of the core aspects of how GNU Emacs’ Dynamic Module system works, and how it can be leveraged to build more efficient Emacs packages in programming languages different than Emacs Lisp. This feature has been introduced in Emacs since version 25 (2016), but rarely taken advantage of, and much less talked about. Why would an Emacser, who very likely also adores Lisp dialects, would ever want to use anything other than Emacs Lisp? The short answer to that is: sadly, the software world we live in as of now doesn’t always cater really nicely to Lisp-based approaches. And the author, as someone who really likes Lisps for their design and implementations, is really sad about this state of conditions in software. GNU Emacs is an embodiment of freedom in the truest sense of the word, not only living to the standards of Free Software from FSF but also, practically enabling the users–programmers and others alike–to build anything they wish on top of Emacshttps://xkcd.com/378 without having to ask for permission of anyone.
Meanwhile, while the problem exceeds more than just being able to build and extend artifacts on top of Emacs, the programmer needs to be considerate of several design decisions and must ensure that the artifacts don’t sacrifice other important qualities (e.g, efficiency, security, modularity etc.) for the sake of integration with Emacs. Due to the existing conditions, certain other programming languages are well-suited for tasks where Emacs Lisp performs poorly. Communicating with well-maintained libraries (that lack bindings to Emacs Lisp, or any Lisp dialect) and performance are two of such pressing concerns.
The author’s moment of realization of this came late, and unlike one that happens with Lisphttps://xkcd.com/224, when I was figuring out the occasional drops in performance of an Emacs package which is ubiquitously used for rendering PDFs in Emacshttps://github.com/vedang/pdf-tools. A detailed explanation is unwarranted for the current text, but to bring the point home on why Dynamic Modules are a much more preferable approach, one recognizes that pdf-tools is essentially built of two parts: a server (written in C) responsible for rendering the PDFs, and the Emacs Lisp client inside Emacs that takes the rendered image data from the server and provides them inside Emacs for viewing or manipulation.
The reason the original author (politza) took such a design decision is simply because one cannotFor the curious reader, the author is working on a PDF (and other formats) renderer for Emacs that takes the approach of Dynamic Modules. In short, the goal is to be able to have the rendering part handled by C using a better library than the one pdf-tools uses (MuPDF over Poppler), but to have that C part of the package be callable from Emacs Lisp directly through FFI. One can argue of the deficiencies of FFI, but whatever they might be, they are relatively efficient than having to wait for a server over RPC to render an image for a page and then manipulating it with Emacs Lisp. If the reader differs on this, then do contact the author on their reasoning for it. (without having a tendency for masochistic self-inflicted pain) write a renderer for PDF in Emacs Lisp from scratch. Not only would it be reinventing the wheel, it would be a wheel whose circumference isn’t smooth. Thus the need for using another language, in this case C. More importantly, since politza began the development of pdf-tools two years before we had Dynamic Modules introduced in Emacs, changing the package’s core design would require a total rewrite, and I am grateful he didn’t do that and be on the verge of leaving the work, we wouldn’t have what we do now, a PDF reader in Emacs that works.
This introduction will explain how the API works internally so that one can leverage it to have their multi-language Emacs packages without having to be forced into designing them in a manner that isn’t suitable.
The dynamic modules we write for Emacs are all managed through a particular API, specifically the emacs-module.h header file, it resides either in /usr/include/ directory of your GNU/Linux distribution or if you’ve built your Emacs from a git checkout, then it’ll be in the src/ directory. If it is the former, then gcc will automatically find it, but in the case of the latter, you’d need to explicitly set the directory of the git checkout:Note: Debugging is tricky, but extremely important when working across FFI. For the current purpose, we recommend to also have the -ggdb3 flag for gcc. This produces information that can be then used by GDB, and the level 3 includes extra information such as macro expansions.
gcc -Wall -I/code/src/emacs-git/src -fPIC -shared -o new-module.so new-module.cThis will create a shared object new-module.so which can then be evaluated by Emacs directly using:
(module-load “new-module.so”)Once it has been successfully evaluated without any errors, any procedures or variables that have been defined by new-module.c will now be available in the current Emacs environment. Evaluation of such functions at runtime happens using the shared object, instead of the typical eval of Emacs Lisp.
If you are to work on package module in languages other than C, you have to create bindings from those langauges to the C API in emacs-module.h. This can vary in complexity depending on your language and it’s toolchain, in Rust, for example this is as easy as just using rust-bindgen to provide it the aforementioned header file, blacklist the main function and generate the required bindingsTutorial for a minimal Emacs module in Rust can be found here:
https://ryanfaulhaber.com/posts/first-emacs-module-rust/.
Regardless of which language you choose to work in, you would have to deal with the C API, thus, understanding how it works is crucial.
A well-defined module is to required by the API convention to have the followingOne of the reasons why Emacs Lisp didn’t have a proper way for FFI earlier was the concern Richard Stallman (and others) had that people would abuse this as a way to violate software freedom and create Emacs packages that are non-free. Cf. this response and the overall thread in emacs-devel: A plea for dynamically loadable extension modules:
<emacs-module.h>int plugin_is_GPL_compatibleint emacs_module_init(struct emacs_runtime *runtime) {}emacs_value simple_function(emacs_env *env, ptrdiff_t nargs, emacs_value args[], void *data) {} How the two functions work, and their respective signatures will be explained in the next section, but what one can observe from this structure above is that when designing a dynamic module, the programmer has to remember that whatever operations the custom-defined functions do, they need to be eventually manually set by the initialization function in emacs_module_init.
To elucidate the structure, consider below a simple “Hello, Emacs” program as a dynamic module:
1 int plugin_is_GPL_compatible;
2
3 // Custom function that we want to evaluate from Emacs
5 static emacs_value simple_function(emacs_env *env, ptrdiff_t nargs, emacs_value args[], void *data) {
6 const char *message = “Hello, Emacs!”;
7 return env->make_string(env, message, strlen(message)); // API function make_string operates on the message
8 }
9
10 // Required Initialization Function
11 int emacs_module_init(struct emacs_runtime *runtime) {
12 emacs_env *env = runtime->get_environment(runtime);
13 emacs_value fset = env->intern(env, “fset”);
14 emacs_value func_name = env->intern(env, “simple-function”); // Define the name which the Elisp function will have
15 emacs_value func = env->make_function(env, 0, 0,
16 simple_function, // Declare the Elisp function
17 “Return a simple greeting.”, // The docstring for the Elisp function
18 NULL);
19 emacs_value args[] = {func_name, func};
20 env->funcall(env, fset, 2, args);
21 return 0;
22 }
The above program can be compiled as earlier described, and the produced .so can then be loaded into Emacs. Once it is loaded without any errors, doing C-h o and searching simple-function will show an option for the function that we just defined and its docstring. And when we evaluate the function, we’d get an echo message of “Hello, Emacs”.
This is a very minimal example, but it describes the way the API will structure a module. You first program your functions, make sure their signature and operations are compatible with the API, then you initialize your module and provide the functions you defined. This needs to be done for every function that should be exposed to the Emacs environment; those which you do not wish to expose can simply be used as helper functions without any API wrapper to be used in other (exposed) functions.
For a multi-module structure, you’d have to have the appropriate header files to glue the functions and structures from .c files and then initialize them in emacs_module_init. We now provide elaboration of the specific aspects of the API that’s used in the above program.
While the reader can stop at the previous section, and provided they have a working knowledge of C and time to tinker, they’d be able to design and program their Emacs package; to those who wish to understand before tinkering, it might seem slightly more convoluted. What is emacs_value, and why is it needed for defining a function? What is the emacs_runtime structure, and why do we need a pointer to that? All of these can be answered by understanding how emacs-module.h is structured for use.
To some of these questions, decent answers exist in the Emacs Lisp Reference ManualAvailable as HTML or inside your Emacs at C-h i m Elisp., and there has been an attempt towards a more detailed documentation as wellhttps://phst.eu/emacs-modules. The following explanation takes the best from the two and tries to knit them in more succinct ways for understanding, but nonetheless, when in doubt, the programmer should consult those resources for reference.
The main components of the API are:
emacs_runtimeemacs_envemacs_valueThese components are the building blocks upon which the functionality provided by the API rests, and it is crucial to understand what they relate to, so as use them correctly.
emacs_runtime: As we saw in the trivial program, the emacs_module_init must point to a struct called emacs_runtime. This is the runtime object with which all other components are woven together. It is defined in emacs_module.h as follows:
1 /* Struct passed to a module init function (emacs_module_init).*/
2 struct emacs_runtime
3 {
4 /* Structure size (for version checking).*/
5 ptrdiff_t size;
6
7 /* Private data; users should not touch this.*/
8 struct emacs_runtime_private *private_members;
9
10 /* Return an environment pointer.*/
11 emacs_env *(*get_environment) (struct emacs_runtime *runtime)
12 };
The emacs_runtime_private contains parts of the runtime that are not to be exposed through the API, they contain the initial environment in which Emacs has been initialized, i.e, all your existing packages, custom Emacs Lisp definitions and so on. They are not to be fiddled with.
In the above struct, the crucial component is the third field, wherein we use the function get_environment that returns a pointer to the environment of the current runtime, precluding the private data. This is the initial environment that goes into emacs_module_init, upon which the module-specific functions or variables are added.
The important thing to note here is that emacs_env is a type alias; it aliases the current version of Emacs for which the runtime exists. It is automatically handled during the configuration of the build of Emacs, i.e., by the ./configure script that creates a configuration for the specific build. This can be noted in emacs_module.h, where after macros are defined, we see:
typedef struct emacs_env_31 emacs_env;This aliases emacs_env for the current Emacs build, and hence runtime, to emacs_env_31, which is itself a struct much like emacs_env, but it has additional functions and objects that are specific to this new Emacs version.
What does emacs_env_31 itself look like? Well, it has two necessary fields that uniquely define it, and the rest is inherited from the previous emacs_env_version or added for a specific one:
1 struct emacs_env_31
2 {
3 ptrdiff_t size;
4 struct emacs_env_private *private_members
5
6 /* Functions and definitions for this new Emacs version */
7 }
The type alias for emacs_env is defined during the ./configure script and shouldn’t be changed manually. So, if a programmer wishes to work on a package for an older Emacs or a newer one, they have to include the emacs-module.h of that Emacs version, whether through installation from a system package manager or manually building it from a git checkout. Though, if you have a newer version, it already has everything from the previous version; thus, your modules are usually backwards compatible by default.
Everything defined by the module, whether a function, variable, or struct, has to be an emacs_value if it is to be exposed to the Emacs environment and used from it. It is simply a type definition in C to communicate between the module and Lisp objects from the environment. It is a pointer to an opaque structure that is to be represented as an Emacs Lisp object. The opaqueness here means that no assumptions should be made about the structure that lies behind an emacs_value, or as Philip’s API documentation says:
“Modules mustn’t make any assumptions about the pointer or its structure; in particular, it is unspecified whether
emacs_valuepointers point to a valid memory location, whetherNULLrepresents a valid Emacs Lisp object, or whether identical Emacs Lisp objects are represented by equal pointers or not.”
What this means for the programmer is that, when you define, say, a function as the following:
static emacs_value simple_function(emacs_env *env, ptrdiff_t nargs, emacs_value args[], void *data)Do you recognize how the arguments to this function almost correspond with the fields of the emacs_env struct? It is because the type emacs_value actually unfolds to be a function pointer as a structure field inside of your emacs_env, which is, as we saw, aliased specific to your version:
1 struct emacs_env_31
2 {
3 /* Previous fields */
4 emacs_value args[] = {func_name, func};
5 emacs_value (*simple_function) (emacs_env *env, emacs_value args[], void *data);
6 }
So, every time you define something of the type emacs_value, which you then have to add in the initialization of emacs_module_init, it goes directly into the struct of emacs_env and becomes part of the existing Emacs environment. And as a pointer, it doesn’t necessarily have to be valid in memory; it should be treated as completely opaque.
Additionally, to not make things messy, the API has another type alias that can be used to distinguish between normal emacs_value objects and functions. It is the emacs_function type alias and is to be used exactly as emacs_value, except for defining functions. So we can change our earlier “Hello, Emacs” program where the type of simple_function should be emacs_function.
The above three components are used in defining the core functions that are exposed by the API. These range from basic function calls to vector manipulation, depending upon the version of the Emacs runtime you’re using. The current list would be inherited from emacs_env_31, from the latest master branch. These functions are the bridge between the module’s operations and the Emacs environment.
It is not an exhaustive list, since that’d be too long, and more will be added with each Emacs version. The previously mentioned referential material should be consulted for such a list or to look at the appropriate emacs_env struct itself in emacs-module.h. There are also elaborate comments in src/emacs-module.c if you have a git checkout of Emacs handy—it’s where all things are implemented. The functions below are the essential ones that every programmer developing an Emacs package might need.
is_not_nil: A boolean that checks if the emacs_value (which is the argument) is not nil.eq: Takes as input two emacs_value objects and returns true if they represent the same Lisp object in the emacs_env.type_of: Shows the type of an emacs_value as an emacs_value. Similar to type-of in Emacs Lisp (C-h o type-of).The API provides a set of functions that are responsible for converting between basic C types and Emacs ones:
make_integer: Simply creates an Emacs Lisp integer from a usual C integer. If it can’t be converted,extract_integer: Same as above, but opposite. Returns the value of an integer in an emacs_value as an intmax_t.make_float: Creates an Emacs Lisp float from a C floating-point value.extract_float: Returns a C floating-point from an Emacs one.make_string: Takes as argument the length of the string and its contents as a pointer to an array. Characters in the array should be of cardinality: length of string + 1.
emacs_value make_string (emacs_env *env, const char *contents, ptrdiff_t length);With the correctly defined function, this will result in a string of the same contents and length in the Emacs environment. It is to be noted that the string should be valid UTF-8. Check the API reference for other caveats.
copy_string_contents: It is the same function but for the opposite direction. It takes as input an Emacs LispLike we saw in the trivial example earlier, there are some functions provided by the API that help in defining our simple_function. Namely, those that are used in emacs_module_init:
env->intern: Checks the symbol table in objarray to see if there is an entry for this string.C-h o intern.env->make_function: Registers a new function in the Emacs environment from the C function in the module.
It makes the module’s C functions visible to Emacs. It takes several arguments, but most importantly, an emacs_value,
which should be the type of your custom module’s functions. Its signature is the following:
1 emacs_value (*make_function) (emacs_env *env, // Pointer to environment
2 ptrdiff_t min_arity, // Min. number of arguments to the function
3 ptrdiff_t max_arity, // Max. number of arguments to the function
4 emacs_value custom_function,
5 const char *docstring,
6 void *data // Additional arguments that might go into the function)
Note that make_function doesn’t return a symbol for the function; it only creates an emacs_value that is to go into the emacs_env.
env->funcall: This simply calls any function with the given arguments. Specifically, its signature is:
1 emacs_value funcall (emacs_env *env,
2 emacs_value custom_function, // Function to call
3 ptrdiff_t nargs, // Number of arguments to the function
4 emacs_value args) // Arguments to the function
These last two functions can be used together to define new C functions in Emacs and use Lisp ones. This is what we do in the last five lines of the trivial example:
1 emacs_value fset = env->intern(env, “fset”);
2 emacs_value func_name = env->intern(env, “simple-function”);
3 emacs_value func = env->make_function(env, 0, 0, simple_function, “Return a simple greeting.”, NULL);
4 emacs_value args[] = {func_name, func};
5 env->funcall(env, fset, 2, args);
Here, we first create an emacs_value to use the fset function from Emacs Lisp. One could also use defalias, but that works slightly differently. Then, we have an emacs_value that holds the symbol of the custom function. env->intern inserts it into the objarray symbol table and returns it. We then use env->make_function to register our function, providing the environment, specifying 0 arguments, adding a simple docstring, and setting void *data to NULL since we have no additional data to provide.
Lastly, we have to perform a funcall, but which function should we call? We can’t directly call simple_function, even though it exists in the symbol table and has been registered, because it hasn’t yet been defined in the Emacs environment. How do we define it? We do the equivalent of (fset ‘simple-function ...), i.e., we need to call fset with the symbol and definition of our simple_function. To do this, we create an array args[] to pass as arguments to funcall, containing the symbol func_name and its definition func. Finally, we call funcall on fset, which finalizes this function in Emacs, allowing it to be used like any other Emacs Lisp native function.
Finally, there are a few functions from the API to create and manage Emacs Lisp vectors:
vec_get takes an Emacs Lisp vector as emacs_value and an index as ptrdiff_t and returns the element at that index as an emacs_value. If the provided value is not a vector or the index is out of bounds, it throws an error.
emacs_value vec_get (emacs_env *env, emacs_value vec, ptrdiff_t index);vec_set takes an Emacs Lisp vector and an index, setting the element at that index to a given emacs_value.
emacs_value vec_set (emacs_env *env, emacs_value vec, ptrdiff_t index, emacs_value value);vec_size returns the number of elements in an Emacs Lisp vector.
ptrdiff_t vec_size (emacs_env *env, emacs_value vec);The above functions together help the programmer in doing some basic operation sharing and exchanging between C and Emacs Lisp. It is minimal, but still enough.
As can be seen above, the API is neither trivial nor exactly smooth to work with, there are a lot of caveats, and as with anything at the level of C, there’s more than what meets the eye. Regardless the attempt of the author was to provide enough information that would encourage and help an Emacs programmer in being more confident towards choosing the path of Dynamic Modules. Needless to say, the API itself doesn’t encourage just replacing existing Emacs Lisp programs in C but rather provide the fundamentals for complimenting Emacs Lisp packages, and help them in ways that are more preferrable. Of course, having a Dynamic Module won’t solve issues specific to Emacs such as lack of real multi-threading, everything from the Dynamic Modules would still be either part of the main Emacs thread or a child of it. But it can at least share some of the burden in processing when developing extensive packages. The author hopes that over time this becomes more prevalent in practice across the Emacs community, and its advantages are understood and taken leverage of.