32.2. External Modules

Platform Dependent: UNIX, Win32 platforms only.

32.2.1. Overview
32.2.2. Module initialization
32.2.3. Module finalization
32.2.4. Function EXT:MODULE-INFO
32.2.5. Dynamic module loading
32.2.6. Example
32.2.7. Module tools
32.2.7.1. Modprep
32.2.7.2. clisp.h
32.2.7.3. Exporting
32.2.8. Trade-offs: FFI vs. C modules
32.2.9. Modules included in the source distribution
32.2.9.1. Base Modules
32.2.9.2. Database, Directory et al
32.2.9.3. Mathematics, Data Mining et al
32.2.9.4. Matching, File Processing et al
32.2.9.5. Communication, Networking
32.2.9.6. Graphics
32.2.9.7. Bindings
32.2.9.8. Toys and Games
32.2.9.9. Miscellaneous

List of Examples

32.1. Create a module set with GNU libc bindings

Modules on Win32

Everything described in the section will work verbatim on Win32 when using Cygwin or MinGW, except for one thing - you will need to replace the run extension in lisp.run with the Win32 executable extension exe.

For historical reasons, all examples appear to assume UNIX and use the run file type (extension) for the CLISP runtime. This does not mean that they will not work on Win32.

32.2.1. Overview

External modules are a mechanism to add extensions (written in C, for example) to CLISP. Extending CLISP using an external module requires creating a module set and adding it to an existing linking set using clisp-link to prodice a new linking set which contains the extension.

Module Set

A module is a piece of code (C or Lisp) which defines extra (non-core) Lisp objects, symbols and functions. Together with link.sh, which describes how to add the module to an existing CLISP, it comprises a module set.

More formally, module set is a directory containing:

link.sh
some /bin/sh commands, which prepare the directory for linking, and set some environment variables, see the section called “Module set variables”
all other files that define the module functionality
needed by link.sh

In link.sh the module set directory is referred to as $modulename/.

A module name must consist of the characters A-Z, a-z, _, 0-9.

The module name clisp is reserved.

Linking Set

A linking set is a collection of files (runtime, memory image &c) which allows performing two major tasks:

  1. Running CLISP: to run a CLISP contained in some linking set directory, call

    $ directory/lisp.run -M directory/lispinit.mem

    or

    $ clisp -K directory

    (recommended, since it also passes -B to the runtime).

  2. Adding a module set to create a new linking set which will contain the module functionality.

The CLISP build directory contains three linking sets in directories boot, base, and full, and a CLISP installation normally contains two linking sets: base, and full

More formally, a linking set is a directory containing at least these files:

lisp.run
the executable runtime
lispinit.mem
the memory image
modules.h
the list of modules contained in this linking set
modules.o
the compiled list of modules contained in this linking set
makevars

some /bin/sh commands, setting the variables

CC

the C compiler

CPPFLAGS

flags for the C compiler, when preprocessing or compiling

CFLAGS

flags for the C compiler, when compiling or linking

CLFLAGS

flags for the C compiler, when linking

LIBS

libraries to use when linking (either present in the linking set directory, or system-wide)

X_LIBS

additional X Window System libraries to use

RANLIB

the ranlib command

FILES

the list of files needed when linking
all the FILES
listed in makevars

Manipulating module sets and linking sets

Use clisp-link to create and install module sets and add them to linking sets.

See also Section 32.2.6, “Example”.

Module set variables

The following variables should be defined in link.sh.

NEW_FILES
the space-separated list of object files that belong to the module set and will belong to every new linking set linked with this module set.
NEW_LIBS
the space-separated list of object files, libraries or C compiler switches that need to be passed to the C compiler when linking the lisp.run belonging to a new linking set.
NEW_MODULES
the space-separated list of the module names belonging to the module set. Normally, every #P".c" file in the module set defines a module of its own. The module name is usually derived from the file name.
TO_LOAD
the space-separated list of Lisp files to load before building the lispinit.mem belonging to a new linking set.
TO_PRELOAD (optional)

the space-separated list of Lisp files to load into an intermediate lispinit.mem file, before building the lispinit.mem belonging to a new linking set. This variable is usually used to create (or unlock) the Lisp PACKAGEs which must be present when the new #P".c" files are initialized. E.g., the FFI:DEF-CALL-IN functions must reside in already defined packages; see Example 32.7, “Calling Lisp from C. You can find a live example in modules/syscalls/preload.lisp and modules/syscalls/link.sh.in.

Warning

If you are unlocking a package, you must also DELETE it from CUSTOM:*SYSTEM-PACKAGE-LIST* (see Section 31.2, “Saving an Image”) here and re-add it to CUSTOM:*SYSTEM-PACKAGE-LIST* in one of the TO_LOAD files. See, e.g., modules/i18n/preload.lisp and modules/i18n/link.sh.in.

32.2.2. Module initialization

Each module has two initialization functions:

void module__name__init_function_1 (struct module_t* module)

called only once when CLISP discovers while loading a memory image that there is a module present in the executable (lisp.run) which was not present at the time the image was saved. It can be used to create Lisp objects, e.g. functions or keywords, and is indeed used for that purpose by modprep.

You do not have to define this function yourself; modprep and FFI will do that for you.

If you use FFI, (FFI:C-LINES :init-once ...) will add code to this function.

Warning

The PACKAGEs must already exist and be unlocked, cf. TO_PRELOAD.

Warning

If you are using modprep and defining your own init-once function, it must call the module__name__init_function_1__modprep function!

void module__name__init_function_2 (struct module_t* module)

called every time CLISP starts. It can be used to bind names to foreign addresses, since the address will be different in each invocation of CLISP, and is indeed used for that purpose by FFI (e.g., by FFI:DEF-CALL-OUT). It can also be used to set parameters of the libraries to which the module interfaces, e.g., the pcre module sets pcre_malloc and pcre_free.

You do not have to define this function yourself; modprep and FFI will do that for you.

If you use FFI, (FFI:C-LINES :init-always ...) will add code to this function.

name is the module name.

See also Section 31.1, “Customizing CLISP Process Initialization and Termination”.

32.2.3. Module finalization

Each module has a finalization function

void module__name__fini_function (struct module_t* module)

called before exiting CLISP.

You do not have to define this function yourself; modprep and FFI will do that for you.

If you use FFI, (FFI:C-LINES :fini ...) will add code to this function.

name is the module name.

See also Section 31.1, “Customizing CLISP Process Initialization and Termination”.

32.2.4. Function EXT:MODULE-INFO

Function (EXT:MODULE-INFO &OPTIONAL name verbose) allows one to inquire about what modules are available in the currently running image. When called without arguments, it returns the list of module names, starting with clisp. When name is supplied and names a module, 3 values are returned - name, subr-count, object-count. When verbose is non-NIL, the full list of module lisp function names written in C (Subrs) and the full list of internal lisp objects available in C code are additionally returned for the total of 5 values.

When name is :FFI, returns the list of shared libraries opened using :LIBRARY. When verbose is non-NIL, return the association list of DLL names and all foreign objects associated with it.

32.2.5. Dynamic module loading

Platform Dependent: Only in CLISP built without configure flag --without-dynamic-modules.

Note

Dynamic loading does not work on all operating systems (dlopen or equivalent is required).

Note

You will probably never need to call the function SYS::DYNLOAD-MODULES explicitly (this is why it is in SYSTEM and not in EXT). You should install your module with

$ clisp-link install name

and load it with (REQUIRE name).

Function (SYS::DYNLOAD-MODULES filename ({name}+)) loads a shared object file or library containing a number of named external CLISP modules.

Note

This facility cannot be used to access arbitrary shared libraries. To do that, use the :LIBRARY argument to FFI:DEF-CALL-OUT and FFI:DEF-C-VAR instead.

External modules for CLISP are shared objects (dynamic libraries) that contain the module__name__subr_tab variable, among others. This serves to register external functions which operate on Lisp-level structures with CLISP.

To use dlopen with modules, you should add -fPIC to the module's compilation options. Something like

$ cc -shared -o name.so name.o

may be needed to produce the shared object file.

32.2.6. Example

Example 32.1. Create a module set with GNU libc bindings

To link in the FFI bindings for the GNU/Linux operating system, the following steps are needed. (Step 1 and step 2 need not be executed in this order.)

  1. Create a new module set

    $ clisp-link create linux /pathname/bindings/linux.c
  2. Modify the newly created linux/link.sh

    1. add -lm to the libraries

      replace

      NEW_LIBS="$file_list"

      with

      NEW_LIBS="$file_list -lm"
    2. load linux.fas before saving the memory image

      replace

      TO_LOAD=''

      with

      TO_LOAD='/pathname/bindings/linux.fas'
  3. Compile linux.lisp, creating linux.c

    $ clisp -c /pathname/bindings/linux.lisp
  4. Create a new linking set

    $ clisp-link add base base+linux linux
  5. Run and try it

    $ base+linux/lisp.run -M base+linux/lispinit.mem -x '(linux:stat "/tmp")'

32.2.7. Module tools

There are some tools to facilitate easy module writing.

32.2.7.1. Modprep

If your module is written in C, you can pre-process your sources with modprep in the CLISP distribution and define lisp functions with the DEFUN macro:

DEFUN(MY-PACKAGE:MY-FUNCTION-NAME, arg1 arg2 &KEY FOO BAR) {
  if (!boundp(STACK_0)) STACK_0 = fixnum(0); /* BAR */
  if (!boundp(STACK_1)) STACK_1 = fixnum(1); /* FOO */
  pushSTACK(`MY-PACKAGE::SOME-SYMBOL`); /* create a symbol in the package */
  pushSTACK(`#(:THIS :IS :A :VECTOR)`); /* some vector, created once */
  pushSTACK(``MY-PACKAGE::MY-FUNCTION-NAME``); /* double `` means FUNCTION */
  VALUES1(listof(7)); /* cons up a new list and clean up the STACK */
}

Then (MY-PACKAGE:MY-FUNCTION-NAME 'A 12 :FOO T) will return (A 12 T 0 MY-PACKAGE::SOME-SYMBOL #(:THIS :IS :A :VECTOR) #<ADD-ON-SYSTEM-FUNCTION MY-PACKAGE:MY-FUNCTION-NAME>) (assuming you EXPORTed MY-FUNCTION-NAME from MY-PACKAGE).

Note that the arguments are passed on the STACK (last argument being the top) which has to be cleaned up before exit.

Another useful macros are:

DEFVAR
create a GC-visible private object
DEFFLAGSET
define a C function which will remove several flag arguments from the STACK and return the combined flag value
DEFCHECKER
define a map from cpp constants to lisp symbols and functions that map between them, checking that the argument is appropriate

See modules/syscalls/calls.c and other included modules for more examples and file modprep for full documentation.

Warning

If you manipulate Lisp objects, you need to watch out for GC-safety.

32.2.7.2. clisp.h

If your module is written in C, you will probably want to #include "clisp.h" to access CLISP objects. You will certainly need to read "clisp.h" and some code in included modules, but here are some important hints that you will need to keep in mind:

32.2.7.3. Exporting

If your module uses FFI to interface to a C library, you might want to make your module package :CASE-SENSITIVE and use exporting.lisp in the CLISP distribution to make FFI forms and DEFUN, DEFMACRO at al export the symbols they define. See modules/netica/, modules/matlab/ and modules/bindings/ for examples.

32.2.8. Trade-offs: FFI vs. C modules

When deciding how to write a module: whether to use FFI or to stick with C and modprep, one has to take into account several issues:

Speed: C wins

FFI has a noticeable overhead: compare RAWSOCK:HTONS (defined in modules/rawsock/rawsock.c) with

(FFI:DEF-CALL-OUT htons (:NAME "htons") (:LIBRARY :default)
  (:ARGUMENTS (s ffi:short)) (:RETURN-TYPE ffi:short) (:LANGUAGE :stdc))

and observe that RAWSOCK:HTONS is almost 3 times as fast (this really does compare the FFI overhead to the normal lisp function call because htons is computationally trivial). This difference will matter only if you call a simple function very many times, in which case it would make sense to put the loop itself into C.

Portability: C wins
  1. FFI is not as widely ported as CLISP, so it is possible that you will face a platform where CLISP runs but FFI is not present.
  2. It is much easier to handle portability in C: observe the alternative implementations of htonl et al in modules/rawsock/rawsock.c.
  3. Certain C structures have different layout on different platforms, and functions may take 64-bit arguments on some platforms and 32-bit arguments on others; so the FFI code has to track those differences, while C will mostly take care of these things for you.
Code size: FFI wins
You need to type much fewer characters with FFI, and, if you use the :LIBRARY argument to FFI:DEF-CALL-OUT and FFI:DEF-C-VAR, you do not need to leave your CLISP session to try out your code. This is a huge advantage for rapid prototyping.
UI: C wins
To produce a nice lispy UI (using &OPTIONAL and &KEYword arguments etc), you will need to write wrappers to your FFI:FOREIGN-FUNCTIONs, while in C you can do that directly. The same goes for polymorphism: accepting different argument types (like, e.g., POSIX:RESOLVE-HOST-IPADDR does) would require a lisp wrapper for FFI:FOREIGN-FUNCTIONs.
Learning curve: unclear

If you are comfortable with C, you might find the CLISP C module facilities (e.g., modprep) very easy to use.

CLISP FFI, on the other hand, is quite high-level, so, if you are more comfortable with high-level languages, you might find it easier to write FFI forms than C code.

Safety: unclear
One can get a segfault either way: if your FFI:DEF-CALL-OUT form does not describe the C function's expectations with respect to the arguments and return values (including ALLOCATION), you will probably learn that the hard way. If the module is written in C, all the opportunities to shoot oneself in the foot (and other body parts) are wide open (although well known to most C users). However, with C, one has to watch for GC-safety too.
System requirements: FFI wins

Some Linux distributions offer separate library (containing shared libraries only) and development (containing static libraries and C headers) packages.

  • When developing using FFI with :LIBRARY (and avoiding FFI:DEF-C-CONST), the development package is not needed.
  • When developing with modprep, the development package is necessary.
  • When distributing module (built with either FFI or modprep), only the library package is needed.

Note

The granularity of the choice is per function: the same module can use both modprep and FFI.

Note

It is not a good idea to have both foo.lisp and foo.c files in a module, because if you ever add an FFI form to the former, COMPILE-FILE will overwrite the latter.

32.2.9. Modules included in the source distribution

A few modules come with the source distribution of CLISP (but are not necessarily built in a particular binary distribution).

To use modules, read unix/INSTALL and build CLISP in a directory build-dir with, e.g.,

$ ./configure --with-module=pcre --with-module=clx/new-clx --cbc build-dir

then run it with

$ ./build-dir/clisp -K full

This will create a base linking set with modules i18n, regexp and syscalls (and maybe readline); and a full linking set with modules clx/new-clx and pcre in addition to the 3 (or 4) base modules.

Here we list the included modules by their general theme. See Chapter 33, Extensions Implemented as Modules for individual module documentation.

32.2.9.1. Base Modules

The default build process includes the following modules in both base and full linking sets:

i18n
Internationalization of User Programs.
regexp
The POSIX Regular Expressions matching, compiling, executing.
syscalls
Use some system calls in a platform-independent way.
readline (only when both GNU readline and FFI are available)
Some advanced readline and history features are exported using this module.

The composition of the full linking set depends on the platform and on the vendor preferences.

32.2.9.2. Database, Directory et al

gdbm
Interface to GNU DataBase Manager by Masayuki Onjo.
berkeley-db
Berkeley DB interface.
dirkey
Directory Access (LDAP, Win32 registry etc).
postgresql
Access PostgreSQL from CLISP.
oracle
Access Oracle RDBMS from CLISP; by John Hinsdale.

32.2.9.3. Mathematics, Data Mining et al

libsvm
Build Support Vector Machine models using LibSVM inside CLISP.
pari
Interface to the computer algebra system PARI.
matlab
Do matrix computations via MATLAB.
netica
Work with Bayesian belief networks and influence diagrams using Netica C API.

32.2.9.4. Matching, File Processing et al

pcre
The Perl Compatible Regular Expressions matching, compiling, executing.
zlib
Compress VECTORs using ZLIB.

32.2.9.5. Communication, Networking

rawsock
Raw socket access.
dbus
Interface to D-Bus.
fastcgi
Access FastCGI from CLISP; by John Hinsdale.

32.2.9.6. Graphics

CLX

Call Xlib functions from CLISP. Two implementations are supplied:

clx/mit-clx, from MIT https://www.x.org/archive/unsupported/lib/CLX/
the standard implementation
clx/new-clx, by Gilbert Baumann

faster, with additional features, but not quite complete yet. Please try it first and use clx/mit-clx only if clx/new-clx does not work for you. clx/new-clx comes with several demos, please try them using

$ clisp -K full -i modules/clx/new-clx/demos/clx-demos.lisp -x '(clx-demos:run-all-demos)'

and follow the intructions.

This functionality is documented in the manual https://common-lisp.net/project/cmucl/doc/clx/, also available in the CLISP source distribution as modules/clx/clx-manual.tar.gz.

gtk2
Use GTK+ and Glade to create GUI by James Bailey.

32.2.9.7. Bindings

Call the operating system functions from CLISP. The following platforms are supported:

32.2.9.8. Toys and Games

queens
Compute the number of solutions to the n-queens problem on a n×n chessboard (a toy example for the users to explore the CLISP module system).
modules/clx/new-clx/demos/sokoban.lisp
a demo which comes with clx/new-clx.

32.2.9.9. Miscellaneous

asdf
A system definition facility.

These notes document CLISP version 2.49.93+Last modified: 2018-02-19