Example of Using External Libraries or Packages in Common Lisp

How to use libraries in Common Lisp?

This is an instance of a common problem people have with CL.

The way CL (and other Lisps) work is in three phases (in fact more than three, but three is enough):

a sequence of characters is read to turn it into a form;
various magic happens to that form;
the result of stage 2 is evaluated, and perhaps the results printed.

Typically this process is iterated to process, say, a file, or a stream of input.

The important thing is that (1), (2), and (3) happen in sequence: (1) is complete before (2) begins, and both (1) and (2) are complete before (3) begins.

What that means is that (1) must be possible before anything that happens in (2) or (3) have happened.

So consider this form:

(progn 
  (ql:quickload "cl-ppcre")
  (cl-ppcre:split "\s+" "1 2 3"))

(This is pretty much one of the forms you are trying to evaluate.)

So the question is: what does it take for (1) to happen? Well, it takes two things:

the QL package (which is essentially a namespace: see below for more on what 'package' means in CL) must exist to read ql:quickload;
the CL-PPCRE package must exist to read cl-ppcre:split.

And now you see the problem: (ql:quickload "cl-ppcre") creates the CL-PPCRE package, and this does not take place until (3). That means that this form can't be read.

You can get around this, in desperation, with various heroic tricks. However you don't actually need to: you can do something else, which (almost: see below) works:

(ql:quickload "cl-ppcre")
(cl-ppcre:split "\s+" "1 2 3")

And this is (almost) fine, because it's not one form: it's two. So (1)-(3) work fine for the first form, and then (1)-(3) work fine for the second form.

So the answer is not to try and bundle everything into a single form. Either put things in a file (probably the best way) or if you really want to run things as command line arguments, you need to arrange that the forms are separate, with, for instance, multiple --eval options.

Above I said that the second, multiple-form, version almost works. And it does only almost work. The reason for this is file compilation. Let's assume I have a file whose contents are:

(ql:quickload "cl-ppcre")
(cl-ppcre:scan ...)
...

Well, what does the compiler do when it compiles that file? It reads it, form by form, but in general it doesn't actually execute code (there are exceptions): it arranges for that code to be executed when the file is loaded. So the compiler will not load CL-PPCRE: it will arrange life so that when the file is loaded CL-PPCRE will be loaded. And now we have a version of the same problem: the second form can't be read by the compiler because the CL-PPCRE package does not yet exist.

Well, there is a solution to that: you can tell the compiler that it must, in fact, execute some code at compile-time:

(eval-when (:compile-toplevel :load-toplevel :execute)
  (ql:quickload "cl-ppcre"))
(cl-ppcre:scan ...)
...

And now the compiler knows, thanks to the eval-when form, that it must call ql:quickload. And it will do so, and so the CL-PPCRE package will be defined at compile time, and all will be well.

A note on packages in CL

The term 'package' in CL has a meaning which is unfortunately not the same as it is in many other languages: that's because of history and can't now be changed.

In common usage, a package is 'some chunk of code which you can perhaps install and which perhaps has dependencies on other packages (chunks of code), all of which might be looked after by a package manager of some kind. You might install Python as a package on your Ubuntu box, or you might use the conda package manager to manage scientific Python packages (including Python itself).

In CL a package is essentially a namespace. If I type 'foo:bar' then this is referring to the symbol named BAR available in the package one of whose names or nicknames is FOO. Further this is an 'external' symbol, which means it's intended to be public in some way. Packages are real objects in CL and can be reasoned about by programs. There is always a notion of a current package, and that package defines what names are available without requiring package prefixes both by directly containing some names and also having a search ('use') list of other packages. There is a lot to packages in CL: far more than I can mention here.

What commonly might be referred to as packages are probably best referred to as 'libraries' in CL: they're chunks of code which you can install and which may have dependencies in their own right. Alternatively they are often referred to as 'systems' as they are often defined using a 'system definition' tool. 'Library' is probably the better term: 'system' is again a historical oddity which can't really be changed now.

External vs Internal Symbols in Common Lisp Package

The author of a Common Lisp package can export a symbol for the user of the package. Then the symbol is an external symbol and you can access it with package-name:external-symbol-name.

Internal symbols are not meant for the user but can be accessed with package-name::symbol-name

More explanations are in Peter Seibel's book and Common Lisp the Language

Common Lisp style: multiple packages in same repo

You could consider using asdf-inferred-package. With that, you could have a mmap/high package that depends on a mmap/low package. With that setup, you can actually ask Quicklisp to load either of them directly:

(ql:quickload "mmap/high")

(ql:quickload "mmap/low")

You can see an example in my cl-bulk repo.

Common Lisp: What's the best way to use libraries in a shared hosting environment?

There are two ways I would look at it:

start a Lisp for each request
This way it would be much better that the Lisp is a saved image with all necessary libraries and data loaded. But that approach does not look very promising to me.
run a Lisp and let a frontend (web browser, another web server, ...) connect to it
This way you can either start a saved image or a Lisp that loads a bunch of stuff once and serves the requests.

I like to use saved images/applications in a deployment scenario. They can be quickly started, contain all the necessary software and are independent of library changes.

So it might be useful to provide pre-configured Lisp images that contain the necessary software or let the user configure and save an image.

How to find all available systems?

All systems registered in ASDF:

(asdf:registered-systems)

I found that one by typing asdf:systems and letting auto-completion suggests a name. The symbol is exported, so it is fair game. Apparently it is undocumented.

Quicklisp has a notion of distributions, dists.

(ql-dist:all-dists)

Each dist has different versions (http://blog.quicklisp.org/2011/08/going-back-in-dist-time.html):

(ql-dist:available-versions (ql-dist:dist "quicklisp"))

Each dist provides systems:

(ql-dist:provided-systems (ql-dist:dist "quicklisp"))

Each system has a release, you can list all releases:

(ql-dist:provided-releases (ql-dist:dist "quicklisp"))

Conforming implementation have a list of *MODULES*, which is useful notably for systems that are available as built-ins by your implementation; for SBCL:

CL-USER> (require 'sb-mpfr)
("SB-MPFR" "SB-GMP")

CL-USER> *modules*
("SB-GMP" "SB-MPFR" ...)

Finding Common Lisp Libraries

There are two separate questions:

Are there any resources you use to check which libraries meet your
needs?

I just use the library's official documentation, or quickdocs as mentioned earlier. I don't think there are any comparison tables between similar libraries. But you can always ask for help on #lisp on Freenode IRC network (since StackOverflow doesn't like questions like "What's the best CL library for parsing CSV?")

How do you determine if a library is still being supported? Are there any heuristics short of visiting individual libraries's websites?

If a library is on Quicklisp then it is supported. Unsupported libraries usually drop out soon enough. Xach (Quicklisp's developer and maintainer) makes sure that there is no library in Quicklisp which can't be built on a supported CL implementation.

How is Lisp code structured? What are Packages and Systems?

A package is a namespace for symbols. One can import and export symbols from and to symbols. A symbol maybe interned in one package. Packages can be used by other packages.

A program is structured into systems. A system is a collection of files, dependencies, build rules, default information and more - depebding on the system tool used. ASDF is one of those. Loading libraries is then done by loading systems. Systems can also be compiled, printed, ...

Packages and systems are independent of each other and not related.

It makes sense for each larger piece of software, library or program, to use one or more custom packages. This avoids name clashes with symbols from other loaded software.