Notes on Deploying Common Lisp Web Services

I’m recently finishing a Common Lisp web service used inside a company. An essential part of the reason why I choose Lisp in this case is, CLOS, one of the most powerful tools for designing complicated concepts. It’s a nice experience using CLOS to successfully solve some hard problems along with extending some components.

However, the embarrassing thing is I’ve never delivered a web application before, all I know is developing, debugging within SLIME and writing some scripts to deliver a binary executable so far. Web applications, however, are long-running services (or better said, “processes”). Once the services are started, you’d wish there are some tools to stop or restart them, basically. And if the service crashes somehow or the machine just has restarted due to some maintenance, it would be wonderful that the service itself could restart automatically. In this article, I therefore try to introduce two solutions that may just work for whoever is or will be searching for similar questions on the Internet.

1. 1. TMUX

tmux¹ is a software application that can be used to multiplex several virtual consoles, allowing a user to access multiple separate terminal sessions inside a single terminal window or remote terminal session. Before I got used to Emacs, I used tmux way much more to manage sessions on Linux. After all, Emacs is just an editor, no matter how much I love it. By using tmux, one can attach/detach sessions in an absolutely sane way. So, here are some recipes.

1.1. 1.1 for creating a new session

tmux new -s session-name

1.2. 1.2 for attaching a session

tmux a -t session-name

1.3. 1.3 for detaching from a session

tmux detach # or use the shortcut 'Ctrl-b d'

1.4. 1.4 for killing a session

tmux kill-session -t session-name

1.5. 1.5 for sending keys to a session

tmux send-keys -t session-name "(+ 1 2 3)" C-m

One might be curious what does send-keys subcommand do exactly. In the above example, first imagine we already have a session where SBCL got started, tmux then sent an expression (+ 1 2 3) to Lisp followed by a new line which is indicated by C-m:

Now you can see, by using send-keys, one can write shell scripts to start, stop, reload (fasl files), restart lisp services or just send any expressions to evaluate without putting too much efforts into it. For more information about tmux’s commands, please check its manual.

2. 2. Supervisor

tmux is both powerful and flexible, however, it can not tell us (through, for example, logging) if the server crashed or even restarted itself automatically after an unexpected crashed. We thus need some sort of processes management tool.

Supervisor is a client/server system that allows its users to monitor and control a number of processes on UNIX-like operating systems². This is a piece of Python software, and it is for Python 2.x only. I’m definitely not the first one who came up with using this tool, as a matter of fact, way back at 2013, Fukamachi wrote an essay talking about how he managed to deploy Quickdocs³ web services. In that paper, Supervisor along with Nginx is proposed, and Clack, a piece of software written by himself was used as well.

Fukamachi didn’t write the essay in English, unfortunately, but it’s not hard to understand what he proposed. Basically, one can prepare a Makefile like this to start the service:

SERVER_PORT=8000
SWANK_PORT=4005

define sbcl
	sbcl --noinform --disable-debugger \
		--load /home/david/quicklisp/setup.lisp \
		--eval '(progn $1)' \
		--eval '(progn $2)'
endef

start:
	$(call sbcl, \
		(ql:quickload :lucerne-hello-world) (ql:quickload :swank), \
		(lucerne:start lucerne-hello-world:app :port $(SERVER_PORT)) \
		(swank:create-server :port $(SWANK_PORT) :style :spawn :dont-close t))

Then define your service in supervisord.conf:

[program:lucerne]
command=make -f /paht-to-your-Makefile/Makefile start
directory=/your-project-path
numprocs=1
autostart=true
autorestart=true
user=david
redirect_stderr=true
stdout_logfile=/var/log/supervisor/lucerne-hello-world.log

For more information about how to use Supervisor, please check its documentation which is very comprehensive and easy to understand. The last thing I want to discuss is, within the above Makefile example, one may notice that we start a swank server besides the Lisp web service. Supervisor can indeed monitor services, but what about if we want to compile lisp code even when the service is running? That is, we don’t want to issue sudo service supervisor restart but just re-compile and load pieces of fasl files into the original Lisp image and keep it running. Within Emacs, one can manage to achieve that goal by issuing M-x slime-connect 127.0.0.1 4005⁴ and then start to re-compile (C-c C-c) definitions or just inspect and debug, dynamically. In the end, one can even connect to a Lisp image which is deployed at a remote machine. Please check the documentation at this page.

Footnotes:

Wikipedia, tmux, https://en.wikipedia.org/wiki/Tmux

Supervisor: A Process Control System, http://supervisord.org

Quickdocs, Library Documentation Hosting for Common Lisp, http://quickdocs.org/

⁴

If your specified the port 4005 for swank, of course.