Web development in single threaded lisp with slime and hunchentoot

It is problematic at first sight to use SLIME for web development in a single threaded lisp with swank:*communication-style* = nil (as it happens with CLISP and SBCL on Windows). The problem is the impossibility to modify application while it is running.

In very informal pseudo code this configuration of swank may be simplified represented as:

(loop
   (read-slime-request)
   (exec-slime-request))

When our next SLIME request starts hunchentoot, hunchentoot enters its own request handling loop (for HTTP requests):

(loop
   (read-slime-request)
   (exec-slime-request :
      (loop
         (read-http-request)
         (handle-http-request)))

This means exec-slime-request will not exit and subsequent slime requests will not be handled until the web server is stopped. SLIME isn't functioning.

The solution is to use web server to invoke SLIME. When in the emacs we are going to talk to swank, send special HTTP request and its handler will enforce swank to handle emacs request.

(loop
   (read-slime-request)
   (exec-slime-request :
      (loop
         (read-http-request)
         (handle-http-request :
            (read-slime-request)
            (exec-slime-request))))

It requires only few lines of code in emacs lisp and in common lisp. The code is here, it's simple, see it for more details.

One note is needed to help fully understand the code. When emacs side sends a request to swank, swank is not always blocked by hunchentoot. As far as I understand it, one slime request may lead to a sequence of communications between swank and emacs.

01: (loop
02:   (read-slime-request)
03:   (exec-slime-request :
04:      (loop
05:         (read-http-request)
06:         (handle-http-request :
07:            (read-slime-request)
08:            (exec-slime-request :
09:               ...
10:               (read-slime-request)
11:               ...))))

In the emacs we should not make an HTTP request before sending the slime request that is to be read on the line 10.

To distinguish this situation on the emacs side I check whether the socket of HTTP request read by swank on the line 05 is still connected. In this case new HTTP request isn't performed.

This solution isn't 100% correct (the socket may be still not closed, but swank isn't going to read anything). But it works almost good. Sometimes a glitch appears - when you execute an expression, you get result of previous expression instead of the just executed. If you execute expressions en, en+1, en+2, you get results of executing expressions en-1, en, en+1. This "phase displacement" is most likely caused by the fact that we not performed HTTP request when it was necessary and the slime request remained not read by swank. When next time we do HTTP request for the next slime request, the previous unread request is read and executed by swank.

In principle, I know how to fix this problem (we may annotate HTTP requests and slime requests with some IDs and do synchronization - ignore unnecessary HTTP requests - on the common lisp side; socket checking on the emacs side will became unnecessary). But I do not want to spend time on it, it is easier to enforce reading enqueued expressions by the "special HTTP request" directly from browser: fetch http://localhost:4343/slime-http-send-handler needed count of times.

How to use it.

Very easy.

Add the following line into your .emacs file:

(load "c:/some/path/slime-over-http.el")

The line should follow the normal slime setup.

Here is an example of common lisp web site:

;; this line may vary in you case; just make
;; sure that asdf is loaded and all the needed
;; libraries are available
(load "c:/usr/unpacked/lisp-libs/setup-asdf-registry.lisp")

(asdf:operate 'asdf:load-op :hunchentoot)
(use-package :hunchentoot)

(defun want-slime-over-http-p ()
  "This predicate indicates whether we
are in the development environment and have no
threading"
  #+:win32 t)

(when (want-slime-over-http-p)
  (load "C:/usr/projects/slime-over-http/slime-over-http.lisp")
  (defun hunchentoot::keep-alive-p ()))

(setq *dispatch-table*
      `(dispatch-easy-handlers
        ,@(when (want-slime-over-http-p)
                (list (create-prefix-dispatcher
                           "/slime-http-send-handler"
                           (find-symbol "SLIME-HTTP-SEND-HANDLER"
                                        "SLIME-HTTP"))))
        default-dispatcher))

(define-easy-handler (hello :uri "/hello")
    ()
  "Hello, change me!")

After executing the above code run emacs command slime-start-hunchentoot. Now you may open http://localhost:4343/hello in the browser and see the phrase "Hello, change me!".

You may change it because you have fully functioning slime with repl, debugger, etc.

To stop the server execute common lisp function (slime-http::stop-instance).

Note how we redefined hunchentoot::keep-alive-p to return nil. Single threaded server can't provide keep-alive, because while it is handling alive connection it can't accept any other connections (in particular from slime; also, IE is often unable to fully open a page with several images or frames in this case; looks like it opens several connections and schedules downloading different parts of the page to different connections; connections other than "kept alive" are not handled by hunchentoot).

In addition to redefining keep-alive-p I also use reverse proxying apache for sure.

Resume

So we see quite simple way (despite the long explanation) to do web development with hunchentoot in free lisps available on windows.

It isn't the only way to develop web applications with common lisp if you are on windows. For example you may install linux virtual machine. Edi Weitz - the hunchentoot author - uses this approach.

PS

My code is called "slime-over-http" because initially I delivered slime requests over the TCP connection used by the HTTP request and only some time later I realized that it is unnecessary. I'd like another name (preferably short), but can't contrive it.

If you have any questions about this configuration - ask, I'll be glad to help.