Simpler Clojure/SLIME Embedding

Thanks to Craig McDaniel's hint, I know a simpler way to embed Clojure/SLIME than I used before. Since Clojure svn revision 1127, macro clojure.main/with-bindings makes the embedding as trivial as this jsp page.

Clojure at Work: Interactively Control Running Java System

We are developing a system on Java. It must call another system and at some point the other system calls our system back via web service. Therefore I can not test this interaction in a stand-alone test, our code must be executed in an HTTP container (we use Tomcat).

In case if the code does not work as desired, I need to stop Tomcat, fix and rebuild Java sources and try again. And what is worse, before the code I am working on can be tested, it performs initialization that takes about a minute. These stumbles are little bit annoying for those who are used to fully dynamic development environments as Lisp.

I decided to embed Clojure into the system during development and connect to it with SLIME to create interactive testing/development environment.

First, I followed Bill Clementson's instructions for usual (not embedded) Clojure/SLIME setup.

After it worked, I started to look for a way to embed Clojure/swank into existing Java system, but didn't find any conventional way to do this. Therefore I torn out a piece of code from the clojure.lang.Repl class and instead of standard input, feed it with commands to start swank (which I found in the *inferior-lisp* buffer when Clojure/SLIME is started "normally", from Emacs).

All this is packed into simple clojure.jsp page in my web application directory. And clojure.jar is placed into WEB-INF/lib.

When the JSP is opened in browser it displays: "swank has been stared".

After that, I use M-X slime-connect from Emacs. The port number, as you can see from the code, is 7777.

What is the result?

I can inspect and control the running system, for example invoke functions I need to test or change object states.

But I must admit that the level of flexibility is lower than I hoped for. The point is that I can not interactively change Java code (which constitutes all the system). To be more flexible, the system must be written on Lisp entirely.

Links

• Bill Clementson's instructions for Clojure/SLIME setup: http://bc.tech.coop/blog/081023.html
• My JSP page to start Clojure/swank: http://paste.lisp.org/display/71209

slime-over-http update

I've just updated my slime-over-http code to work with the current SLIME.

Web development in single threaded lisp with slime and hunchentoot

It is problematic at first sight to use SLIME for web development in a single threaded lisp with swank:*communication-style* = nil (as it happens with CLISP and SBCL on Windows). The problem is the impossibility to modify application while it is running.

In very informal pseudo code this configuration of swank may be simplified represented as:

(loop
   (read-slime-request)
   (exec-slime-request))

When our next SLIME request starts hunchentoot, hunchentoot enters its own request handling loop (for HTTP requests):

(loop
   (read-slime-request)
   (exec-slime-request :
      (loop
         (read-http-request)
         (handle-http-request)))

This means exec-slime-request will not exit and subsequent slime requests will not be handled until the web server is stopped. SLIME isn't functioning.

The solution is to use web server to invoke SLIME. When in the emacs we are going to talk to swank, send special HTTP request and its handler will enforce swank to handle emacs request.

(loop
   (read-slime-request)
   (exec-slime-request :
      (loop
         (read-http-request)
         (handle-http-request :
            (read-slime-request)
            (exec-slime-request))))

It requires only few lines of code in emacs lisp and in common lisp. The code is here, it's simple, see it for more details.

One note is needed to help fully understand the code. When emacs side sends a request to swank, swank is not always blocked by hunchentoot. As far as I understand it, one slime request may lead to a sequence of communications between swank and emacs.

01: (loop
02:   (read-slime-request)
03:   (exec-slime-request :
04:      (loop
05:         (read-http-request)
06:         (handle-http-request :
07:            (read-slime-request)
08:            (exec-slime-request :
09:               ...
10:               (read-slime-request)
11:               ...))))

In the emacs we should not make an HTTP request before sending the slime request that is to be read on the line 10.

To distinguish this situation on the emacs side I check whether the socket of HTTP request read by swank on the line 05 is still connected. In this case new HTTP request isn't performed.

This solution isn't 100% correct (the socket may be still not closed, but swank isn't going to read anything). But it works almost good. Sometimes a glitch appears - when you execute an expression, you get result of previous expression instead of the just executed. If you execute expressions en, en+1, en+2, you get results of executing expressions en-1, en, en+1. This "phase displacement" is most likely caused by the fact that we not performed HTTP request when it was necessary and the slime request remained not read by swank. When next time we do HTTP request for the next slime request, the previous unread request is read and executed by swank.

In principle, I know how to fix this problem (we may annotate HTTP requests and slime requests with some IDs and do synchronization - ignore unnecessary HTTP requests - on the common lisp side; socket checking on the emacs side will became unnecessary). But I do not want to spend time on it, it is easier to enforce reading enqueued expressions by the "special HTTP request" directly from browser: fetch http://localhost:4343/slime-http-send-handler needed count of times.

How to use it.

Very easy.

Add the following line into your .emacs file:

(load "c:/some/path/slime-over-http.el")

The line should follow the normal slime setup.

Here is an example of common lisp web site:

;; this line may vary in you case; just make
;; sure that asdf is loaded and all the needed
;; libraries are available
(load "c:/usr/unpacked/lisp-libs/setup-asdf-registry.lisp")

(asdf:operate 'asdf:load-op :hunchentoot)
(use-package :hunchentoot)

(defun want-slime-over-http-p ()
  "This predicate indicates whether we
are in the development environment and have no
threading"
  #+:win32 t)

(when (want-slime-over-http-p)
  (load "C:/usr/projects/slime-over-http/slime-over-http.lisp")
  (defun hunchentoot::keep-alive-p ()))

(setq *dispatch-table*
      `(dispatch-easy-handlers
        ,@(when (want-slime-over-http-p)
                (list (create-prefix-dispatcher
                           "/slime-http-send-handler"
                           (find-symbol "SLIME-HTTP-SEND-HANDLER"
                                        "SLIME-HTTP"))))
        default-dispatcher))

(define-easy-handler (hello :uri "/hello")
    ()
  "Hello, change me!")

After executing the above code run emacs command slime-start-hunchentoot. Now you may open http://localhost:4343/hello in the browser and see the phrase "Hello, change me!".

You may change it because you have fully functioning slime with repl, debugger, etc.

To stop the server execute common lisp function (slime-http::stop-instance).

Note how we redefined hunchentoot::keep-alive-p to return nil. Single threaded server can't provide keep-alive, because while it is handling alive connection it can't accept any other connections (in particular from slime; also, IE is often unable to fully open a page with several images or frames in this case; looks like it opens several connections and schedules downloading different parts of the page to different connections; connections other than "kept alive" are not handled by hunchentoot).

In addition to redefining keep-alive-p I also use reverse proxying apache for sure.

Resume

So we see quite simple way (despite the long explanation) to do web development with hunchentoot in free lisps available on windows.

It isn't the only way to develop web applications with common lisp if you are on windows. For example you may install linux virtual machine. Edi Weitz - the hunchentoot author - uses this approach.

PS

My code is called "slime-over-http" because initially I delivered slime requests over the TCP connection used by the HTTP request and only some time later I realized that it is unnecessary. I'd like another name (preferably short), but can't contrive it.

If you have any questions about this configuration - ask, I'll be glad to help.