Hunchentoot on ECL

It was reported that all Hunchentoot dependencies are already ported to ECL and a patch exists for Hunchentoot itself to make it running on ECL.

I am glad to hear this, because ECL is the only free Common Lisp on Windows with multithreading support (at the same time an active work on multithreading for CLISP is in progress, Clozure CL has beta port for Windows, and a lot of work was also done for threading in SBCL Windows port, although the work is not finished).

I tried Hunchentoot on ECL and for those who are interested too, I reproduce here the steps I took. My environment: Windows XP + Microsoft Visual Studio.

1. You need the latest versions of Hunchentoot and all the dependencies. If you are lazy, you may get all them at once from this archive I prepared.
2. Apply the patch from this post (already done in my archive): save the port-ecl.lisp to the hunchentoot directory and add #+:ecl (:file "port-ecl") to hunchentoot.asd as here.
3. Checkout fresh ECL from CVS:

     cvs -z3 -d:pserver:anonymous@ecls.cvs.sourceforge.net:/cvsroot/ecls checkout ecl
   

   I recommend fresh checkout because I had problems building ECL from sources already existing on my computer after update with cvs update -dAP.
4. Configure ECL. Open ecl\msvc\Makefile and set

   ECL_THREADS  = 1
   ECL_UNICODE  = 1
   ECL_ASDF     = 1
   ECL_SOCKETS  = 1
   

5. Build ECL. Run Visual Studio command prompt: menu Start->All Programs -> Microsoft Visual Studio 2005 -> Visual Studio Tools -> Visual Studio 2005 Command Prompt. This script sets up environment variables so that C compiler cl.exe, nmake.exe and other things are in your PATH, all includes and libraries are available, etc. In this console window type:

    > cd <DIR_WITH_ECL_SOURCES>\ecl\msvc
    > nmake
   

   After the build completes, copy resulting binaries somewhere:

     > nmake install prefix=c:\somewhere\ecl

   Now you have working ECL, lets test it:

   > c:\somewhere\ecl\ecl.exe
   ECL (Embeddable Common-Lisp) 0.9l (CVS 2008-07-12 18:54)
   Copyright (C) 1984 Taiichi Yuasa and Masami Hagiya
   Copyright (C) 1993 Giuseppe Attardi
   Copyright (C) 2000 Juan J. Garcia-Ripoll
   ECL is free software, and you are welcome to redistribute it
   under certain conditions; see file 'Copyright' for details.
   Type :h for Help.  Top level.
   > (+ 2 2)
   4
   > (quit)
   

   Works.

   Do not close the Visual Studio command prompt, we will need it soon.

6. As Hunchentoot uses ASDF, we need Hunchentoot with all its dependencies to be registered in ASDF:*CENTRAL-REGISTY*. For this I have a special lisp file, setup-asdf-registry.lisp (already included in the archive from the step 1):

   ;; this file must be stored in the same directory
   ;; where all the lisp libraries are stored 
   ;; (but libraries, of course, have their own subdirectories)
   
   (require 'asdf)
   
   (flet
       ((reg (relative-lib-dir)
          (let ((lib-dir
                 (directory-namestring 
                  (merge-pathnames relative-lib-dir
                                   (load-time-value
                                    (or #.*compile-file-pathname* 
                                        *load-pathname*))))))
            (pushnew lib-dir
                     asdf:*central-registry*
                     :test #'equalp))))
     (reg "alexandria\\")
     (reg "asdf-binary-locations\\")
     (reg "babel_0.2.0\\")
     (reg "cffi-080926\\")
     (reg "chunga-0.4.3\\")
     (reg "cl-base64-3.3.2\\")
     (reg "cl-fad-0.6.2\\") 
     (reg "cl-ppcre-2.0.1\\")
     (reg "cl-who-0.11.0\\")
     (reg "cl+ssl\\")
     (reg "flexi-streams-1.0.7\\")
     (reg "hunchentoot-0.15.7\\")
     (reg "md5-1.8.5\\")
     (reg "rfc2388\\")
     (reg "trivial-features_0.1\\")
     (reg "trivial-gray-streams\\")
     (reg "url-rewrite-0.1.1\\"))
   
   (asdf:operate 'asdf:load-op :asdf-binary-locations) 

7. Compile and load Hunchentoot. For this go back to the Visual Studio command prompt.

   > c:\somewhere\ecl\ecl.exe  
   > (load "c:/archive-from-the-step-1/setup-asdf-registry.lisp") 
   > (pushnew :hunchentoot-no-ssl *features*)
   > (asdf:operate 'asdf:load-op :hunchentoot-test)
   

   ECL must be started from Visual Studio command prompt because it uses C compiler when compiling Lisp. (pushnew :hunchentoot-no-ssl *features*) is only necessary if you do not have OpenSSL installed (get it here for Win32).
8. Start Hunchentoot:

   > (hunchentoot:start-server :port 4242) 
   

Now you may open http://localhost:4242/hunchentoot/test and see Hunchentoot test site.

At the moment of this post writing, almost all examples from the test site work, except for two: "UTF-8 demo (writing UTF-8 characters directly to the stream)" and "UTF-8 demo (returning a string)". The reason is already found and I am sure it will be fixed very soon.

slime-over-http update

I've just updated my slime-over-http code to work with the current SLIME.

Web development in single threaded lisp with slime and hunchentoot

It is problematic at first sight to use SLIME for web development in a single threaded lisp with swank:*communication-style* = nil (as it happens with CLISP and SBCL on Windows). The problem is the impossibility to modify application while it is running.

In very informal pseudo code this configuration of swank may be simplified represented as:

(loop
   (read-slime-request)
   (exec-slime-request))

When our next SLIME request starts hunchentoot, hunchentoot enters its own request handling loop (for HTTP requests):

(loop
   (read-slime-request)
   (exec-slime-request :
      (loop
         (read-http-request)
         (handle-http-request)))

This means exec-slime-request will not exit and subsequent slime requests will not be handled until the web server is stopped. SLIME isn't functioning.

The solution is to use web server to invoke SLIME. When in the emacs we are going to talk to swank, send special HTTP request and its handler will enforce swank to handle emacs request.

(loop
   (read-slime-request)
   (exec-slime-request :
      (loop
         (read-http-request)
         (handle-http-request :
            (read-slime-request)
            (exec-slime-request))))

It requires only few lines of code in emacs lisp and in common lisp. The code is here, it's simple, see it for more details.

One note is needed to help fully understand the code. When emacs side sends a request to swank, swank is not always blocked by hunchentoot. As far as I understand it, one slime request may lead to a sequence of communications between swank and emacs.

01: (loop
02:   (read-slime-request)
03:   (exec-slime-request :
04:      (loop
05:         (read-http-request)
06:         (handle-http-request :
07:            (read-slime-request)
08:            (exec-slime-request :
09:               ...
10:               (read-slime-request)
11:               ...))))

In the emacs we should not make an HTTP request before sending the slime request that is to be read on the line 10.

To distinguish this situation on the emacs side I check whether the socket of HTTP request read by swank on the line 05 is still connected. In this case new HTTP request isn't performed.

This solution isn't 100% correct (the socket may be still not closed, but swank isn't going to read anything). But it works almost good. Sometimes a glitch appears - when you execute an expression, you get result of previous expression instead of the just executed. If you execute expressions en, en+1, en+2, you get results of executing expressions en-1, en, en+1. This "phase displacement" is most likely caused by the fact that we not performed HTTP request when it was necessary and the slime request remained not read by swank. When next time we do HTTP request for the next slime request, the previous unread request is read and executed by swank.

In principle, I know how to fix this problem (we may annotate HTTP requests and slime requests with some IDs and do synchronization - ignore unnecessary HTTP requests - on the common lisp side; socket checking on the emacs side will became unnecessary). But I do not want to spend time on it, it is easier to enforce reading enqueued expressions by the "special HTTP request" directly from browser: fetch http://localhost:4343/slime-http-send-handler needed count of times.

How to use it.

Very easy.

Add the following line into your .emacs file:

(load "c:/some/path/slime-over-http.el")

The line should follow the normal slime setup.

Here is an example of common lisp web site:

;; this line may vary in you case; just make
;; sure that asdf is loaded and all the needed
;; libraries are available
(load "c:/usr/unpacked/lisp-libs/setup-asdf-registry.lisp")

(asdf:operate 'asdf:load-op :hunchentoot)
(use-package :hunchentoot)

(defun want-slime-over-http-p ()
  "This predicate indicates whether we
are in the development environment and have no
threading"
  #+:win32 t)

(when (want-slime-over-http-p)
  (load "C:/usr/projects/slime-over-http/slime-over-http.lisp")
  (defun hunchentoot::keep-alive-p ()))

(setq *dispatch-table*
      `(dispatch-easy-handlers
        ,@(when (want-slime-over-http-p)
                (list (create-prefix-dispatcher
                           "/slime-http-send-handler"
                           (find-symbol "SLIME-HTTP-SEND-HANDLER"
                                        "SLIME-HTTP"))))
        default-dispatcher))

(define-easy-handler (hello :uri "/hello")
    ()
  "Hello, change me!")

After executing the above code run emacs command slime-start-hunchentoot. Now you may open http://localhost:4343/hello in the browser and see the phrase "Hello, change me!".

You may change it because you have fully functioning slime with repl, debugger, etc.

To stop the server execute common lisp function (slime-http::stop-instance).

Note how we redefined hunchentoot::keep-alive-p to return nil. Single threaded server can't provide keep-alive, because while it is handling alive connection it can't accept any other connections (in particular from slime; also, IE is often unable to fully open a page with several images or frames in this case; looks like it opens several connections and schedules downloading different parts of the page to different connections; connections other than "kept alive" are not handled by hunchentoot).

In addition to redefining keep-alive-p I also use reverse proxying apache for sure.

Resume

So we see quite simple way (despite the long explanation) to do web development with hunchentoot in free lisps available on windows.

It isn't the only way to develop web applications with common lisp if you are on windows. For example you may install linux virtual machine. Edi Weitz - the hunchentoot author - uses this approach.

PS

My code is called "slime-over-http" because initially I delivered slime requests over the TCP connection used by the HTTP request and only some time later I realized that it is unnecessary. I'd like another name (preferably short), but can't contrive it.

If you have any questions about this configuration - ask, I'll be glad to help.