Positronia

Building a REPL OS in Common Lisp

Building a REPL OS in Common Lisp, Part 3: Threading

Part 3 of a series where we build a custom operating system that runs inside a Lisp REPL.

What’s Next for Our REPL?#

Our REPL has commands now, but it’s missing some basic conveniences. Before we tackle the big feature of this article, threading, let’s add two simple but useful features: a prompt function and command history.

The Prompt#

We’ve been printing the prompt inline in the REPL loop. This is anticipatory, but as we add new features to our system, we’ll need to print the prompt from other places. Let’s extract it into a function:

(defun print-prompt ()
  (format t "~%λ ")
  (force-output))

Simple, but now we have one place to change if we want to modify the prompt later. If you prefer a different symbol, like > or * or ⊢, just change it here. You could even make it dynamic by using a variable:

(defvar *prompt* "λ")

(defun print-prompt ()
  (format t "~%~A " *prompt*)
  (force-output))

You could customize it at runtime with (setf *prompt* ">"). I’ll keep the simple version for now, but the option is there.

History#

A classic, straight from the UNIX shell: Command history lets users see what they’ve typed and re-run previous commands. The implementation is simple:

(defvar *history* '())

(defun record-history (form)
  (push form *history*))

(defun get-history ()
  (reverse *history*))

We record every form entered, and can retrieve them in order. Two commands to use it:

(register-command
 (make-command
  :name 'history
  :description "Show command history"
  :function (lambda ()
              (loop for form in (get-history)
                    for i from 1
                    do (format t "~%  ~A: ~S" i form)))))

(register-command
 (make-command
  :name 'replay
  :description "Re-run a history entry: (replay 3)"
  :function (lambda (n)
              (let ((form (nth (1- n) (get-history))))
                (if form
                    (progn
                      (format t "~%Replaying: ~S" form)
                      (dispatch form))
                    (format t "~%No history entry ~A." n))))))

history lists what you’ve typed. replay re-runs a specific entry by number using the nth function.

The Updated REPL Loop#

The main loop now uses print-prompt and records history:

(defun my-repl ()
  "Start the REPL. Type (help) for commands, (quit) to exit."
  (format t "~%Welcome to REPL OS v3")
  (format t "~%Type (help) for commands, or any Lisp expression.~%")
  (catch 'repl-exit
    (loop
      (print-prompt)
      (let ((form (read)))
        (record-history form)
        (handler-case
            (dispatch form)
          (error (e)
            (format t "~%ERROR: ~A" e)))))))

With these basics in place I think we are more comfortable now to continue, so let’s tackle the main feature.

The Problem with Synchronous Code#

Our REPL works, but everything blocks. When you run a command, nothing else can happen until it finishes. What if you want to schedule a command to run later? Or run something periodically in the background while you keep working? Or you want to run something that you know will take some time, but you still want to use the system.

We need concurrency. We need threads.

Common Lisp doesn’t have a standard threading API, and each implementation does it differently. Bordeaux Threads is a library that provides a portable interface across all major implementations.

If you’re using Quicklisp:

(ql:quickload :bordeaux-threads)

The library gives us bt:make-thread to spawn threads, bt:destroy-thread to kill them, and bt:thread-alive-p to check their status.

The Thread Registry#

Just like we have a command registry, we’ll have a thread registry. We need to track running threads so we can list and kill them:

(defvar *threads* (make-hash-table))
(defvar *thread-id* 0)

(defun next-thread-id ()
  (incf *thread-id*))

Each thread gets a unique numeric ID. Now the core functions:

(defun spawn (name function)
  "Start a background thread and register it."
  (let* ((id (next-thread-id))
         (thread (bt:make-thread function :name name)))
    (setf (gethash id *threads*)
          (list :id id :name name :thread thread))
    id))

spawn creates a thread running function, registers it with an ID and name, and returns the ID. The registry entry is a property list with the ID, name, and thread object.

(defun kill-thread (id)
  "Kill a thread by its ID."
  (let ((entry (gethash id *threads*)))
    (when entry
      (bt:destroy-thread (getf entry :thread))
      (remhash id *threads*)
      t)))

kill-thread looks up the thread, destroys it, removes it from the registry, and returns t on success.

(defun list-threads ()
  "Print all registered threads."
  (maphash (lambda (id entry)
             (declare (ignore id))
             (format t "~%  [~A] ~A (~A)"
                     (getf entry :id)
                     (getf entry :name)
                     (if (bt:thread-alive-p (getf entry :thread))
                         "alive" "dead")))
           *threads*))

list-threads iterates over the registry and prints each thread’s status.

Thread Commands#

Now we expose this to users with ps and kill:

(register-command
 (make-command
  :name 'ps
  :description "List running threads"
  :function (lambda ()
              (format t "~%Threads:")
              (list-threads))))

(register-command
 (make-command
  :name 'kill
  :description "Kill a thread by id: (kill 1)"
  :function (lambda (id)
              (if (kill-thread id)
                  (format t "~%Killed thread ~A." id)
                  (format t "~%No thread with id ~A." id)))))

Simple wrappers around our thread functions.

Scheduling: The After Command#

Let’s build something useful like a command that runs another command after a delay:

(register-command
 (make-command
  :name 'after
  :description "Run command after N seconds: (after 5 (help))"
  :function (lambda (seconds command)
              (let ((id (spawn "after"
                          (lambda ()
                            (sleep seconds)
                            (format t "~%[after]")
                            (dispatch command)
                            (print-prompt)))))
                (format t "~%Scheduled (thread ~A)." id)))))

(after 5 (help)) spawns a thread that sleeps for 5 seconds, prints [after] so you know it fired, then dispatches the command (help). The second argument is the form to execute, which is the same form you’d type at the prompt. We pass it directly to dispatch, which already knows how to run commands or evaluate Lisp. After the command finishes, we reprint the prompt so the user knows the REPL is ready.

This approach is composable, so you can do things like (after 5 (after 3 (help))) and it will work. You are scheduling a scheduler. The form you pass is just data until it’s time to run it. Perks of Lisp.

Repeating: The Every Command#

What about tasks that need to run repeatedly? Monitoring, polling, periodic checks:

(register-command
 (make-command
  :name 'every
  :description "Run command every N seconds: (every 5 (ps))"
  :function (lambda (seconds command)
              (let ((id (spawn "every"
                          (lambda ()
                            (loop
                              (sleep seconds)
                              (format t "~%[every]")
                              (dispatch command)
                              (print-prompt))))))
                (format t "~%Started (thread ~A). Use (kill ~A) to stop." id id)))))

(every 10 (ps)) runs (ps) every 10 seconds until you kill it. The thread will loop forever: sleep, print [every], dispatch, repeat. You stop it with (kill N).

Project Structure#

We have advanced quite a lot at this point. Our REPL OS handles exceptions, runs Lisp expressions, runs our own Commands, supports multi-threading and now we can start building interesting and useful stuff. If you are not familiar with Lisp it may feel weird, but I think that the code is small and easy enough to follow by anyone that likes to develop software and has a bit of curiosity about Lisp.

Now let’s review the project structure and all files:

repl/
  repl.asd       # Now depends on bordeaux-threads
  packages.lisp  # Exports thread and history functions
  core.lisp      # Command system (unchanged from v2)
  threads.lisp   # Thread registry
  repl.lisp      # Main loop and all commands

repl.asd#

repl.asd
(asdf:defsystem :repl
  :description "A custom REPL in Common Lisp - Version 3: Threading"
  :version "3.0.0"
  :depends-on (:bordeaux-threads)
  :serial t
  :components
  ((:file "packages")
   (:file "core")
   (:file "threads")
   (:file "repl")))

packages.lisp#

packages.lisp
(defpackage :repl
  (:use :cl)
  (:export
   ;; Entry point
   #:my-repl

   ;; Command system
   #:command
   #:make-command
   #:command-name
   #:command-description
   #:command-function
   #:register-command
   #:find-command
   #:dispatch
   #:*commands*

   ;; Thread system
   #:spawn
   #:kill-thread
   #:list-threads
   #:*threads*

   ;; History
   #:record-history
   #:get-history
   #:*history*))

threads.lisp#

threads.lisp
(in-package :repl)

(defvar *threads* (make-hash-table))
(defvar *thread-id* 0)

(defun next-thread-id ()
  (incf *thread-id*))

(defun spawn (name function)
  "Start a background thread and register it."
  (let* ((id (next-thread-id))
         (thread (bt:make-thread function :name name)))
    (setf (gethash id *threads*)
          (list :id id :name name :thread thread))
    id))

(defun kill-thread (id)
  "Kill a thread by its ID."
  (let ((entry (gethash id *threads*)))
    (when entry
      (bt:destroy-thread (getf entry :thread))
      (remhash id *threads*)
      t)))

(defun list-threads ()
  "Print all registered threads."
  (maphash (lambda (id entry)
             (declare (ignore id))
             (format t "~%  [~A] ~A (~A)"
                     (getf entry :id)
                     (getf entry :name)
                     (if (bt:thread-alive-p (getf entry :thread))
                         "alive" "dead")))
           *threads*))

Running It#

First, make sure you have Bordeaux Threads:

(ql:quickload :bordeaux-threads)

Then:

cd repl
sbcl

(load "repl.asd")
(asdf:load-system :repl)
(repl:my-repl)

Welcome to REPL OS v3
Type (help) for commands, or any Lisp expression.

λ (after 3 (help))
Scheduled (thread 1).

λ (+ 1 2)
=> 3

[after]
Available commands:
  (QUIT) - Exit the REPL
  (HELP) - List available commands
  ...
λ
λ (every 5 (ps))
Started (thread 2). Use (kill 2) to stop.

λ (ps)
Threads:
  [1] after (dead)
  [2] every (alive)

[every]
Threads:
  [1] after (dead)
  [2] every (alive)
λ
λ (kill 2)
Killed thread 2.

λ (history)
  1: (AFTER 3 (HELP))
  2: (+ 1 2)
  3: (EVERY 5 (PS))
  4: (PS)
  5: (KILL 2)
  6: (HISTORY)

after scheduled (help) to run 3 seconds later. While waiting, we did some math. Then help appeared. We started every to run (ps) every 5 seconds, saw it trigger, then killed it.

What We Built#

  1. print-prompt: centralized prompt printing
  2. history/replay: command recall
  3. Thread registry: track spawned threads by ID
  4. spawn/kill/list: core thread management
  5. ps and kill commands: user-facing thread control
  6. after: schedule a command for later
  7. every: run a command repeatedly

There you have it. Our REPL now has real concurrency. Background tasks run while we keep working. It’s starting to feel like an operating system. Sort of.

What’s Next?#

We have commands and threads. But where do we store data? In Part 4, we’ll build a virtual filesystem with directories and files that exist only in memory. We’ll add persistence so the filesystem survives restarts. And later we’ll implement a mount system so we can access real files alongside virtual ones.

Next: Part 4: The Filesystem