Positronia

Building a REPL OS in Common Lisp

Building a REPL OS in Common Lisp, Part 5: Persistence

Part 5 of a series where we build a custom operating system that runs inside a Lisp REPL.

The Problem#

Our virtual filesystem works, but it’s ephemeral. Quit the REPL, and everything is gone. Every OS needs persistence: a way to save state and restore it later.

In this new chapter we’ll add the ability to save the filesystem to disk and load it back on startup. We’ll also add tree and find commands to visualize and search the filesystem.

Serialization#

We need to convert our tree of nodes to something we can write to a file. Most languages require you to choose a format (JSON, XML, YAML) and use a library to parse and generate it.

Lisp has a superpower: S-expressions. The same syntax we use for code is also a data format. And Lisp has built-in functions to read and write it:

  • print writes any Lisp object to a stream in a format that can be read back
  • read parses that format back into a Lisp object

No library needed. No schema definition. No version compatibility headaches. The same reader that parses your source code parses your data files.

(defvar *fs-path* (merge-pathnames ".repl-fs.dat" (asdf:system-source-directory :repl)))

(defun node-to-sexp (node)
  "Convert a node tree to an S-expression for serialization."
  (list :name (node-name node)
        :type (node-type node)
        :contents (node-contents node)
        :children (mapcar #'node-to-sexp (node-children node))))

(defun sexp-to-node (sexp)
  "Convert an S-expression back to a node tree."
  (make-node :name (getf sexp :name)
             :type (getf sexp :type)
             :contents (getf sexp :contents)
             :children (mapcar #'sexp-to-node (getf sexp :children))))

Simple. node-to-sexp recursively converts the tree to nested property lists. sexp-to-node does the reverse. The data file lives in the project directory.

Save and Load#

(defun save-fs ()
  "Save the virtual filesystem to disk."
  (with-open-file (out *fs-path*
                       :direction :output
                       :if-exists :supersede)
    (print (node-to-sexp *fs*) out))
  (format t "~%Filesystem saved to ~A" *fs-path*))

(defun load-fs ()
  "Load the virtual filesystem from disk."
  (if (probe-file *fs-path*)
      (progn
        (with-open-file (in *fs-path*)
          (setf *fs* (sexp-to-node (read in))))
        (format t "~%Filesystem loaded from ~A" *fs-path*))
      (format t "~%No saved filesystem found. Starting fresh.")))

save-fs writes the tree to .repl-fs.dat. load-fs reads it back if the file exists, or starts fresh if not.

Here’s what the file looks like after creating /home with a notes.txt file:

(:NAME "/" :TYPE :DIRECTORY :CONTENTS ""
 :CHILDREN
 ((:NAME "home" :TYPE :DIRECTORY :CONTENTS ""
   :CHILDREN
   ((:NAME "notes.txt" :TYPE :FILE
     :CONTENTS "Remember to add mounts in v6!"
     :CHILDREN NIL)))))

It’s just a nested property list: the same S-expression syntax we use everywhere in Lisp. Perks of Lisp.

This is the beauty of print/read as a serialization pair:

  • print outputs data in a format read can parse
  • read returns exactly what print wrote
  • They’re inverses of each other, built into the language

No JSON library, no schema, no versioning issues. If you can build it from lists, symbols, strings, and numbers, you can serialize it. This is one of Lisp’s quiet superpowers.

Executing Code from Files#

We can store data in files. But what about code?

Imagine we want a tree command to visualize the filesystem. We could hardcode it in repl.lisp, but there’s a more interesting approach: store the command definition in a file and load it.

This is how real operating systems work. Commands live in /bin or /usr/bin. They’re just files. We can do the same.

First, we need a way to execute code from a file:

(register-command
 (make-command
  :name 'load-file
  :description "Load and execute a Lisp file: (load-file \"/bin/tree.lisp\")"
  :function (lambda (path)
              (let ((node (find-node path)))
                (if (and node (eq (node-type node) :file))
                    (progn
                      (with-input-from-string (in (node-contents node))
                        (loop for form = (read in nil :eof)
                              until (eq form :eof)
                              do (dispatch form)))
                      (format t "~%Loaded ~A" path))
                    (format t "~%No such file: ~A" path))))))

This reads all forms from a file and passes each to dispatch, the same function the REPL uses. So a file can contain commands, Lisp code, or new command definitions.

Creating User Commands#

Let’s create a /bin directory and a simple command:

λ (mkdir "/bin")
Created /bin

λ (write "/bin/hello.lisp" "(register-command (make-command :name 'hello :description \"Say hello\" :function (lambda () (format t \"~%Hello from user space!\"))))")
Created and wrote to /bin/hello.lisp

λ (load-file "/bin/hello.lisp")
Loaded /bin/hello.lisp

λ (hello)
Hello from user space!

The command we just created works. This is self-hosting: the OS can extend itself.

There’s a practical limitation: the REPL reads a complete form when you press Enter, so you can’t easily write multi-line code. Simple one-liners work, but complex commands like tree would be tedious to type. I leave to the reader the pleasure of experimenting with it.

The ideal solution would be to have a nice text editor (perhaps in the future). For now, the real solution is to edit files externally, which is exactly why we need mounts (Part 6). For now, we include tree and find as built-ins.

Built-in Commands: Tree and Find#

Since complex commands are hard to type at the REPL, we include tree and find as built-ins:

(register-command
 (make-command
  :name 'tree
  :description "Show filesystem tree"
  :function (lambda ()
              (labels ((print-tree (node depth)
                         (format t "~%~A~A~A"
                                 (make-string (* depth 2) :initial-element #\Space)
                                 (node-name node)
                                 (if (eq (node-type node) :directory) "/" ""))
                         (dolist (child (node-children node))
                           (print-tree child (1+ depth)))))
                (print-tree *fs* 0)))))

Remember DFS? Depth-first search: visit a node, then recurse into each child before backtracking. We print the current node, recurse into children with increased depth. The depth controls indentation.

We use labels to define a local recursive function. Why not defun?

  • print-tree is only used inside this command
  • No need to pollute the global namespace
  • The function is defined right where it’s used, making the code self-contained

labels is like let for functions. It creates local bindings that can reference themselves (for recursion) or each other.

(register-command
 (make-command
  :name 'find
  :description "Find files by name: (find \"notes\")"
  :function (lambda (pattern)
              (let ((results '()))
                (labels ((dfs (node path)
                           (let ((current-path (if (string= path "")
                                                   (node-name node)
                                                   (format nil "~A/~A" path (node-name node)))))
                             (when (search pattern (node-name node))
                               (push current-path results))
                             (dolist (child (node-children node))
                               (dfs child current-path)))))
                  (dfs *fs* ""))
                (if results
                    (dolist (path (nreverse results))
                      (format t "~%  ~A" path))
                    (format t "~%No matches for ~A" pattern))))))

Also DFS. Why DFS and not BFS (breadth-first search)?

  • DFS explores one branch completely before moving to the next
  • BFS explores all nodes at depth 1, then depth 2, etc.
  • DFS is simpler to implement with recursion
  • For “find all matches,” both work. DFS just finds them in a different order

BFS would require a queue to track nodes to visit. DFS uses the call stack naturally. For our use case, DFS is simpler and works fine.

A few things to note:

  • We use search (a built-in) to check if pattern is a substring of the node name
  • Results are collected with push, which builds the list in reverse order
  • We use nreverse at the end to restore the original order (root-to-leaf)
  • The path is built up as we descend, so each result has its full path

Auto-Save and Auto-Load#

The REPL now loads the filesystem on startup:

(defun my-repl ()
  "Start the REPL. Type (help) for commands, (quit) to exit."
  (load-fs)
  (format t "~%Welcome to REPL OS v5")
  (format t "~%Type (help) for commands, or any Lisp expression.~%")
  (catch 'repl-exit
    (loop
      (print-prompt)
      (let ((form (read)))
        (record-history form)
        (handler-case
            (dispatch form)
          (error (e)
            (format t "~%ERROR: ~A" e)))))))

And quit auto-saves:

(register-command
 (make-command
  :name 'quit
  :description "Exit the REPL"
  :function (lambda ()
              (save-fs)
              (format t "~%Goodbye.~%")
              (throw 'repl-exit nil))))

Running It#

(repl:my-repl)

No saved filesystem found. Starting fresh.
Welcome to REPL OS v5
Type (help) for commands, or any Lisp expression.

λ (mkdir "/bin")
Created /bin

λ (write "/bin/greet.lisp" "(register-command (make-command :name 'greet :description \"Greet someone\" :function (lambda (name) (format t \"~%Hello, ~A!\" name))))")
Created and wrote to /bin/greet.lisp

λ (load-file "/bin/greet.lisp")
Loaded /bin/greet.lisp

λ (greet "world")
Hello, world!

λ (tree)
//
  bin/
    greet.lisp

λ (quit)
Filesystem saved to /path/to/repl/.repl-fs.dat
Goodbye.

Start again:

Filesystem loaded from /path/to/repl/.repl-fs.dat
Welcome to REPL OS v5
Type (help) for commands, or any Lisp expression.

λ (greet "world")
ERROR: Undefined function: GREET

λ (load-file "/bin/greet.lisp")
Loaded /bin/greet.lisp

λ (greet "world")
Hello, world!

The filesystem persisted, but the command didn’t. It was only in memory. We need to reload it each session. An init file could solve this, or we could build auto-loading into the REPL.

What We Built#

  1. Serialization: convert node tree to S-expressions and back
  2. Persistence: save to disk, load on startup
  3. Auto-save/load: filesystem survives restarts
  4. load-file: execute Lisp code stored in virtual files
  5. User space: commands can be created and stored in /bin

This is the beginning of self-hosting. The OS can now extend itself. Users can write their own commands, store them in the filesystem, and load them when needed.

We showed tree and find as examples of what user commands look like. (In the actual code, they’re built-in for convenience, but they could live in /bin.) They use DFS (depth-first search) to traverse the filesystem tree. You could write more: grep to search file contents, cp to copy files, mv to rename them.

What’s Missing#

We can create and run code, but we can’t access the real filesystem. The virtual filesystem is an island. We can’t read the source code of the REPL itself.

In Part 6, we’ll add mounts: the ability to map virtual paths to real directories. (mount "/sys" "/path/to/repl/") will let us read and write actual files. Then we can truly modify the OS from within.

Next: Part 6: Mounts