Positronia

Building a REPL OS in Common Lisp

Building a REPL OS in Common Lisp, Part 7: Unified Filesystem

Part 7 of a series where we build a custom operating system that runs inside a Lisp REPL.

The Problem#

In Part 6, we added mounts that broght us the ability to access real directories through virtual paths. But we left a gap: tree and find only saw the virtual filesystem. ls "/" showed /home but not /sys.

Two worlds fractured that we are going to unify now. Every command will see the same filesystem so that virtual directories and mounted paths seem to coexist in the same abstract storage space.

In order to achieve this we need to change the point of view, now we need functions that work with paths, not nodes, and we take care of the specific routing internally. A path like /sys/repl.lisp should work the same whether it points to a virtual file or a mounted one.

So we need three operations:

  • list-children: What’s inside this directory?
  • get-entry-type: Is this a file or directory (or nothing)?
  • get-entry-contents: What’s in this file?

If these work uniformly across both worlds, every command can use them without caring about the underlying storage.

Finding Mount Children#

First, we need to detect mount points that appear as children of a path. If /sys is mounted, then ls "/" should show sys/.

(defun get-mount-children (path)
  "Return mount points that are direct children of path.
   Returns list of (name . :directory) pairs."
  (let ((results '())
        (prefix (if (string= path "/") "/" (concatenate 'string path "/"))))
    (maphash (lambda (vpath rpath)
               (declare (ignore rpath))
               (when (and (> (length vpath) (length prefix))
                          (string= prefix (subseq vpath 0 (length prefix))))
                 (let* ((remainder (subseq vpath (length prefix)))
                        (slash-pos (position #\/ remainder)))
                   (unless slash-pos
                     (push (cons remainder :directory) results)))))
             *mounts*)
    (when (string= path "/")
      (maphash (lambda (vpath rpath)
                 (declare (ignore rpath))
                 (when (and (> (length vpath) 1)
                            (char= (char vpath 0) #\/)
                            (not (position #\/ vpath :start 1)))
                   (push (cons (subseq vpath 1) :directory) results)))
               *mounts*))
    results))

Let’s break this down.

prefix is the string we expect each candidate mount path to start with. For path /home, we want mounts that begin with /home/. For the root /, prefix is just /. The special case avoids producing //, which would never match anything.

maphash iterates over every entry in the *mounts* hash table, calling the lambda with the virtual path and the real path of each one. We declare (ignore rpath) because we only care that a mount exists at this level, not where it points to.

For each mount, we ask two questions:

  • Does the mount path start with our prefix? (subseq vpath 0 (length prefix)) pulls the first few characters of vpath, and string= compares them. If they match, this mount lives under our path.
  • Is it a direct child, not a grandchild? After stripping the prefix, the remainder must contain no further slashes. A mount at /home/inner/deep would leave remainder = "inner/deep", and we’d skip it. We only list one level at a time.

If both pass, we push the remainder as a directory entry. Mount points always look like directories from the outside.

The second maphash, guarded by (string= path "/"), picks up mounts that hang directly off the root. /sys is a child of / with nothing in between. We extract its name by skipping the leading slash with (subseq vpath 1).

Either way, the function returns a list of (name . :directory) pairs. That shape will rule for the rest of this chapter: every helper below either produces it or consumes it.

Unified Listing#

Now we can build list-children:, and every command will lean on it:

(defun list-children (path)
  "List children at path, merging virtual nodes and mount points.
   Returns list of (name . type) pairs."
  (multiple-value-bind (real-path mounted) (resolve-mount path)
    (if mounted
        (list-real-directory real-path)
        (let ((entries '()))
          (let ((node (find-node path)))
            (when (and node (eq (node-type node) :directory))
              (dolist (child (node-children node))
                (push (cons (node-name child) (node-type child)) entries))))
          (dolist (mount-child (get-mount-children path))
            (unless (assoc (car mount-child) entries :test #'string=)
              (push mount-child entries)))
          entries))))

Let’s read it carefully.

resolve-mount tells us whether path itself is inside a mount. If it is, we delegate the listing entirely to list-real-directory. The real filesystem already knows what’s there, and we trust it.

If the path is not inside a mount, we’re in the virtual world. We find the corresponding node and collect its children into entries, each as a (name . type) pair, matching the same shape list-real-directory returns. Two worlds, one data structure.

Then we ask: are any mount points hanging off this directory? get-mount-children answers that. We merge those in, skipping any name that’s already present (a virtual directory and a mount could collide on the same name). assoc is the list equivalent of gethash: it scans for the first pair whose car matches the key. With :test #'string= it compares strings instead of using the default eql.

The result is a single list mixing whichever virtual children exist and whichever mount points appear at this level. Now callers don’t need to know which is which.

Type and Contents#

Two more helpers complete the API:

(defun get-entry-type (path)
  "Return the type (:file or :directory) of the entry at path, or nil."
  (multiple-value-bind (real-path mounted) (resolve-mount path)
    (if mounted
        (let ((probe (probe-file real-path)))
          (when probe
            (if (uiop:directory-pathname-p probe) :directory :file)))
        (let ((node (find-node path)))
          (when node (node-type node))))))

(defun get-entry-contents (path)
  "Return the contents of the file at path, or nil."
  (multiple-value-bind (real-path mounted) (resolve-mount path)
    (if mounted
        (read-real-file real-path)
        (let ((node (find-node path)))
          (when (and node (eq (node-type node) :file))
            (node-contents node))))))

Same shape, twice in a row. resolve-mount decides which world we’re in, and then we ask that world the question.

In the mounted world, get-entry-type uses probe-file (a Common Lisp built-in that returns the canonical pathname if the file exists, or nil if it doesn’t) and uiop:directory-pathname-p to tell directories from files. get-entry-contents calls read-real-file, the wrapper we wrote in Part 6.

In the virtual world, we walk the node tree with find-node (from Part 4) and read the slots we already have: node-type and node-contents.

The repetition is on purpose. Each helper exposes exactly one question, and the dispatch lives in the helper rather than scattered across every command. Once the questions are isolated, the commands collapse into a few lines.

Simplified Commands#

With the unified API, commands become simpler. Here’s the new ls:

(register-command
 (make-command
  :name 'ls
  :description "List directory contents: (ls \"/home\")"
  :function (lambda (&optional (path "/"))
              (let ((entries (list-children path)))
                (if entries
                    (dolist (entry entries)
                      (format t "~%  ~A~A"
                              (car entry)
                              (if (eq (cdr entry) :directory) "/" "")))
                    (format t "~%  (empty)"))))))

No more multiple-value-bind. No branching on mounted vs virtual. Just list-children.

And cat:

(register-command
 (make-command
  :name 'cat
  :description "Show file contents: (cat \"/home/notes.txt\")"
  :function (lambda (path)
              (let ((entry-type (get-entry-type path)))
                (cond
                  ((null entry-type)
                   (format t "~%No such file: ~A" path))
                  ((eq entry-type :directory)
                   (format t "~%~A is a directory" path))
                  (t
                   (format t "~%~A" (get-entry-contents path))))))))

Unified Tree#

The tree command now traverses everything:

(register-command
 (make-command
  :name 'tree
  :description "Show filesystem tree"
  :function (lambda ()
              (labels ((print-tree (path name depth)
                         (let ((entry-type (get-entry-type path)))
                           (format t "~%~A~A~A"
                                   (make-string (* depth 2) :initial-element #\Space)
                                   name
                                   (if (eq entry-type :directory) "/" ""))
                           (when (eq entry-type :directory)
                             (dolist (child (list-children path))
                               (let ((child-path (if (string= path "/")
                                                     (format nil "/~A" (car child))
                                                     (format nil "~A/~A" path (car child)))))
                                 (print-tree child-path (car child) (1+ depth))))))))
                (print-tree "/" "/" 0)))))

Compare this with the tree we wrote in Part 5. Back then, print-tree walked a graph of node objects directly. That worked when everything was a virtual node. It breaks the moment some children come from mounts, because mount entries aren’t nodes; they’re just (name . type) pairs.

So the recursion now travels by path string. At each step we carry the current path and the name we just printed. We use list-children to discover whatever lives there (virtual nodes, mount points, mounted directories), and for each child we build the next path by appending its name to the current one.

The (if (string= path "/") ...) branch is small but important. If path is the root /, gluing a child name with ~A/~A would yield //foo. We special-case the root and use /~A instead. This is the mirror image of what get-mount-children did with its prefix logic, just in reverse: we’re building paths now, not parsing them.

Notice the function no longer cares about mounts. It calls list-children and get-entry-type and lets them sort it out. The abstraction is doing its job.

Unified Find#

Same approach for find:

(register-command
 (make-command
  :name 'find
  :description "Find files by name: (find \"notes\")"
  :function (lambda (pattern)
              (let ((results '()))
                (labels ((dfs (path name)
                           (when (search pattern name)
                             (push path results))
                           (when (eq (get-entry-type path) :directory)
                             (dolist (child (list-children path))
                               (let ((child-path (if (string= path "/")
                                                     (format nil "/~A" (car child))
                                                     (format nil "~A/~A" path (car child)))))
                                 (dfs child-path (car child)))))))
                  (dfs "/" "/"))
                (if results
                    (dolist (path (nreverse results))
                      (format t "~%  ~A" path))
                    (format t "~%No matches for ~A" pattern))))))

The structure mirrors print-tree above. We descend by path string, use list-children to get the next level, and build child paths with the same root vs non-root split. The only difference is what we do at each node: instead of printing, we check whether the name matches the search pattern with search (a built-in that returns the position of the substring or nil), and collect matches in results.

push builds the list in reverse order (each new match goes to the front). nreverse flips it back at the end so matches print in traversal order. We saw the same in Part 5. It’s the cheap way to assemble a list when you don’t know the size in advance.

And like tree, find doesn’t know mounts exist. It just walks paths.

Running It#

Filesystem loaded from /path/to/repl/.repl-fs.dat
Welcome to REPL OS v7
Type (help) for commands, or any Lisp expression.

λ (ls)
  home/
  sys/

λ (tree)
/
  home/
    notes.txt
  sys/
    packages.lisp
    commands.lisp
    threads.lisp
    fs.lisp
    repl.lisp
    repl.asd

λ (find "lisp")
  /sys/packages.lisp
  /sys/commands.lisp
  /sys/threads.lisp
  /sys/fs.lisp
  /sys/repl.lisp

Everything is visible. ls "/" shows both home/ (virtual) and sys/ (mounted). tree descends into both. find searches everywhere.

Now the two worlds are unified.

What We Built#

  1. get-mount-children: detect mount points that appear as children of a path
  2. list-children: unified directory listing (virtual + mounted)
  3. get-entry-type: check if a path is a file, directory, or doesn’t exist
  4. get-entry-contents: read file contents regardless of storage
  5. Simplified commands: ls, cat, tree, find all use the unified API

The abstraction layer hides the complexity. Commands don’t know or care whether they’re accessing virtual nodes or real files. They just work with paths.

The Pattern#

This is a common pattern in systems programming: present a uniform interface over heterogeneous storage. Unix does this with VFS (Virtual File System). We did it with four functions.

The key is identifying the right operations. We needed listing, type checking, and content reading. Once those work uniformly, everything else falls into place.

Looking Back#

Seven chapters in, it’s worth pausing to see what you built.

Version 1 was (loop (print (eval (read)))) on steroids. You built a bare REPL with a prompt, error handling, and a clean way to quit. Twenty lines that already gave you a working language.

Version 2 added a command system. You built a registry, a struct, and a dispatcher, and the REPL stopped being a fixed set of behaviors. It became extensible. You could register new commands at runtime.

Version 3 brought concurrency. With Bordeaux Threads you spawned background tasks, scheduled work with after, repeated it with every, and managed threads with ps and kill. Your REPL learned to do more than one thing at a time.

Version 4 gave you storage. You built a virtual filesystem as a tree of nodes, with ls, mkdir, touch, cat, write, rm. Everything lived in RAM, gone the moment you quit.

Version 5 made it persistent. You serialized the tree to S-expressions, saved on quit, loaded on startup. You added tree and find, and used load-file to execute Lisp stored inside virtual files. An early taste of self-hosting.

Version 6 bridged worlds. You added mounts to map virtual paths to real directories, so the OS could read and write actual files, including its own source code.

Version 7, this one, made the split disappear. You built a small abstraction layer over paths, and now every command sees one unified filesystem, no matter where the bytes actually live.

A loop became a language. The language gained commands, then threads, then storage, then persistence, then access to the real world, and finally a clean view over all of it.

None of this is production software. It’s a sketch of an operating system, drawn in the fewest lines I could manage while still letting each piece teach something. But every concept here exists in real systems too. Filesystems have abstraction layers. Schedulers have registries. Threads have IDs. You built tiny versions of real ideas.

And it all fits inside a REPL. No kernel, no bootloader, no syscall interface. Just a language talking to itself, with you along for the ride.

The foundation is solid. Where you take it next is up to you, and there’s a lot of room.

A text editor would be the obvious first move. Editing files at the REPL prompt is brutal, you’ve felt it. A simple modal editor that reads keystrokes and writes back to the filesystem would change how the OS feels to use. After that, networking opens up a lot of ground: a curl command that fetches a URL into a virtual file, a ping to check if a host is alive, a small web server that listens on a port and serves files from any path, virtual or mounted. From there, permissions, users, a process supervisor, a package manager, a shell language. One after another, each compounding on the last, every addition making the REPL OS feel a little more like a real system.

Whatever you build, the pattern is the same. Identify the abstraction. Write the smallest version that captures it. Wire it into the REPL. Try it.

I may come back to this series at some point and add more chapters. I also have an old experiment of mine lying around that explored similar ideas in Node.js, and I’d like to revive it one of these days, to see how the same abstractions translate to a different language and runtime. We’ll see.

Now go write some Lisp. Happy hacking.