Raphael De Lio

Software Engineer

Separating OpenClaw’s Compute from State with Redis Agent File System

•

•

9 min read

This blog post is also available as video. Watch it here.

As part of my first engagement as a Forward Deployed Engineer, I was assigned to a project where thousands of users would each receive an OpenClaw environment.

OpenClaw is a stateful application. It writes the information it needs between sessions to its home directory. This includes user instructions, skills, artifacts, configuration, session data, identity files, and SQLite databases.

Keeping one OpenClaw instance running for every user would preserve those files, but it would also reserve compute while most users were inactive. We needed to stop inactive environments and later give returning users access to the same state.

Starting with Kubernetes storage

The environments were going to run on Kubernetes, with OpenClaw running inside pods. The first storage option to consider was one PersistentVolume for each user, attached to the user’s pod through a PersistentVolumeClaim.

The volume would survive the pod, so a replacement pod could access the files. The lifecycle created two problems.

First, the cluster would need a PersistentVolume and PersistentVolumeClaim for every user workspace. At the expected scale, Kubernetes would have to manage a large number of storage objects.

Second, a pod declares its volumes when it is created. Kubernetes cannot replace one user’s PersistentVolumeClaim with another user’s claim while the pod remains running.

We could keep a pod and volume reserved for every user, which would retain the idle compute. We could also create a new pod for each active session, which would put pod scheduling, container startup, and storage attachment in the user’s request path.

The prototype tested another model: a shared pool of running pods with persistent workspaces that could be mounted and unmounted as users became active or inactive.

Defining disposable compute

This model separates the lifecycle of the OpenClaw process from the lifecycle of the user’s files.

The environment running OpenClaw can be replaced. The persistent workspace remains available independently from it. When the user returns, another environment mounts the workspace and starts a new OpenClaw process.

A request in progress, an open connection, a child process, and anything held only in memory disappear when the environment stops. The replacement process can reconstruct only the state that was persisted.

The prototype therefore had two requirements:

1. The workspace had to survive after an OpenClaw environment stopped.
2. A replacement OpenClaw process had to access that workspace through a regular filesystem path.

Introducing Redis Agent File System

Redis Agent File System, or AFS, is a filesystem layer backed by Redis. It stores directory structure, file metadata, and file contents in Redis, then exposes them through filesystem operations such as open, read, write, rename, and readdir.

These are POSIX style operations. That describes the interface presented to applications. It does not establish that every POSIX guarantee involving locking, atomicity, cache coherence, and durability has been validated.

The main unit in AFS is a workspace. Each workspace has an identifier and its own directory tree. The prototype assigned one workspace to each user.

user: raphael
workspace: ws_9f84653cd1b563af

workspace/
  AGENTS.md
  skills/
  artifacts/
  project/

The workspace ID tells AFS which filesystem to open. It does not prove that the requesting user owns the workspace. The application assigning environments still needs authentication, ownership checks, and scoped credentials.

AFS also supports checkpoints, workspace forks, file history, and configurable file versioning. It exposes an MCP server for agents that access files through tools. I used its filesystem mount because OpenClaw already expects local paths.

Mounting AFS through FUSE

The prototype ran on Google Kubernetes Engine. Each pod could be assigned to a user, released after inactivity, and reused for a later assignment.

AFS exposes a live Linux filesystem through FUSE. FUSE allows a user space process to implement a mounted filesystem. OpenClaw can continue using ordinary paths and file operations while the AFS process translates them into reads and writes against Redis.

The mount does not copy the complete workspace into the container before OpenClaw starts. When OpenClaw lists a directory or reads a file, the operation passes through FUSE. AFS retrieves the required metadata or content and can cache it locally.

Each pod uses an emptyDir as the local mountpoint:

securityContext:
  privileged: true
volumeMounts:
  - name: fuse
    mountPath: /dev/fuse
  - name: home
    mountPath: /home/node/.openclaw
volumes:
  - name: fuse
    hostPath:
      path: /dev/fuse
      type: CharDevice
  - name: home
    emptyDir: {}

The emptyDir contains the mountpoint and disposable local files. Persistent user files live in AFS. No Kubernetes PersistentVolume is attached for the user workspace.

The container needs access to /dev/fuse and permission to create the mount. The prototype uses a privileged container. This expands the pod’s security boundary and requires further evaluation for tenant isolation.

The resulting architecture

The system contains a pool control plane, an AFS control plane, OpenClaw pods, and two Redis Cloud databases.

The OpenClaw Pool Control Plane is the application built for this prototype. It assigns pods, tracks leases, and releases inactive environments.

The AFS Control Plane comes from the AFS project. It creates workspaces and manages their metadata. A separate Redis database stores the AFS filesystem data.

Each OpenClaw pod contains the OpenClaw runtime, the AFS mount executable, and a small mount manager. The mount manager remains running while the pod is available. OpenClaw starts only after a user is assigned and the workspace is ready.

The second Redis database stores pool state. The pods run in Kubernetes. Redis stores records describing their availability, assignments, and leases.

Reserving a pod

The Pool Control Plane must prevent two requests from reserving the same pod. The prototype uses a Redis List for the available queue, a Set to prevent duplicate queue entries, Hashes for pod and instance records, a Sorted Set for lease deadlines, and a Stream for lifecycle events.

An atomic Lua script removes a pod from the available queue and creates its assignment.

Each instance moves through this state machine:

AVAILABLE
  to ASSIGNING
  to ACTIVE
  to RELEASING
  to AVAILABLE

Lifecycle error
  to FAILED

The Redis reservation and the HTTP call to the mount manager are separate operations. Redis may reserve a pod even if the subsequent mount fails. The ASSIGNING and FAILED states make incomplete lifecycle operations visible to reconciliation.

The control plane periodically compares its Redis records with the Ready pods reported by Kubernetes. It removes stale queue entries and assignments after pod replacement or pool resizing.

Assigning an environment

After reserving a pod, the Pool Control Plane sends its mount manager a workspace ID and an instance ID:

{
  "workspaceId": "ws_example",
  "instanceId": "7d0c8dd0-2b94-4f53-88a1-3c62650ca835"
}

The mount manager performs the assignment in this order:

1. Lock the pod against another lifecycle operation.
2. Confirm that the pod is not already assigned.
3. Create the OpenClaw home mountpoint.
4. Start the AFS mount process for the selected workspace.
5. Wait for Linux to report that the mount is ready.
6. Start OpenClaw as an unprivileged user.
7. Wait for the OpenClaw readiness endpoint.
8. Record the identifiers and process information needed during release.

The AFS executable is built into the image from a pinned source revision. Its invocation is equivalent to:

agent-filesystem-mount \
  --redis "$AFS_REDIS_ADDR" \
  --user "$AFS_REDIS_USERNAME" \
  --password "$AFS_REDIS_PASSWORD" \
  --db "${AFS_REDIS_DB:-0}" \
  --allow-other \
  "$WORKSPACE_ID" \
  /home/node/.openclaw

Once the mount and OpenClaw readiness check succeed, the control plane marks the assignment as ACTIVE.

Releasing an environment

Release follows the reverse lifecycle. The mount manager verifies the instance ID, stops OpenClaw, persists the selected state, unmounts AFS, and removes disposable runtime files.

The Pool Control Plane returns the pod to the available queue only after cleanup succeeds. A failed unmount quarantines the pod so another user cannot inherit the previous workspace.

Each assignment has a lease deadline in the Redis Sorted Set. Activity extends the deadline. A scheduled process releases expired instances through the same cleanup path as an explicit release.

The activity signal must represent OpenClaw use. A browser heartbeat proves only that the browser remains connected. A deployment with a custom client should derive activity from authenticated OpenClaw requests or task execution.

Deciding what belongs in AFS

For the prototype, I mounted the complete ~/.openclaw directory through AFS. This made it possible to test whether OpenClaw’s file state could follow a user between pods.

The files do not all have the same persistence requirements.

Regular files are the clearest fit for AFS. SQLite requires more investigation. It depends on file locking, atomic rename, and fsync() behavior. Storing a SQLite database on the mount does not prove that AFS provides every guarantee SQLite expects during a pod crash, network interruption, or Redis failure.

The remaining SQLite state needs to be classified. Some data may be disposable or reconstructable. Data that must survive needs storage with the required locking and durability behavior.

Redis Agent Memory can replace OpenClaw’s long term and session memory capabilities. It does not replace all SQLite backed runtime state. Its coverage and recovery behavior need a separate evaluation.

The prototype also allows only one active assignment for a workspace. This reduces concurrent writers. It does not validate every cache coherence or failure scenario involving multiple mounts.

What the prototype demonstrated

I created files through one OpenClaw instance, released the environment, mounted the same workspace into another pod, and read the files again.

The prototype demonstrated that:

  1. A workspace can be mounted into an existing OpenClaw pod.
  2. OpenClaw can use the workspace through normal filesystem operations.
  3. Files survive release and reassignment to another pod.
  4. An inactive environment can be released automatically.
  5. A cleaned pod can serve another user.
  6. User workspaces do not require individual Kubernetes PersistentVolumes or PersistentVolumeClaims.

The test did not establish SQLite correctness, safe concurrent mounts, tenant isolation, or operation at the expected production scale. It also did not preserve active requests when a pod stopped.

What I will test next

The next performance tests should cover concurrent assignment and release across 100 or 1,000 pods. Measurements should separate AFS mount time, OpenClaw startup, control plane work, and release so each source of latency remains visible.

OpenClaw state should be classified by persistence requirement. SQLite needs tests covering locking, journal modes, atomic rename, and fsync() during pod, node, network, and Redis failures.

Redis Agent Memory should be evaluated against OpenClaw’s complete long term and session memory behavior. This includes retrieval quality, session continuity, migration, and recovery.

The security evaluation should cover user authentication, workspace ownership, scoped Redis credentials, cleanup between tenants, and whether privileged FUSE mounts provide an acceptable isolation boundary.