Hardening a Docker-based agent: Read-only rootfs, non-root, and digest pinning

Canvass runs OpenFOAM on hardware the customer controls. That is the whole point: geometry, meshes, solver logs, and results stay on the customer's machine. The control plane coordinates work; it does not run the solver.

That architecture creates a sharp security boundary. The agent needs access to the host Docker daemon to start solver containers. On a standard Docker host, that socket is effectively root. A solver container that can reach the socket, mount arbitrary host paths, or write to the host filesystem is not a sandbox — it is a remote shell with extra steps.

This is a write-up of the container policy we use for solver and meshing jobs, the mistakes we made while tightening it, and the risks that remain.

Note

This describes the Canvass agent, not a general-purpose multi-tenant sandbox. The threat model is a compromised solver image, a malicious case, or a bad job payload against a customer-owned host. It is not a claim that Docker is a security boundary between mutually hostile tenants.

The threat model

The agent is a long-lived process on the customer host. It receives a job dispatch, prepares a case directory, starts a container, streams logs, and collects artifacts. The solver image is third-party OpenFOAM. The case files are customer-controlled. The job payload crosses the control plane.

Threat	What it could do	Control
Compromised solver image	Read host files, reach the network, persist	Read-only rootfs, non-root user, dropped capabilities, no network, no Docker socket
Malicious case files	Exploit the solver or the image entrypoint	Non-root user, read-only rootfs, no host mounts outside the case, resource limits
Bad job payload	Request a tag instead of a digest, mount the host, run as root	Server-side validation, digest pinning, mount allowlist, user validation
Compromised agent	Already has the Docker socket; game over for the host	Keep the agent small, avoid extra privileges, use rootless Docker where possible
Host kernel / Docker escape	Container breakout	Host patching, rootless Docker, seccomp/AppArmor, not a Canvass-specific fix

The important part is the first three rows. The agent cannot defend the host from a compromised Docker daemon or a kernel exploit. It can make the solver container boring: no root, no network, no host paths, no extra capabilities, and a pinned image.

One spec, one create path

Every solver and meshing container is built as a ContainerSpec and validated before Docker sees it. The same policy module checks both paths, so there is no "meshing is different" escape hatch.

interface ContainerSpec {
  jobId: string
  workspaceId: string
  leaseId: string
  agentId: string
  solver: string
  image: string
  imageDigest: string
  command: string[]
  workdir: string
  cpuLimit: number
  memoryLimitMb: number
  pidsLimit: number
  timeoutSeconds: number
  mounts: Mount[]
  environment: Record<string, string>
  labels: Record<string, string>
  user: string
  readOnlyRootfs: boolean
  capDrop: string[]
  networkMode: string
  privileged: boolean
}

The validator rejects the spec before create if any of these are wrong:

the image embeds a tag instead of a separate digest;
the command contains a shell metacharacter or a NUL byte;
the workdir is relative or contains ..;
the user is root or malformed;
the root filesystem is writable;
the network mode is not none;
a mount target is outside /case;
a mount source resolves outside the workspace root or names the Docker socket;
a required label is missing;
an environment key is on the denylist or starts with CANVASS_.

That last one matters. The agent's own environment contains credentials. A job payload must never be able to inject CANVASS_AGENT_TOKEN, LD_PRELOAD, or a loader variable into the container.

Digest pinning is not a tag check

A tag is mutable. opencfd/openfoam-run:latest can point at a different image tomorrow. Checking the tag once and then creating from the tag leaves a window where the daemon resolves something else.

The agent resolves the image to a repository digest, compares it with the expected digest, pulls the exact digest on a mismatch, and creates from image@sha256:<digest>. If the digest cannot be resolved, the job fails before create.

const digest = await daemon.imageDigest(image)
if (expectedDigest && digest !== expectedDigest) {
  await daemon.pull(`${image}@${expectedDigest}`)
}
await daemon.create({
  ...spec,
  image: `${image}@${digest}`,
})

No image-ID fallback. An image ID is local and mutable; it is not a registry identity. A job that cannot prove the image digest is a job that does not start.

Non-root is a policy, not a suggestion

The solver container runs as an explicit non-root UID:GID. The policy rejects 0:0 and rejects a bare UID without a GID.

For meshing, the work root has a dedicated owner. If the agent runs as root, it defaults the meshing user to 1000:1000 and transfers ownership of the dedicated mesh-work subtree to that user. It does not chown the agent state directory, and it does not accept the state root itself as the work root.

state/
  agent.json
  cases.json
  mesh-work/          <- chowned to the meshing user
    <job-id>/         <- mounted at /case

The work root must be under a mesh-work component, not merely under the state root. A misconfigured CANVASS_OPENFOAM_WORK_DIR=/ is rejected. A work root equal to the allowed root is rejected. The container gets one writable mount: the dedicated work directory at /case.

Read-only, no caps, no network

The container policy is deliberately small:

read_only: true for the root filesystem;
init: true so PID 1 reaps child processes;
cap_drop: ["ALL"];
security_opt: ["no-new-privileges:true"];
network_mode: "none";
pids_limit bounded;
memory and swap capped together;
bounded JSON-file logs;
CPU limit from the job request;
one writable mount under /case.

No Docker socket. No host home directory. No /var/run. No extra mounts.

Warning

The OpenFOAM image entrypoint still writes NSS files to /tmp. With a read-only rootfs it logs NWRAP_ERROR and Read-only file system warnings, then continues. blockMesh and checkMesh complete. We have not added a writable /tmp tmpfs yet; that is a known rough edge, not a silent success.

Meshing used to be the exception

The first version of the agent had a separate meshing launch path. It checked the digest, but it created the container with its own Docker kwargs. That meant meshing did not get init, a read-only rootfs, or the same user policy.

We moved meshing onto the same ContainerSpec and create path. The meshing runner still owns its stage watchdogs — castellation, snap, cell budget — but the container policy is shared. The regression tests assert the hardened create kwargs, and a real Docker smoke test runs both a solver-shaped and a meshing-shaped container.

The lesson is general: if a second code path can create a container, it will drift. Make the policy the only way to create one.

What we got wrong

A few real bugs surfaced while tightening this:

Blank environment values are not unset. CANVASS_MESH_RUN_AS= produced an empty user string, which failed validation. The fix treats blank/whitespace as unset and falls back to the agent UID/GID, or 1000:1000 for a root agent.
Allowed root equal to work root. Validating the mount source against the work root itself is a tautology. A work_root=/ or work_root=/var/run would pass. The fix requires a dedicated mesh-work subtree and a separate allowed root.
A started container could leak. If containers.get() failed after start, the cleanup path only removed a container object it never obtained. The fix retains the container ID and removes it through the daemon in finally.
Logs can be symlinks. The meshing container can write to /case. A compromised image could replace checkMesh.log with a symlink to a host file. The agent now rejects symlinked mesh logs and mesh files before reading or copying them.

None of these are exotic. They are the normal failure modes of a second launch path, a tautological check, and a cleanup path that only handles the happy case.

What still worries us

This is not a complete sandbox.

The Docker socket is root-equivalent. The agent needs it. Rootless Docker is preferred where the solver image supports it, but it is a deployment decision, not something the agent can enforce.
There is no per-job disk quota. A solver can fill the writable mount. Monitor the state volume and bound it at the host level.
No custom seccomp or AppArmor profile. Docker's defaults and the host's profile apply. A deployment that needs more should add one and test it against the OpenFOAM image.
Registry metadata is parsed before the digest is verified. A compromised registry or Docker daemon can influence pull behavior. Creation is still digest-pinned.
The host kernel is the final boundary. A kernel or Docker escape is outside the agent's control.

If you are running untrusted, multi-tenant workloads, use a stronger isolation boundary than a shared Docker daemon. This design is for a customer-owned agent running that customer's jobs on that customer's hardware.

Verification

At the time of writing, the agent suite is 382 passed, 3 skipped, with a real Docker smoke test that creates solver and meshing containers under the hardened policy. The smoke test checks the digest-pinned image reference, non-root user, read-only rootfs, init, dropped capabilities, no-new-privileges, no network, and the memory/swap and log caps.

The tests are not a proof. They are a regression fence around the policy, and they are the reason the meshing path no longer has its own rules.