Skip to content

Isolate Cgroup v2 + Docker: Problems, Fixes & Learnings

This document records the full journey of fixing Isolate 2.x cgroup initialization inside privileged Docker containers. It is intended as a reference for future debugging and maintenance of the Showdown sandbox execution pipeline.

Background

Showdown uses Isolate (v2.7) to run untrusted code in sandboxed processes with resource limits (CPU, memory, time).

Isolate 2.x uses cgroup v2 and requires a helper daemon, isolate-cg-keeper, to:

  1. Detect its own cgroup (via /proc/self/cgroup).
  2. Create a daemon sub-cgroup and move itself into it.
  3. Enable controllers (+cpuset +memory) on its parent cgroup via cgroup.subtree_control.
  4. Write the resulting cgroup root path to /run/isolate/cgroup.

The isolate binary, when invoked with --cg, reads cg_root = auto:/run/isolate/cgroup from its config, which tells it to read the path from that file. It then creates box-<id> sub-cgroups under that root for each sandbox.

On a normal Linux host with systemd, isolate-cg-keeper is started as a systemd service (isolate.service) inside isolate.slice, which provides a pre-delegated cgroup. Docker containers do not run systemd as PID 1, so this entire mechanism is absent.

Problems Encountered (in order)

Problem 1: isolate-cg-keeper is not currently running

Symptom: 

This error occurs because isolate-cg-keeper is not currently running.

Cause: No mechanism in the Docker image started isolate-cg-keeper. On a host, systemd starts it via isolate.service. In a container, there is no systemd.

Fix (intermediate): Added an entrypoint script to the Compilers base Docker image that starts isolate-cg-keeper as a background daemon before the main process runs. This was later replaced by a direct cgroup setup approach (see The Final Fix).

Problem 2: mkdir: cannot create directory '/run/isolate': Permission denied

Symptom: 

mkdir: cannot create directory '/run/isolate': Permission denied

Cause: The entrypoint was running as the showdown user (non-root, set via USER showdown in Worker/Standalone Dockerfiles). The mkdir -p /run/isolate/locks command requires root because /run is owned by root.

Fix: Removed the mkdir from the entrypoint entirely. The directory creation is handled by the programs themselves running as root: - isolate-cg-keeper creates /run/isolate via make_dir_for() before writing /run/isolate/cgroup. - isolate creates /run/isolate/locks via make_dir() when it runs.

Learning: Don't duplicate in shell what the C programs already do internally. Read the source code of the tools you're wrapping.

Problem 3: sudo: a terminal is required to read the password

Symptom: 

sudo: a terminal is required to read the password; either use the -S option
to read from standard input or configure an askpass helper
sudo: a password is required

Cause: The entrypoint tried to run sudo mkdir and sudo isolate-cg-keeper, but only specific commands were in the sudoers allowlist (NOPASSWD). sudo mkdir was not allowed, so sudo prompted for a password — which fails in a non-interactive container.

Fix: Split the entrypoint into two scripts: 1. isolate-cg-setup.sh — the root-level setup script (added to sudoers). 2. isolate-entrypoint.sh — the Docker ENTRYPOINT, which calls the setup script via sudo (if non-root) or directly (if root), then exec "$@" to run the original CMD.

This way only one command (isolate-cg-setup.sh) needs to be in sudoers, and all root operations inside it run without further sudo calls.

Learning: When using sudoers with specific command allowlists, run all privileged operations inside a single allowlisted script rather than calling sudo on individual commands.

Problem 4: Cannot write to /sys/fs/cgroup//cgroup.subtree_control: Device or resource busy

Symptom: 

Cannot write to /sys/fs/cgroup//cgroup.subtree_control: Device or resource busy

Note the double slash (/sys/fs/cgroup//) — a path construction artifact.

Cause: In a privileged Docker container, /proc/self/cgroup reports 0::/ (the root cgroup). The keeper's get_my_cgroup() function reads this and constructs /sys/fs/cgroup/ (with trailing slash). When it appends cgroup.subtree_control, the path becomes /sys/fs/cgroup//cgroup.subtree_control.

The EBUSY error occurs because of the cgroup v2 "no internal processes" rule: a cgroup cannot have controllers enabled in its cgroup.subtree_control if it directly contains running processes. The root cgroup (/sys/fs/cgroup) has all container processes in it, so writing +cpuset +memory fails.

Fix (first attempt): Moved all processes from root into /sys/fs/cgroup/init, then enabled controllers on the now-empty root. This fixed the EBUSY on root.

Learning: The double slash was a symptom, not the bug. The real issue was the cgroup v2 no-internal-processes rule. Always understand the kernel constraint behind the error code.

Problem 5: Cannot write to /sys/fs/cgroup/init/box-0/memory.max: No such file or directory

Symptom: 

Cannot write /sys/fs/cgroup/init/box-0/memory.max: No such file or directory

Cause: After moving all processes to /sys/fs/cgroup/init, the keeper's /proc/self/cgroup now reported /init. It wrote /sys/fs/cgroup/init as cg_root to /run/isolate/cgroup. Isolate then created box-0 under /sys/fs/cgroup/init/box-0.

But memory.max didn't exist there because the memory controller was never enabled on /sys/fs/cgroup/init. The keeper tried to enable it (echo "+cpuset +memory" > /sys/fs/cgroup/init/cgroup.subtree_control) but that also failed with EBUSY — because init contained the keeper process itself (and all other container processes).

This was a chicken-and-egg problem: - init has processes → can't enable controllers - Can't enable controllers → box-0/memory.max doesn't exist - Can't move processes out of init because the keeper itself is in init

Problem 6: Cannot write to /sys/fs/cgroup/init/cgroup.subtree_control: Device or resource busy

Symptom: 

Cannot write to /sys/fs/cgroup/init/cgroup.subtree_control: Device or resource busy

Cause: Same root cause as Problem 5. The keeper was running inside /sys/fs/cgroup/init, so init had processes in it, so it could not enable controllers on itself.

The fundamental insight: The cgroup where cgroup.subtree_control is enabled must be empty of processes. The keeper must run in a sibling cgroup, not the same one where it enables controllers.

Problem 7: Keeper fails on server — stray processes in isolate/

Symptom (server only, worked locally): 

cat /sys/fs/cgroup/isolate/cgroup.subtree_control  → empty
cat /sys/fs/cgroup/cgroup.subtree_control          → cpuset cpu memory

Root controllers were enabled correctly, but the keeper failed to enable controllers on isolate/. The isolate/cgroup.subtree_control was empty.

Cause: The subshell that launched the keeper left a stray bash process in /sys/fs/cgroup/isolate. Even with exec, the timing of PID writes to cgroup.procs and the keeper's own setup_cg() created a race condition where isolate/ was not empty when the keeper tried to enable controllers. The keeper moved itself to isolate/daemon/, but the parent subshell process remained in isolate/, causing EBUSY.

This worked locally but failed on the server due to different process scheduling timing — a classic race condition that only manifests under different load conditions.

The Final Fix: Skip the Keeper Entirely

The breakthrough insight: isolate doesn't need the keeper to be running. The keeper is just a setup daemon that: 1. Detects its cgroup via /proc/self/cgroup 2. Writes that path to /run/isolate/cgroup 3. Creates a daemon/ sub-cgroup, moves itself there 4. Enables controllers on its parent 5. Sleeps forever (for (;;) pause())

Isolate itself only reads /run/isolate/cgroup (via cg_root = auto:...) and creates box-<id> sub-cgroups under that path. It never communicates with the keeper at runtime.

Since we already know the cgroup path (/sys/fs/cgroup/isolate) and the directory is empty when we create it, we can do all the setup directly in our script — no keeper daemon needed, no PID race conditions, no subshell/exec complexity.

The cgroup hierarchy

/sys/fs/cgroup/                  <-- root (controllers enabled here)
├── cgroup.subtree_control       <-- "+memory +cpu +cpuset"
├── app/                         <-- all container processes (entrypoint, server)
│   └── cgroup.procs
└── isolate/                     <-- cg_root (written to /run/isolate/cgroup)
    ├── cgroup.subtree_control   <-- "+cpuset +memory" (enabled by our script)
    └── box-0/                   <-- isolate creates sandboxes here
        └── memory.max           <-- exists because isolate/ enabled +memory

Note: no daemon/ sub-cgroup is needed because we don't run the keeper.

Step-by-step setup script (isolate-cg-setup.sh)

  1. Move all processes from /sys/fs/cgroup/cgroup.procs into /sys/fs/cgroup/app/cgroup.procs. Root is now empty.

  2. Enable controllers at root: echo "+memory +cpu +cpuset" > /sys/fs/cgroup/cgroup.subtree_control. Succeeds because root has no processes.

  3. Create /sys/fs/cgroup/isolate — an empty cgroup, no processes in it.

  4. Enable controllers on isolate/: echo "+cpuset +memory" > /sys/fs/cgroup/isolate/cgroup.subtree_control. Succeeds because isolate/ is empty (we just created it, nothing moved into it).

  5. Write /sys/fs/cgroup/isolate to /run/isolate/cgroup — this is the file isolate reads via cg_root = auto:/run/isolate/cgroup.

  6. Create /run/isolate/locks — pre-create the lock directory.

  7. Isolate reads /run/isolate/cgroup, creates box-0 under /sys/fs/cgroup/isolate/box-0. memory.max exists because isolate/ has the memory controller enabled.

Why this is better than running the keeper

  • No race conditions: We don't need to move any process into isolate/, so there's no risk of stray processes blocking controller enablement.
  • No daemon needed: The keeper's only purpose was setup; it sleeps forever after that. We do the setup and exit.
  • Deterministic: The same commands run the same way every time, regardless of server load or process scheduling.
  • Simpler: No subshell, no exec, no $BASHPID, no waiting/polling loop.

Dockerfile changes

docker/Compilers (base image): - Added isolate-cg-setup.sh — the root-level cgroup setup script (no keeper). - Added isolate-entrypoint.sh — calls setup (via sudo if non-root), then exec "$@". - Set ENTRYPOINT ["/usr/local/bin/isolate-entrypoint.sh"].

docker/Worker and docker/Standalone: - Added isolate-cg-setup.sh to sudoers so the showdown user can run it passwordlessly.

docker/Dev: - No changes needed (already has NOPASSWD: ALL).

docker/Manager: - No changes needed (doesn't use isolate).

Thought Process & Key Learnings

1. Read the source code of the tools you depend on

Understanding isolate-cg-keeper.c and cg.c was essential. Key findings: - get_my_cgroup() reads /proc/self/cgroup and constructs the path. - setup_cg() creates a daemon sub-cgroup, moves itself there, then enables controllers on the parent. - The auto: config mode means "read cg_root from a file written by the keeper." - Isolate itself only reads the file — it doesn't communicate with the keeper at runtime. This realization allowed us to skip the keeper entirely.

Without reading the source, we would have kept guessing at shell-level fixes.

2. Understand the kernel constraint behind the error

EBUSY on cgroup.subtree_control is not a permissions issue — it's the cgroup v2 "no internal processes" rule. A cgroup that has processes directly attached cannot delegate controllers to children. This is a kernel-level constraint, not a configuration issue.

3. The cgroup hierarchy must be planned, not improvised

The failed approaches all tried to use a single cgroup for both processes and controller delegation. The correct approach separates them: - Process cgroup (app/) — contains running processes, no subtree control. - Delegation cgroup (isolate/) — empty of processes, has controllers enabled, children (boxes) inherit the controllers.

4. Docker containers are not full Linux systems

Systemd services, cgroup delegation, and other host-level infrastructure don't exist in containers. Any solution that depends on them must be replaced with an equivalent container-level mechanism (entrypoint scripts, manual cgroup setup).

5. Sudoers in containers: use a single allowlisted script

Instead of adding many individual commands to sudoers, wrap all privileged operations in one script and allowlist that script. This is simpler, more secure, and avoids the "terminal required for password" error when sudo is called on non-allowlisted commands.

6. The double slash was a red herring

/sys/fs/cgroup//cgroup.subtree_control looked like a path construction bug, but it was just /proc/self/cgroup reporting 0::/ (root cgroup with trailing slash). The real issue was EBUSY from the no-internal-processes rule, not the path format.

7. Race conditions manifest differently on different machines

The keeper approach worked locally but failed on the server due to process scheduling differences. When a solution involves moving processes between cgroups and then enabling controllers, there's an inherent race: the process must be fully moved out before controllers can be enabled. Different CPU scheduling on the server caused the keeper to attempt enabling controllers before all processes were moved out.

Lesson: If a setup step has a race condition, eliminate the race rather than trying to fix the timing. In our case, we eliminated the race by not putting any processes in isolate/ at all.

8. Don't run a daemon if you don't need one

The keeper is a daemon that sets up cgroups and then sleeps forever. If we can do the setup ourselves and the daemon provides no runtime service, we should skip it. Fewer moving parts = fewer failure modes.

9. Iterative debugging in containers

Each fix revealed the next problem. The progression was: 1. Keeper not running → start it in entrypoint 2. Permission denied on mkdir → remove redundant mkdir 3. Sudo password required → use allowlisted script 4. EBUSY on root → move processes out of root 5. EBUSY on init → keeper was in init 6. memory.max missing → controllers never enabled on the right cgroup 7. EBUSY on isolate/ (server only) → stray subshell process race condition 8. Final fix: skip the keeper, set up cgroups directly

This is typical of cgroup debugging — each layer reveals the next constraint.

Files Modified

File Change
docker/Compilers Added isolate-cg-setup.sh, isolate-entrypoint.sh, and ENTRYPOINT
docker/Worker Added isolate-cg-setup.sh to sudoers
docker/Standalone Added isolate-cg-setup.sh to sudoers

Requirements for the Fix to Work

  • Container must be started with --privileged (already the case in docker-compose.yml for standalone, worker, and dev).
  • The cgroup v2 filesystem must be mounted at /sys/fs/cgroup (default in privileged containers).