How to Run Untrusted Python Tests in a Sandbox Without Docker

If you need to run untrusted Python tests in a sandbox and Docker isn't an option, the working setup is: a fresh virtualenv per task, an environment with no secrets in it, Linux resource limits on CPU time and address space, a wall-clock timeout, low scheduler priority, and a kill that takes out the whole process group when time runs out. All of it is Python standard library plus two coreutils, and it runs on any Linux box you have a shell on.

That's what my pipeline does for code written by AI agents. Each task's tests run in a throwaway venv with a scrubbed environment, under nice 19 and ionice idle, with a 900 s wall timeout, a CPU-seconds cap, a 1536 MB address-space cap, and a process-group kill. If the suite can't even be collected, the result is UNVERIFIED, never a pass. The rest of this post goes through each piece with the code to do it.

What this kind of sandbox does and doesn't protect#

Be clear about the threat before you build anything. Resource limits and a clean environment contain accidents and limit blast radius: a test that loops forever, a fixture that allocates until the box swaps, a token that leaks into a log, a server process that outlives pytest. They do not stop code that deliberately reads files your worker user can read, or opens a socket to the internet. Without namespaces, the test process sees the same filesystem and network your worker sees.

So treat this as a seatbelt, not a vault. If you run tests from repos you don't control and you can't tolerate a read of your worker's home directory, put the worker under its own unprivileged user at minimum, and consider a real container. If Docker is already on the box, use it; the Docker Compose tutorial covers the basics of running an app in a container. This post is for the case where it isn't, and where the code under test was written by an agent a minute ago and mostly fails in boring ways.

Create a throwaway venv for every task#

A venv per task gives you two things: a clean dependency set, so one repo's pinned package can't break another's run, and a directory you can delete afterwards without thinking. Creating one costs a few seconds.

TASK=$(mktemp -d /tmp/task-XXXXXX)
mkdir -p "$TASK/home"
python3 -m venv "$TASK/venv"
"$TASK/venv/bin/pip" install --quiet pytest
"$TASK/venv/bin/pip" install --quiet -e "$REPO"

One thing people miss: pip install -e . runs the project's build backend, and for anything with a setup.py that is arbitrary code. The install step is already running untrusted code, so it has to run under the same limits and the same scrubbed environment as the tests themselves. Don't install with your worker's full environment and then sandbox only pytest.

When the run finishes, remove the task directory. If your worker crashes mid-run, stale directories pile up in /tmp, so a periodic sweep helps; a systemd timer unit pair is a simple way to schedule that cleanup.

Scrub the environment before the first process starts#

This is the cheapest fix with the biggest payoff, and it's the one I got wrong first. Before I added scrubbing, tests inherited the worker's environment, including the Claude token. Any test, or any conftest.py the agent wrote, could have read it with one line of os.environ.

The fix is an allowlist, not a denylist. Don't try to enumerate secret-looking names; start from an empty dict and add only what Python and pytest need to run.

import os

ALLOW = {"LANG", "LC_ALL", "TERM"}

def scrubbed_env(venv_bin, home):
    env = {k: v for k, v in os.environ.items() if k in ALLOW}
    env["PATH"] = venv_bin + os.pathsep + "/usr/bin:/bin"
    env["HOME"] = home
    env["VIRTUAL_ENV"] = os.path.dirname(venv_bin)
    env["PYTHONDONTWRITEBYTECODE"] = "1"
    return env

# Simulate a worker that has a secret in its environment.
os.environ["EXAMPLE_API_TOKEN"] = "this-must-not-reach-the-tests"

env = scrubbed_env("/tmp/task/venv/bin", "/tmp/task/home")
print("keys passed to tests:", sorted(env))
print("token leaked:", "EXAMPLE_API_TOKEN" in env)

Setting HOME to a directory inside the task folder matters too. Lots of tools write caches and config under ~, and with a fake home those writes land somewhere you'll delete anyway instead of in your worker's real home. PATH keeps /usr/bin so nice and ionice resolve, with the venv first so python and pytest are the task's own copies.

Cap CPU seconds, memory and wall time with rlimits#

Linux resource limits are per-process, inherited across fork and exec, and need no privileges to lower. Python exposes them through the resource module, and the details of each limit are in the setrlimit(2) man page. You set them in the child between fork and exec, which is exactly what subprocess.Popen offers through preexec_fn.

Three limits cover most of what goes wrong:

• RLIMIT_AS caps the virtual address space. I use 1536 MB. When a process tries to map past it, malloc fails and Python raises MemoryError instead of the box swapping to death. The catch: some libraries reserve large virtual ranges up front even when they touch little of it, and those will fail under a tight address-space cap even though their real memory use is small. If a suite dies with MemoryError at import time, that's the first thing to check.
• RLIMIT_CPU caps CPU seconds consumed. When it's hit the kernel sends SIGXCPU, then SIGKILL. This catches a hot loop quickly, independent of wall time.
• A wall-clock timeout in the parent, 900 s in my case, catches the opposite problem: a test that blocks on a socket or sleeps forever burns no CPU, so RLIMIT_CPU never fires.

You need both time limits. CPU seconds for spinning, wall seconds for hanging.

import resource

ADDRESS_SPACE = 1536 * 1024 * 1024
CPU_SECONDS = 600  # pick what your slowest honest suite needs

def limit_resources():
    resource.setrlimit(resource.RLIMIT_AS, (ADDRESS_SPACE, ADDRESS_SPACE))
    resource.setrlimit(resource.RLIMIT_CPU, (CPU_SECONDS, CPU_SECONDS))

If you'd rather limit real memory than address space, and you have root on a systemd box, systemd-run -p MemoryMax=1536M puts the command in its own cgroup with a hard memory ceiling. The systemd-run man page lists the properties it accepts. Address-space limits are the no-root, no-daemon option, which is why I use them.

Kill the whole process group, not just pytest#

The timeout argument on subprocess.run kills the child it started. It does not kill anything that child spawned. A test suite that starts a dev server, a database, or a multiprocessing pool and then times out leaves those running, holding ports and memory until someone notices.

The fix is start_new_session=True, which puts the child in a fresh session and process group whose ID equals the child's PID. On timeout you call os.killpg on that ID and everything the tests started goes with it. The subprocess docs describe both options; here they are together with the limits above.

import os
import signal
import subprocess

WALL_SECONDS = 900

def run_tests(repo_dir, venv_bin, env):
    cmd = [
        "nice", "-n", "19", "ionice", "-c", "3",
        os.path.join(venv_bin, "python"), "-m", "pytest", "-q",
    ]
    proc = subprocess.Popen(
        cmd,
        cwd=repo_dir,
        env=env,
        stdout=subprocess.PIPE,
        stderr=subprocess.STDOUT,
        start_new_session=True,
        preexec_fn=limit_resources,
    )
    try:
        out, _ = proc.communicate(timeout=WALL_SECONDS)
        return proc.returncode, out.decode(errors="replace")
    except subprocess.TimeoutExpired:
        os.killpg(proc.pid, signal.SIGKILL)
        out, _ = proc.communicate()
        return None, out.decode(errors="replace")

Two notes on this. preexec_fn runs in the child after fork and isn't safe if your worker is multithreaded; if it is, wrap the test command in a tiny launcher script that sets the limits and then execs pytest. And the limits set in preexec_fn apply to nice, which passes them through ionice to Python, because rlimits survive exec.

A returned None means the wall timeout fired. Treat it as its own status, not as a failure of the code under test. A hang in a test and a bug in the code are different facts.

Keep the box responsive with nice and ionice#

On a shared box the test run shouldn't compete with whatever else the machine does. nice -n 19 gives the process the lowest CPU priority, so it only gets cycles nothing else wants. ionice -c 3 puts it in the idle I/O class, so a suite that hammers the disk waits for everything else first.

Neither is a security measure. They're what keeps your web server and your agent loop snappy while a suite runs for fifteen minutes in the background. Both are inherited by children, so wrapping the pytest command once covers everything the suite spawns.

Report what you couldn't verify, and the green badge I shipped#

A sandbox that runs tests is half the job. The other half is being honest about what the run proved. Pytest exits 0 when every test passed and 1 when tests ran and some failed. Other codes mean the tests didn't really run: 2 covers interruption, which includes errors during collection, 3 is an internal error, 4 is a usage error, and 5 means nothing was collected. A None from the runner above means the wall timeout hit.

Only exit 0 is a pass. Exit 1 is a real failure. Everything else is a third state. In my pipeline that state is UNVERIFIED: the repo's suite couldn't be collected, or pytest never got to the tests, so I know nothing about whether the code works. Reporting that as a pass would be lying, and reporting it as a failure would send an agent off fixing code that might be fine.

The mistake I made was in displaying it. My first UI colored the status by checking whether the label contained the substring VERIFIED. TESTS UNVERIFIED contains VERIFIED, so it rendered in green. The runner was telling the truth and the badge was turning it into a lie. The fix is to match status values exactly, never by substring, and to make the unknown state its own color that nobody confuses with success.

def naive_color(label):
    return "green" if "VERIFIED" in label else "red"

COLORS = {
    "TESTS VERIFIED": "green",
    "TESTS FAILED": "red",
    "TESTS UNVERIFIED": "gray",
    "TESTS TIMED OUT": "gray",
}

def strict_color(label):
    return COLORS.get(label, "gray")

for label in COLORS:
    print(f"{label:17}  naive={naive_color(label):5}  strict={strict_color(label)}")

If you'd asked me beforehand, I'd have said the risky part of this system was the resource limits. It was a string comparison.

What I'd do differently from the start#

Scrub the environment on day one, before the first test ever runs. It's five lines and it's the difference between a leaked API token and nothing. And design the result type with three states before building any UI on top of it, so there's no label for a substring match to get wrong.

Checklist for sandboxing untrusted Python tests without Docker#

• Create a new venv per task in its own directory, and delete the directory when done.
• Run pip install of the repo under the same limits as the tests. Build backends are code.
• Build the child environment from an allowlist. Set HOME to a directory inside the task folder.
• Set RLIMIT_AS and RLIMIT_CPU in preexec_fn, or in a launcher script if your worker has threads.
• Enforce a wall-clock timeout in the parent. CPU limits don't catch hangs.
• Start the child with start_new_session=True and kill with os.killpg on timeout.
• Wrap the command in nice -n 19 ionice -c 3 so the box stays responsive.
• Map pytest exit codes to three states: pass, fail, unverified. Only exit 0 is a pass.
• Match status labels exactly in any UI. Never by substring.
• Remember what this doesn't do: no filesystem or network isolation. Use a dedicated user or a container if you need those.

Quick answers