The problem
Reserved VM containers appear to use an overlayfs root mounted rw,relatime, with SOCI snapshots as the read-only lower layers and /mnt/scratch as the writable upper layer device.
When Python imports modules on a fresh container, the kernel appears to update each file’s access time (atime). On overlayfs, updating atime requires copying the entire file to the upper layer (copy-up). For a Django app with numpy/pandas this seems to amount to ~2.7 GB written to /mnt/scratch in the first 15 minutes.
When /mnt/scratch fills, we observed:
-
Root overlay writes failing with ENOSPC
-
/tmp(on the root overlay) stopping working -
Gunicorn uses
/tmpfor worker heartbeat files → workers appear dead → Gunicorn kills them and can’t restart -
Complete silence — process alive but serving nothing, no logs, no recovery
The trap: df -m / shows ~93 GB free the entire time. The virtual overlay counts lower SOCI layers, so the reported free space has no relationship to the physical capacity of /mnt/scratch. The scratch constraint is invisible to standard disk checks.
What we tried
Moving --worker-tmp-dir to /dev/shm, redirecting .pyc writes to /dev/shm/pycache, pre-copying .pythonlibs to a writable path before gunicorn starts, and adding pre-start and runtime scratch-device guards all help. But ongoing copy-up from SOCI-backed files appears to continue throughout the container’s lifetime.
The apparent fix
noatime on the root mount. No atime updates → no copy-up → scratch stays flat. We already attempt this on every startup:
mount -o remount,noatime /
It fails silently every time — the Reserved VM container doesn’t have CAP_SYS_ADMIN.
Request
Could Replit either mount the Reserved VM root with noatime by default, or grant CAP_SYS_ADMIN so deployment processes can remount themselves? We’re not 100% certain this is the full picture and would love input from anyone who knows the Reserved VM container internals better. Happy to share full logs and scratch growth timelines.