Nix Store Sharing in Containers Risks and GC Concerns

by

Summary

Sharing the Nix store (/nix) between containers can eliminate repeated package downloads and speed up workflows, but it introduces security, reproducibility, and garbage‑collection concerns that must be managed carefully.

Root Cause

  • The Nix store is designed as an immutable, content‑addressed database of build outputs.
  • Containers that write to /nix can corrupt the store’s immutability guarantees.
  • Simultaneous read/write access leads to race conditions on the underlying filesystem.
  • Garbage collection (nix-collect-garbage) may delete packages still in use by another container.

Why This Happens in Real Systems

  • Nix expects a single, trusted nix-daemon managing the store; multiple containers bypass this mediation.
  • OverlayFS or bind‑mounts used for container isolation expose the store as a regular directory, allowing accidental writes.
  • The store’s hard‑link‑based layout assumes stable inodes; container layering can break those links.
  • CI/CTF workflows often run privileged containers, increasing the chance of unintended modifications.

Real-World Impact

  • Non‑deterministic builds if a package is altered or removed mid‑process.
  • Security exposure: a compromised container could inject malicious binaries into the shared store.
  • Cache thrash: frequent garbage collection forces re‑downloads, negating the intended benefit.
  • Debugging complexity: errors appear only when containers intersect, making root cause hard to trace.

Example or Code (if necessary and relevant)

docker run -v /host/nix:/nix:ro -it mynfctf-image

How Senior Engineers Fix It

  • Mount /nix read‑only (:ro) to prevent accidental writes.
  • Use a nix daemon running on the host and configure containers to talk to it via UNIX socket.
  • Leverage nix‑cache or binary caches (e.g., Cachix) to share pre‑built artifacts without exposing the store.
  • Isolate the store per‑container with copy‑on‑write snapshots (e.g., using btrfs subvolumes or zfs clones) when write access is truly needed.
  • Schedule garbage collection only during host maintenance windows, ensuring no container relies on the store at that moment.

Why Juniors Miss It

  • Assuming that a shared directory is inherently safe for any workload.
  • Overlooking the immutability model of Nix and treating it like a regular package cache.
  • Not recognizing the risks of concurrent garbage collection in multi‑tenant environments.
  • Failing to differentiate between read‑only sharing and read‑write sharing when designing container volumes.

Leave a Comment