nixos-config/modules/nixos/linux/incus/incus-pet/SKILL.md
Sridhar Ratnakumar 776c4425b0 incus-pet: shared /nix/store, restart-to-deploy, drop anywhen flake input
The incus-pet CLI gains three big changes that together cut per-container
disk usage by ~5GB, drop nix-rebuild + nix-copy from the deploy path, and
make `deploy` actually pull latest on every invocation. anywhen migrates
from a host-installed service to an incus-pet container in the process.

* Shared /nix/store mount. Each container gets host's /nix/store
  bind-mounted read-only (with shift=true for idmapped unprivileged
  containers). Zero per-container duplication of nixpkgs closures. The
  CLI builds toplevels on the host; the container sees them via the
  shared mount, no `nix copy` needed.

* Restart-to-deploy. switch-to-configuration is removed entirely (it
  was stalling on dbus-broker reload and hitting the same lock-
  contention bug across redeploys). New activation: update
  /nix/var/nix/profiles/system, repoint /sbin/init through the profile
  symlink, `incus restart`. The new toplevel's own init script handles
  /etc setup on boot. Trade-off: ~5s of container reboot per deploy.

* Marshaling flake exposes only nixosModules.default. No nixpkgs
  input, no nixosConfigurations entry. The CLI picks nixpkgs at build
  time (pinned to nixos-25.11 — matches the LXC image baseline, dodges
  unstable regressions) and assembles the full nixosSystem via
  `nix build --impure --expr`. Lets us bump nixpkgs without touching
  per-deploy state.

* `nix flake update --refresh` on every deploy. Was `nix flake lock`,
  which only locks first-seen inputs — re-deploying `github:srid/anywhen`
  stayed pinned at the first commit ever resolved. Now every deploy
  bumps the target input to the current tip of its ref and bypasses
  nix's fetcher cache.

* /sbin/init repoint. The NixOS LXC image hard-links /sbin/init to its
  own toplevel's init path. Without switch-to-configuration, `incus
  restart` re-execs that old init, which sets up /etc from the old
  toplevel — running services keep using the OLD anywhen binary even
  though the profile pointer moved. CLI now also does
  `ln -sfn /nix/var/nix/profiles/system/init /sbin/init` so the profile
  actually drives boot. Idempotent.

* Host gcroot via `nix-store --add-root --indirect`. Container's nix DB
  doesn't know about the path (it lives behind the shared mount), so
  without a host-side gcroot the host could nix-collect-garbage the
  running system out from under the container. Symlink at
  ~/.local/state/incus-pet/<name>/system. Only the LATEST toplevel is
  pinned per app — no rollback today (file a 5-line ring buffer when
  rollback matters).

* anywhen flake input removed. `modules/nixos/linux/anywhen.nix` is
  gone; the host config no longer imports anywhen as a flake input.
  The deploy command takes the flake ref as an argument, so locking it
  here was just bloat. `just pureintent anywhen-deploy` runs the
  current invocation.

* pureintent ops recipes. New `configurations/nixos/pureintent/mod.just`
  with anywhen-{deploy,rm,backup,restore,status,shell}. Namespaced via
  `mod pureintent` so each host's recipes live behind their own prefix.

* Drop ssh from deploy path. Bootstrap still pushes the operator's
  pubkey (for interactive debugging), but activation goes through
  `incus exec` exclusively. No NIX_SSHOPTS, no nix-copy-via-ssh,
  no host-key TOFU dance during deploys.

Tested end-to-end on pureintent: anywhen migrated from host-installed
(21-task SQLite DB at /var/lib/anywhen) to an incus-pet container with
the DB intact; hello-web deployed fresh with shared /nix/store; both
serve over the tailscale proxy at 100.122.32.106:{6111,8081}; lock
bumps + new toplevels confirmed after master advances.
2026-05-23 18:50:10 -04:00

5 KiB

name description argument-hint
incus-pet Deploy a NixOS service flake into a per-app incus container ("pet PaaS") on the local host — OR add the required nixosModules.incus contract to a flake so it becomes deployable. Use when the user says "deploy X", "make this app deployable", "add incus to this flake", or "publish this on my box". deploy <flake-ref> [<name>] [--port N] [--listen IP]

incus-pet

A per-app incus container deploys a flake-shipped NixOS service onto the local host, exposed on a chosen <listen-ip>:<host-port> that proxies to a fixed 8080 inside the container.

Pick the branch from context:

  • User in their OWN app's repo, "make this deployable" → Branch C
  • User wants to add the contract to an UPSTREAM they can PR to (e.g. juspay/kolu) — agent has push/PR access → Branch C
  • Flake-ref already ships nixosModules.incusBranch A
  • Upstream won't accept a PR and user explicitly wants a local wrapper → Branch B

Branch A — deploy a flake that ships the contract

Run:

incus-pet deploy <flake-ref> [<name>] [--port N] [--listen IP]

--port is required on the first deploy of a given <name> and recorded in container metadata; subsequent deploys read it back. Report the <listen>:<port> URL to the user.

Branch B — local stand-alone wrapper (RARE; Branch C is preferred)

Only when upstream won't accept the contract. Create a tiny wrapper flake locally:

  • flake.nix exposing nixosModules.incus per the contract
  • service-module.nix authoring services.<app> as a NixOS system module (mirror the upstream's home-manager module if any)
  • Add the upstream as a flake input

Then incus-pet deploy ./path/to/wrapper <app>.

Branch C — add the contract to a flake (yours or upstream-via-PR)

Steps:

  1. Verify the flake exposes packages.<system>.default. If not, stop — prerequisite is a package derivation with meta.mainProgram.

  2. Verify nixosModules.default (the service module). If MISSING but the flake has homeManagerModules.default (kolu case):

    • Author nix/nixos/module.nix as a system NixOS module mirroring the home-manager surface — same options (enable, package, host, port, ...), emitting systemd.services.<app> instead of a home-manager unit.
    • Expose as nixosModules.default in flake outputs.

    If MISSING entirely (no module at all), stop — prerequisite.

  3. Add nixosModules.incus to outputs (fill in <app>):

    nixosModules.incus = { config, lib, pkgs, ... }: {
      imports = [ self.nixosModules.default ];
      incus.container = {
        enable   = true;
        hostname = lib.mkDefault "<app>";
      };
      system.stateVersion = "25.05";
      services.<app> = {
        enable  = true;
        package = lib.mkDefault
          self.packages.${pkgs.stdenv.hostPlatform.system}.default;
        host    = lib.mkDefault "0.0.0.0";
        port    = 8080;  # fixed by the incus-pet contract
      };
    };
    
  4. If services.<app> doesn't expose {host, port, package} (or equivalents), surface as prerequisite. Do not invent options that aren't there — add them to the service module first.

  5. Smoke-test:

    nix flake check
    nix eval .#nixosModules.incus
    
  6. If upstream (PR mode), open the PR. Else commit.

  7. Tell the user how to deploy:

    incus-pet deploy github:<owner>/<repo> --port <chosen> --listen <ip>
    

Do NOT add example/-style sub-flakes or VM tests in this branch — separate scope.

Contract for nixosModules.incus

The module must set:

  • incus.container.enable = true
  • incus.container.hostname = "<app>" (mkDefault ok)
  • system.stateVersion = "25.05"
  • services.<app>.{enable, package, host, port} (port = 8080, hardcoded — this is the incus-pet convention)

The module must NOT set:

  • services.<app>.port to anything other than 8080
  • networking.firewall.allowedTCPPortscontainer.nix opens 8080

Idempotence + failure modes

  • Re-running incus-pet deploy <flake-ref> against an existing container is safe: marshaling flake is rewritten and re-locked; container bootstrap is idempotent; proxy device is set if present, else add.
  • Container metadata (user.incus-pet.{host-port,listen,flake-ref}) is the source of truth for the operator-chosen exposure. incus-pet list filters by the flake-ref key.
  • If --port is missing on the first deploy, the command fails before launching anything.
  • Activation is two ssh steps: nix-env --set (atomic pointer swap on /nix/var/nix/profiles/system) then switch-to-configuration switch. If the second fails partway, the system profile is already at the new toplevel; running deploy again finishes the switch.
  • Exit 4 from switch-to-configuration is treated as success (a unit failed to reload — typically dbus-broker — but activation itself completed).