mirror of
https://github.com/srid/nixos-config.git
synced 2026-07-16 22:01:33 +08:00
incus-pet: shared /nix/store, restart-to-deploy, drop anywhen flake input
The incus-pet CLI gains three big changes that together cut per-container
disk usage by ~5GB, drop nix-rebuild + nix-copy from the deploy path, and
make `deploy` actually pull latest on every invocation. anywhen migrates
from a host-installed service to an incus-pet container in the process.
* Shared /nix/store mount. Each container gets host's /nix/store
bind-mounted read-only (with shift=true for idmapped unprivileged
containers). Zero per-container duplication of nixpkgs closures. The
CLI builds toplevels on the host; the container sees them via the
shared mount, no `nix copy` needed.
* Restart-to-deploy. switch-to-configuration is removed entirely (it
was stalling on dbus-broker reload and hitting the same lock-
contention bug across redeploys). New activation: update
/nix/var/nix/profiles/system, repoint /sbin/init through the profile
symlink, `incus restart`. The new toplevel's own init script handles
/etc setup on boot. Trade-off: ~5s of container reboot per deploy.
* Marshaling flake exposes only nixosModules.default. No nixpkgs
input, no nixosConfigurations entry. The CLI picks nixpkgs at build
time (pinned to nixos-25.11 — matches the LXC image baseline, dodges
unstable regressions) and assembles the full nixosSystem via
`nix build --impure --expr`. Lets us bump nixpkgs without touching
per-deploy state.
* `nix flake update --refresh` on every deploy. Was `nix flake lock`,
which only locks first-seen inputs — re-deploying `github:srid/anywhen`
stayed pinned at the first commit ever resolved. Now every deploy
bumps the target input to the current tip of its ref and bypasses
nix's fetcher cache.
* /sbin/init repoint. The NixOS LXC image hard-links /sbin/init to its
own toplevel's init path. Without switch-to-configuration, `incus
restart` re-execs that old init, which sets up /etc from the old
toplevel — running services keep using the OLD anywhen binary even
though the profile pointer moved. CLI now also does
`ln -sfn /nix/var/nix/profiles/system/init /sbin/init` so the profile
actually drives boot. Idempotent.
* Host gcroot via `nix-store --add-root --indirect`. Container's nix DB
doesn't know about the path (it lives behind the shared mount), so
without a host-side gcroot the host could nix-collect-garbage the
running system out from under the container. Symlink at
~/.local/state/incus-pet/<name>/system. Only the LATEST toplevel is
pinned per app — no rollback today (file a 5-line ring buffer when
rollback matters).
* anywhen flake input removed. `modules/nixos/linux/anywhen.nix` is
gone; the host config no longer imports anywhen as a flake input.
The deploy command takes the flake ref as an argument, so locking it
here was just bloat. `just pureintent anywhen-deploy` runs the
current invocation.
* pureintent ops recipes. New `configurations/nixos/pureintent/mod.just`
with anywhen-{deploy,rm,backup,restore,status,shell}. Namespaced via
`mod pureintent` so each host's recipes live behind their own prefix.
* Drop ssh from deploy path. Bootstrap still pushes the operator's
pubkey (for interactive debugging), but activation goes through
`incus exec` exclusively. No NIX_SSHOPTS, no nix-copy-via-ssh,
no host-key TOFU dance during deploys.
Tested end-to-end on pureintent: anywhen migrated from host-installed
(21-task SQLite DB at /var/lib/anywhen) to an incus-pet container with
the DB intact; hello-web deployed fresh with shared /nix/store; both
serve over the tailscale proxy at 100.122.32.106:{6111,8081}; lock
bumps + new toplevels confirmed after master advances.
This commit is contained in:
Sridhar Ratnakumar
parent
5997c1c848
commit
776c4425b0
9 changed files with 271 additions and 249 deletions
|
|
@ -1,16 +0,0 @@
|
|||
{ flake, pkgs, ... }:
|
||||
|
||||
let
|
||||
inherit (flake) inputs;
|
||||
in
|
||||
{
|
||||
imports = [
|
||||
inputs.anywhen.nixosModules.default
|
||||
];
|
||||
|
||||
services.anywhen = {
|
||||
enable = true;
|
||||
package = inputs.anywhen.packages.${pkgs.stdenv.hostPlatform.system}.default;
|
||||
port = 6111;
|
||||
};
|
||||
}
|
||||
|
|
@ -117,13 +117,22 @@ incus-pet deploy github:srid/anywhen --port 7700 --listen 100.122.32.106
|
|||
|
||||
This:
|
||||
|
||||
1. Synthesises a marshaling flake under `~/.local/state/incus-pet/anywhen/`.
|
||||
2. Launches `images:nixos/25.11` as container `anywhen` (with
|
||||
`security.nesting=true` so `nixos-rebuild` works inside).
|
||||
3. Bootstraps sshd + your pubkey via `incus exec`.
|
||||
4. Runs `nixos-rebuild switch --target-host root@<container-ip>`.
|
||||
5. Records `--port` and `--listen` in container metadata.
|
||||
6. Adds the `web` proxy device.
|
||||
1. Synthesises a marshaling flake under `~/.local/state/incus-pet/anywhen/`
|
||||
exposing `nixosModules.default` (essentials + app's
|
||||
`nixosModules.incus` + operator pubkey overlay) — no nixpkgs pin in
|
||||
per-deploy state.
|
||||
2. Launches `images:nixos/25.11` as container `anywhen` with
|
||||
`-c security.nesting=true`.
|
||||
3. Pushes your `~/.ssh/id_ed25519.pub` into the container's
|
||||
`/root/.ssh/authorized_keys` (the image already runs sshd via systemd
|
||||
socket activation).
|
||||
4. Builds the system toplevel against `github:nixos/nixpkgs/nixos-25.11`
|
||||
(stable, matching the image baseline) via `nix build --impure --expr`.
|
||||
5. Copies the closure with `nix copy --to ssh://root@<container-ip>
|
||||
--substitute-on-destination`.
|
||||
6. SSHes to set the system profile and run `switch-to-configuration switch`.
|
||||
7. Records `--port` and `--listen` in container metadata.
|
||||
8. Adds the `web` proxy device.
|
||||
|
||||
### Subsequent deploys
|
||||
|
||||
|
|
@ -165,9 +174,13 @@ container.
|
|||
- **`Failed to start Firewall.` inside the container** — the launch
|
||||
flag `-c security.nesting=true` is supposed to handle this; if it
|
||||
recurs, see `../README.md` and lxc/incus#526.
|
||||
- **SSH fails after first deploy** — the bootstrap step writes
|
||||
`/etc/nixos/incus-pet-bootstrap.nix` and runs `nixos-rebuild` inside
|
||||
the container; if that fails, `incus exec <name> -- journalctl -xeu nixos-rebuild` is the place to look.
|
||||
- **`switch-to-configuration` exits 4** — "some units failed to
|
||||
reload/restart". Treated as success: the activation itself
|
||||
completed, only a unit reload (typically dbus-broker) failed. The
|
||||
service is up.
|
||||
- **`switch-to-configuration` exits 11 / "Could not acquire lock"** —
|
||||
a previous activation is still running (often hung). `incus exec
|
||||
<name> -- pkill -9 -f switch-to-configuration` clears it.
|
||||
|
||||
## State
|
||||
|
||||
|
|
|
|||
|
|
@ -131,6 +131,10 @@ The module must NOT set:
|
|||
filters by the `flake-ref` key.
|
||||
- If `--port` is missing on the first deploy, the command fails before
|
||||
launching anything.
|
||||
- Network failures during `nixos-rebuild --target-host` leave the
|
||||
container running its previous generation — `nixos-rebuild switch` is
|
||||
itself atomic.
|
||||
- Activation is two ssh steps: `nix-env --set` (atomic pointer swap on
|
||||
`/nix/var/nix/profiles/system`) then `switch-to-configuration switch`.
|
||||
If the second fails partway, the system profile is already at the new
|
||||
toplevel; running deploy again finishes the switch.
|
||||
- Exit 4 from `switch-to-configuration` is treated as success (a unit
|
||||
failed to reload — typically dbus-broker — but activation itself
|
||||
completed).
|
||||
|
|
|
|||
|
|
@ -6,9 +6,19 @@
|
|||
# host-side <listen-ip>:<host-port> is unique per app and stored in
|
||||
# container metadata so re-deploys are flagless. See ./README.md for
|
||||
# the network model and operator flow.
|
||||
#
|
||||
# Marshaling-flake-as-module design (vs. nixos-rebuild --target-host):
|
||||
# the per-deploy state-dir flake exposes ONLY `nixosModules.default`
|
||||
# (combined container essentials + app's nixosModules.incus +
|
||||
# operator-key overlay). The CLI picks nixpkgs at build time (pinned
|
||||
# to nixos-25.11, matching the LXC image baseline) and assembles the
|
||||
# nixosSystem via `nix build --impure --expr`. Closure copy + activation
|
||||
# happen via `nix copy --to ssh://…` + ssh-driven
|
||||
# switch-to-configuration. Lets us dodge unstable nixpkgs regressions
|
||||
# (e.g. the dbus-broker reload stall) without touching the marshaling
|
||||
# flake's shape.
|
||||
{ writeShellApplication
|
||||
, nix
|
||||
, nixos-rebuild
|
||||
, openssh
|
||||
, jq
|
||||
, coreutils
|
||||
|
|
@ -24,7 +34,7 @@ in
|
|||
writeShellApplication {
|
||||
name = "incus-pet";
|
||||
|
||||
runtimeInputs = [ nix nixos-rebuild openssh jq coreutils gnused ];
|
||||
runtimeInputs = [ nix openssh jq coreutils gnused ];
|
||||
|
||||
meta.description = "Deploy a NixOS service flake into a per-app incus container (pet PaaS).";
|
||||
|
||||
|
|
@ -35,6 +45,10 @@ writeShellApplication {
|
|||
CONTAINER_PORT=8080
|
||||
STATE_ROOT="''${XDG_STATE_HOME:-$HOME/.local/state}/incus-pet"
|
||||
INCUS_IMAGE="images:nixos/25.11"
|
||||
# Pinned to stable to match the LXC image baseline. Unstable triggers
|
||||
# a dbus-broker reload stall in switch-to-configuration (~Jul 2026
|
||||
# systemd/dbus-broker regression).
|
||||
NIXPKGS_REF="github:nixos/nixpkgs/nixos-25.11"
|
||||
|
||||
log() { printf '[incus-pet] %s\n' "$*" >&2; }
|
||||
die() { log "error: $*"; exit 1; }
|
||||
|
|
@ -97,8 +111,12 @@ writeShellApplication {
|
|||
}
|
||||
|
||||
write_marshaling_flake() {
|
||||
local name="$1" flake_ref="$2" pubkey="$3"
|
||||
local dir="$STATE_ROOT/$name"
|
||||
# Per-deploy state-dir flake. Exposes ONLY nixosModules.default —
|
||||
# the CLI assembles the full nixosSystem at build time, picking
|
||||
# nixpkgs (so we can pin stable to dodge unstable regressions and
|
||||
# not bake the choice into per-deploy state).
|
||||
local flake_ref="$1" pubkey="$2"
|
||||
local dir="$STATE_ROOT/$3" # caller-chosen subdir (probe or real name)
|
||||
mkdir -p "$dir"
|
||||
|
||||
# Copy the essentials module into the marshaling flake dir as
|
||||
|
|
@ -110,19 +128,15 @@ writeShellApplication {
|
|||
cat > "$dir/flake.nix" <<EOF
|
||||
# Generated by incus-pet — do not edit. Re-run \`incus-pet deploy\` to refresh.
|
||||
{
|
||||
inputs.nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
|
||||
inputs.target.url = "$flake_ref";
|
||||
outputs = { nixpkgs, target, ... }: {
|
||||
nixosConfigurations."$name" = nixpkgs.lib.nixosSystem {
|
||||
system = "x86_64-linux";
|
||||
modules = [
|
||||
inputs.target.url = "$flake_ref";
|
||||
outputs = { target, ... }: {
|
||||
nixosModules.default = { ... }: {
|
||||
imports = [
|
||||
./container.nix
|
||||
target.nixosModules.incus
|
||||
({ ... }: {
|
||||
users.users.root.openssh.authorizedKeys.keys = [
|
||||
"$pubkey"
|
||||
];
|
||||
})
|
||||
];
|
||||
users.users.root.openssh.authorizedKeys.keys = [
|
||||
"$pubkey"
|
||||
];
|
||||
};
|
||||
};
|
||||
|
|
@ -138,17 +152,40 @@ writeShellApplication {
|
|||
fi
|
||||
}
|
||||
|
||||
eval_hostname() {
|
||||
local dir="$1" name="$2"
|
||||
nix eval --raw \
|
||||
"$dir#nixosConfigurations.\"$name\".config.incus.container.hostname"
|
||||
# Boilerplate that assembles the full nixosSystem on demand. Pinned
|
||||
# nixpkgs (NIXPKGS_REF) is fetched via builtins.getFlake; --impure
|
||||
# is required because builtins.getFlake on a URL isn't
|
||||
# pure-evaluable. The marshaling flake is loaded by absolute store
|
||||
# path so the operator's machine doesn't re-fetch it.
|
||||
sys_expr() {
|
||||
local dir="$1"
|
||||
printf 'let
|
||||
marshaling = builtins.getFlake "path:%s";
|
||||
nixpkgs = builtins.getFlake "%s";
|
||||
in nixpkgs.lib.nixosSystem {
|
||||
system = "x86_64-linux";
|
||||
modules = [ marshaling.nixosModules.default ];
|
||||
}' "$dir" "$NIXPKGS_REF"
|
||||
}
|
||||
|
||||
eval_config_attr() {
|
||||
local dir="$1" attr="$2"
|
||||
nix eval --raw --impure --expr "($(sys_expr "$dir")).config.$attr"
|
||||
}
|
||||
|
||||
build_toplevel() {
|
||||
local dir="$1"
|
||||
nix build --print-out-paths --no-link --impure --expr \
|
||||
"($(sys_expr "$dir")).config.system.build.toplevel"
|
||||
}
|
||||
|
||||
bootstrap_container() {
|
||||
# The official NixOS incus image already runs sshd via systemd
|
||||
# socket activation on port 22; the only thing we need to add is
|
||||
# the operator's pubkey to root's authorized_keys. Idempotent —
|
||||
# `tee` overwrites, so a rotated key gets picked up on next deploy.
|
||||
# the operator's pubkey to root's authorized_keys (for interactive
|
||||
# ssh debugging — the deploy path uses `incus exec`, not ssh).
|
||||
# Idempotent — `tee` overwrites, so a rotated key gets picked up
|
||||
# on next deploy.
|
||||
local name="$1" pubkey="$2"
|
||||
log "ensuring authorized_keys on $name"
|
||||
incus exec "$name" -- mkdir -p /root/.ssh
|
||||
|
|
@ -157,6 +194,38 @@ writeShellApplication {
|
|||
incus exec "$name" -- chmod 600 /root/.ssh/authorized_keys
|
||||
}
|
||||
|
||||
# Mount the host's /nix/store into the container as read-only —
|
||||
# `shift=true` idmaps host UIDs into the unprivileged container.
|
||||
# Idempotent; takes effect on next container start. With this in
|
||||
# place, every container shares the host's /nix/store on disk —
|
||||
# zero duplication of nixpkgs closures across containers.
|
||||
ensure_nix_store_mount() {
|
||||
local name="$1"
|
||||
if incus config device show "$name" 2>/dev/null | grep -q '^nix-store:'; then
|
||||
return 0
|
||||
fi
|
||||
log "adding shared /nix/store mount on $name (host's store, read-only)"
|
||||
incus config device add "$name" nix-store disk \
|
||||
source=/nix/store \
|
||||
path=/nix/store \
|
||||
readonly=true \
|
||||
shift=true
|
||||
}
|
||||
|
||||
# Pin the toplevel against host-side GC. The container's nix DB
|
||||
# doesn't know about the path (it lives behind the shared mount),
|
||||
# so without this the host could nix-collect-garbage the running
|
||||
# system out from under the container. `nix-store --add-root
|
||||
# --indirect` creates an indirect root: a user-writable symlink
|
||||
# (in our state dir, no sudo) plus a tracked entry in
|
||||
# /nix/var/nix/gcroots/auto/. Idempotent; overwrites prior.
|
||||
register_host_gcroot() {
|
||||
local name="$1" toplevel="$2"
|
||||
local link="$STATE_ROOT/$name/system"
|
||||
mkdir -p "$STATE_ROOT/$name"
|
||||
nix-store --add-root "$link" --indirect --realise "$toplevel" >/dev/null
|
||||
}
|
||||
|
||||
wire_proxy_device() {
|
||||
local name="$1" listen="$2" host_port="$3"
|
||||
local devname="web"
|
||||
|
|
@ -214,19 +283,29 @@ writeShellApplication {
|
|||
# placeholder; we'll re-evaluate after.
|
||||
local tentative_name="''${name:-_probe}"
|
||||
local dir
|
||||
dir=$(write_marshaling_flake "$tentative_name" "$flake_ref" "$pubkey")
|
||||
dir=$(write_marshaling_flake "$flake_ref" "$pubkey" "$tentative_name")
|
||||
|
||||
log "locking marshaling flake at $dir"
|
||||
( cd "$dir" && nix flake lock )
|
||||
# `nix flake update` (not `lock`) so every deploy bumps the target
|
||||
# input to the current tip of its ref (otherwise re-running `deploy
|
||||
# github:srid/anywhen` would stay pinned at the first commit we
|
||||
# saw). `--refresh` bypasses nix's fetcher cache so the resolution
|
||||
# is fresh, not a 5-min-stale entry.
|
||||
( cd "$dir" && nix flake update --refresh )
|
||||
|
||||
if [ -z "$name" ]; then
|
||||
name=$(eval_hostname "$dir" "_probe")
|
||||
name=$(eval_config_attr "$dir" "incus.container.hostname")
|
||||
log "container name from flake: $name"
|
||||
# Re-emit the marshaling flake under the real name (the
|
||||
# nixosConfigurations attr key must match `--flake .#<name>`).
|
||||
# Re-emit the marshaling flake under the real name so the
|
||||
# state-dir matches what `incus-pet list` shows.
|
||||
rm -rf "$STATE_ROOT/_probe"
|
||||
dir=$(write_marshaling_flake "$name" "$flake_ref" "$pubkey")
|
||||
( cd "$dir" && nix flake lock )
|
||||
dir=$(write_marshaling_flake "$flake_ref" "$pubkey" "$name")
|
||||
# `nix flake update` (not `lock`) so every deploy bumps the target
|
||||
# input to the current tip of its ref (otherwise re-running `deploy
|
||||
# github:srid/anywhen` would stay pinned at the first commit we
|
||||
# saw). `--refresh` bypasses nix's fetcher cache so the resolution
|
||||
# is fresh, not a 5-min-stale entry.
|
||||
( cd "$dir" && nix flake update --refresh )
|
||||
fi
|
||||
|
||||
# Resolve host port (CLI flag > container metadata > error).
|
||||
|
|
@ -258,16 +337,39 @@ writeShellApplication {
|
|||
|
||||
bootstrap_container "$name" "$pubkey"
|
||||
|
||||
log "activating: nixos-rebuild switch --flake $dir#$name --target-host root@$ip"
|
||||
# accept-new = trust-on-first-use; reject if a known host's key
|
||||
# changed. Fresh containers get a fresh host key, and the operator
|
||||
# launched the container themselves, so TOFU is fine.
|
||||
( cd "$dir" \
|
||||
&& NIX_SSHOPTS='-o StrictHostKeyChecking=accept-new' \
|
||||
nixos-rebuild switch \
|
||||
--flake ".#$name" \
|
||||
--target-host "root@$ip" \
|
||||
--use-substitutes )
|
||||
log "building toplevel (pinned $NIXPKGS_REF)"
|
||||
local toplevel
|
||||
toplevel=$(build_toplevel "$dir")
|
||||
log "toplevel: $toplevel"
|
||||
|
||||
log "pinning host gcroot for $name → $toplevel"
|
||||
register_host_gcroot "$name" "$toplevel"
|
||||
|
||||
# Update the container's system profile FIRST (while it can still
|
||||
# see its own /nix/store), then add the shared-mount device if
|
||||
# missing, then restart so the new init picks up via the shared
|
||||
# mount. Order matters: if we added the mount before updating
|
||||
# the profile, the container's existing /sbin/init -> old toplevel
|
||||
# would point at a path masked by the mount and the restart would
|
||||
# fail.
|
||||
log "setting system profile on $name → $toplevel"
|
||||
incus exec "$name" -- ln -sfn "$toplevel" /nix/var/nix/profiles/system
|
||||
# The NixOS LXC image hard-links /sbin/init to the image's own
|
||||
# toplevel; re-pointing it through the profile symlink is what
|
||||
# makes "update profile + restart" actually pick up the new
|
||||
# toplevel's init (which sets up /etc, runs activation, then
|
||||
# execs systemd). NixOS's switch-to-configuration normally does
|
||||
# this — we have to do it ourselves since we skip s-t-c.
|
||||
# Idempotent across deploys.
|
||||
incus exec "$name" -- ln -sfn /nix/var/nix/profiles/system/init /sbin/init
|
||||
|
||||
ensure_nix_store_mount "$name"
|
||||
|
||||
log "restarting $name to boot via shared /nix/store"
|
||||
incus restart "$name"
|
||||
|
||||
ip=$(container_ipv4 "$name")
|
||||
log "$name back up at $ip"
|
||||
|
||||
# Persist host-port + listen IP for next deploy. The `key=value`
|
||||
# form is required since incus 6.x; the space-separated form is
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue