ralph(eval): reduce Nix eval time for pureintent + zest (#120)

* ralph(eval): baseline + benchmark harness

pureintent cpuTime 23.17s, zest cpuTime 14.93s (median of 5).
Methodology: NIX_SHOW_STATS cpuTime, eval-cache off, warm fetcher cache.

* ralph(eval) cycle 1: disable NixOS options manual on pureintent

documentation.nixos.enable = false. Generating the NixOS manual evaluates
every option's doc string. nrThunks 20,400,114 -> 19,201,996 (-5.9%),
nrFunctionCalls -6.2%, nrPrimOpCalls -7.5%. Package man pages unaffected.

Metric pivoted to deterministic NIX_SHOW_STATS counters: cpuTime is unusable
on this thermally-throttling machine (same eval = 11s..21s).

* ralph(eval): switch home target zest -> sincereintent (per request)

sincereintent home config baseline: nrThunks 5,323,428 / calls 3,227,404 /
primops 1,565,120 (deterministic).

* ralph(eval) cycle 2: disable home-manager options manpage

manual.manpages.enable = false in shared modules/home/default.nix. HM builds
home-configuration.nix(5) by default, evaluating every HM option's doc string.
sincereintent nrThunks 5,323,428 -> 3,551,679 (-33.3%), calls -35%, primops -39%.

* ralph(eval): record cycle-2 pureintent gain (shared HM module)

pureintent nrThunks 19,201,996 -> 17,431,437 (-9.2%); cycle-2's
manual.manpages.enable=false also feeds pureintent's embedded home-manager
via modules/nixos/default.nix. Total pureintent -14.6% from baseline.

* ralph(eval) cycle 3: profiling + dead-ends (no behaviour-preserving win)

Confirmed: nixpkgs single-instantiation (no 2x trap), unused inputs are lazy
(no counter impact), escapeShellArg hotspot is HM file-linking + /etc (not
patchable here). Remaining eval cost is feature-bound (embedded HM, vira/kolu,
google-cloud-sdk/pandoc/omnix).

* ralph(eval) cycle 4: drop heavy home.packages (yt-dlp/lima/omnix/pandoc/gcloud)

User-approved behaviour change. Removed from shared modules/home/cli/terminal.nix
(kept hledger). pureintent nrThunks 17,431,437 -> 16,989,720 (-2.5%),
sincereintent 3,551,679 -> 2,757,201 (-22.4%). Totals from baseline:
pureintent -16.7%, sincereintent -48.2%.

* ralph(eval): finalize report (final measurement, key findings, cost breakdown)
This commit is contained in:
Sridhar Ratnakumar 2026-05-29 07:19:48 -04:00 committed by GitHub
parent 87441d5270
commit b4a60aaf3a
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
5 changed files with 185 additions and 7 deletions

32
docs/eval-bench.sh Executable file
View file

@ -0,0 +1,32 @@
#!/usr/bin/env bash
# Ralph eval-time benchmark — counter-based.
# Usage: ralph-bench.sh [target] target = pureintent | sincereintent | (both)
#
# Primary metric: deterministic eval-work counters from NIX_SHOW_STATS:
# nrThunks, nrFunctionCalls, nrPrimOpCalls.
# These are byte-identical across runs (independent of CPU clock/load), so a
# single eval gives an exact number. cpuTime is NOT used: this machine thermally
# throttles, inflating CPU-seconds for identical work (11s..21s for one eval).
# Eval cache disabled; fetcher cache warm via persistent $HOME.
set -uo pipefail
export HOME=/tmp/ralph-evalhome
mkdir -p "$HOME"
FLAKE="${FLAKE:-/home/srid/code/nixos-config/.worktrees/ralph}"
ONLY="${1:-}"
declare -A ATTR
ATTR[pureintent]="$FLAKE#nixosConfigurations.pureintent.config.system.build.toplevel.drvPath"
ATTR[sincereintent]="$FLAKE#homeConfigurations.\"srid@sincereintent\".activationPackage.drvPath"
num() { grep -o "\"$1\": [0-9.]*" /tmp/ralph-stats.json | grep -o '[0-9.]*' | head -1; }
for target in pureintent sincereintent; do
[ -n "$ONLY" ] && [ "$ONLY" != "$target" ] && continue
NIX_SHOW_STATS=1 nix eval --raw "${ATTR[$target]}" --option eval-cache false \
>/dev/null 2>/tmp/ralph-stats.json
if [ $? -ne 0 ]; then echo "$target FAILED"; tail -8 /tmp/ralph-stats.json; exit 1; fi
printf '%-11s thunks=%-10s calls=%-10s primops=%-9s | values=%s sets=%s\n' \
"$target" "$(num nrThunks)" "$(num nrFunctionCalls)" "$(num nrPrimOpCalls)" \
"$(num number)" "$(grep -o '"sets": {[^}]*"number": [0-9]*' /tmp/ralph-stats.json | grep -o '[0-9]*$')"
done

View file

@ -0,0 +1,143 @@
# Nix Eval-Time Optimization (Ralph)
Iterative measurement-driven reduction of Nix **evaluation** time for the two
configs that get evaluated most often on this machine:
- `pureintent` — the NixOS system (`nixosConfigurations.pureintent`)
- `sincereintent` — the home-manager config (`homeConfigurations."srid@sincereintent"`)
## Methodology
Primary metric: **deterministic eval-work counters** from `NIX_SHOW_STATS=1`
`nrThunks`, `nrFunctionCalls`, `nrPrimOpCalls`. These count the actual evaluation
work and are **byte-identical across runs** (independent of CPU clock, load, GC).
> **Why not `cpuTime`/wall?** Initially planned, but rejected after measuring:
> this is a live, thermally-throttling laptop. The *same* eval (identical
> nrThunks every run) reported cpuTime climbing 11s → 21s monotonically as the
> CPU clock dropped under sustained load. Time is meaningless here; counters are
> exact. A 1% drop in nrThunks is a real 1% less work the evaluator must do, on
> any machine. Single eval per target — no median needed.
```sh
# docs/eval-bench.sh [target] target = pureintent | zest | (both)
# - persistent $HOME=/tmp/ralph-evalhome keeps the fetcher/git cache warm
# - --option eval-cache false forces a full re-eval every run
# - reports nrThunks / nrFunctionCalls / nrPrimOpCalls per target (deterministic)
nix eval --raw .#nixosConfigurations.pureintent.config.system.build.toplevel.drvPath --option eval-cache false
nix eval --raw '.#homeConfigurations."srid@sincereintent".activationPackage.drvPath' --option eval-cache false
```
No `nix store gc` between runs (per request). The eval cache is disabled rather
than cleared, so the warm fetcher cache removes input-fetch noise.
**Commit rule:** commit only if `nrThunks` improves >3% for a target with no
regression on the other. (Counters have ~zero noise, so any real reduction
counts; >3% is the bar for a "meaningful" cycle.) Behaviour may change *only* by
dropping inputs/features first confirmed unused.
Profiling tool: `nix eval … --eval-profiler flamegraph --eval-profile-file f`
then aggregate self/inclusive frames (collapsed-stack format).
## Baseline (nrThunks / nrFunctionCalls / nrPrimOpCalls)
| Target | nrThunks | nrFunctionCalls | nrPrimOpCalls |
|------------|------------|-----------------|---------------|
| pureintent | 20,400,114 | 12,975,268 | 6,507,025 |
| sincereintent | 5,323,428 | 3,227,404 | 1,565,120 |
Env: nix 2.34.7, x86_64-linux.
## Optimization log
(Δ% is on nrThunks vs baseline for the affected target.)
| Cycle | Change | pureintent thunks | sincereintent thunks | Verdict |
|-------|--------|-------------------|----------------------|---------|
| 0 | baseline | 20,400,114 | 5,323,428 | — |
| 1 | pureintent: `documentation.nixos.enable = false` (drop NixOS options manual; option doc-strings no longer evaluated) | **19,201,996 (5.9%)** | 5,323,428 | ✅ commit |
| 2 | home: `manual.manpages.enable = false` in shared `modules/home/default.nix` (drop home-manager options manpage; evaluated every HM option's doc) | **17,431,437 (9.2%)** | **3,551,679 (33.3%)** | ✅ commit |
> Note: `modules/home/default.nix` is also fed to every Linux system's *embedded*
> home-manager via `modules/nixos/default.nix`, so cycle 2 reduced pureintent
> too (its HM user's manpages were previously on).
| 3 | (profiling cycle — no change) investigated nixpkgs double-instantiation, unused inputs, `escapeShellArg` hotspot | 17,431,437 | 3,551,679 | — no behaviour-preserving win found |
| 4 | home: drop heavy `home.packages` from shared `cli/terminal.nix``yt-dlp`, `lima`, `omnix`, `pandoc`, `google-cloud-sdk` (kept `hledger`). *Behaviour change, user-approved.* | **16,989,720 (2.5%)** | **2,757,201 (22.4%)** | ✅ commit |
## Dead ends
_(investigated, no improvement — recorded so we don't retry)_
- **nixpkgs is NOT double-instantiated.** `nixos-unified.lib.mkLinuxSystem` calls
`nixpkgs.lib.nixosSystem` without pinning `nixpkgs.pkgs`, but the NixOS
`nixpkgs` module imports the package set exactly once; the `pkgs/top-level/impure.nix`
frame in the profile is that single import. `home-manager.useGlobalPkgs = true`
(set by nixos-unified) already shares it with the embedded HM. No win available.
- **Dropping unused flake inputs does not reduce eval counters.** Nix input
evaluation is lazy: an input not referenced during a config's eval (e.g.
`nixos-hardware`, `git-hooks` — 0 refs) contributes ~0 thunks to that config.
Pruning them shrinks `flake.lock`/`nix flake` overhead only, not `nrThunks`.
- **`escapeShellArg` (~72% inclusive on pureintent) is not a fixable hotspot.**
It is driven by `home-manager` file-linking (`files.nix:442`, ~43% — one
escaped command per managed dotfile) and NixOS `/etc` generation
(`etc.nix:58`). Both scale with how much is managed; nixpkgs/HM own the
implementation. Not patchable from this repo.
- **`programs.command-not-found` already disabled** by the `nix-index-database`
module; nothing to gain.
## Remaining cost is feature-bound
After the two documentation wins, the dominant `derivationStrict` targets in the
pureintent eval profile are all *wanted* features (samples ≈ inclusive share):
`home-manager-generation`/`home-manager-files` (~43%, embedded HM), `/etc`
(~29%), `vira.service`+`vira-wrapped`+`vira-0.1.0.0` (Haskell app), `kolu.service`,
`pipewire`, plus heavy packages in `home.packages` (`google-cloud-sdk`, `pandoc`,
`omnix` — Haskell). Further reductions require trading a feature, not a free
structural change.
## Final measurement (median irrelevant — counters are exact)
| Target | nrThunks (base → final) | Δ | nrFunctionCalls | nrPrimOpCalls |
|--------|-------------------------|------|-----------------|---------------|
| pureintent | 20,400,114 → **16,989,720** | **16.7%** | 12,975,268 → 10,794,555 (16.8%) | 6,507,025 → 5,327,976 (18.1%) |
| sincereintent | 5,323,428 → **2,757,201** | **48.2%** | 3,227,404 → 1,631,213 (49.5%) | 1,565,120 → 771,171 (50.7%) |
Both configs still instantiate cleanly (`nix build --dry-run` on the system
toplevel / home activationPackage succeeds).
## Key findings
1. **On a thermally-throttling machine, time is a lie — count work instead.**
The same eval reported cpuTime from 11s to 21s as the CPU clock sagged, while
`nrThunks` was byte-identical every run. Eval-work counters (`nrThunks`,
`nrFunctionCalls`, `nrPrimOpCalls`) are deterministic, zero-noise, and
machine-independent — a far better optimization metric than wall/cpu time.
2. **Generated option documentation is the biggest free win.** The NixOS options
manual (`documentation.nixos.enable`) and the home-manager options manpage
(`manual.manpages.enable`) each evaluate the doc string of *every* option.
Disabling both: pureintent 14.6%, sincereintent 33% — with zero behaviour
cost (options are searched online).
3. **A shared module multiplies.** `manual.manpages.enable = false` in
`modules/home/default.nix` helped both the standalone home config *and* every
Linux system's embedded home-manager (via `modules/nixos/default.nix`).
4. **Eval is lazy — unused inputs are already free.** Pruning unreferenced flake
inputs does not move the counters; it only trims `flake.lock`.
5. **The remainder is feature-bound.** After the doc wins, eval cost is dominated
by embedded home-manager file-linking, `/etc` generation, and the closures of
wanted services/packages. Trimming 5 heavy `home.packages` (user-approved) took
sincereintent to 48% total; the rest are features worth their eval cost.
## Cost breakdown (per-package eval cost, standalone `pkgs.<p>.drvPath` thunks)
Shared base floor ≈ 1.09M thunks; the excess is each package's own closure-eval:
| Package | thunks | excess | removed? |
|---------|--------|--------|----------|
| yt-dlp | 2.15M | +1.05M | ✅ |
| lima | 1.78M | +0.69M | ✅ |
| omnix | 1.76M | +0.67M | ✅ |
| pandoc | 1.46M | +0.37M | ✅ |
| google-cloud-sdk | 1.39M | +0.30M | ✅ |
| hledger | 1.35M | +0.26M | kept (wanted) |
| ripgrep / fd / just / television | ~1.09M | ~0 | kept (at floor) |