mirror of
https://github.com/srid/nixos-config.git
synced 2026-07-16 22:01:33 +08:00
ralph(eval): reduce Nix eval time for pureintent + zest (#120)
* ralph(eval): baseline + benchmark harness pureintent cpuTime 23.17s, zest cpuTime 14.93s (median of 5). Methodology: NIX_SHOW_STATS cpuTime, eval-cache off, warm fetcher cache. * ralph(eval) cycle 1: disable NixOS options manual on pureintent documentation.nixos.enable = false. Generating the NixOS manual evaluates every option's doc string. nrThunks 20,400,114 -> 19,201,996 (-5.9%), nrFunctionCalls -6.2%, nrPrimOpCalls -7.5%. Package man pages unaffected. Metric pivoted to deterministic NIX_SHOW_STATS counters: cpuTime is unusable on this thermally-throttling machine (same eval = 11s..21s). * ralph(eval): switch home target zest -> sincereintent (per request) sincereintent home config baseline: nrThunks 5,323,428 / calls 3,227,404 / primops 1,565,120 (deterministic). * ralph(eval) cycle 2: disable home-manager options manpage manual.manpages.enable = false in shared modules/home/default.nix. HM builds home-configuration.nix(5) by default, evaluating every HM option's doc string. sincereintent nrThunks 5,323,428 -> 3,551,679 (-33.3%), calls -35%, primops -39%. * ralph(eval): record cycle-2 pureintent gain (shared HM module) pureintent nrThunks 19,201,996 -> 17,431,437 (-9.2%); cycle-2's manual.manpages.enable=false also feeds pureintent's embedded home-manager via modules/nixos/default.nix. Total pureintent -14.6% from baseline. * ralph(eval) cycle 3: profiling + dead-ends (no behaviour-preserving win) Confirmed: nixpkgs single-instantiation (no 2x trap), unused inputs are lazy (no counter impact), escapeShellArg hotspot is HM file-linking + /etc (not patchable here). Remaining eval cost is feature-bound (embedded HM, vira/kolu, google-cloud-sdk/pandoc/omnix). * ralph(eval) cycle 4: drop heavy home.packages (yt-dlp/lima/omnix/pandoc/gcloud) User-approved behaviour change. Removed from shared modules/home/cli/terminal.nix (kept hledger). pureintent nrThunks 17,431,437 -> 16,989,720 (-2.5%), sincereintent 3,551,679 -> 2,757,201 (-22.4%). Totals from baseline: pureintent -16.7%, sincereintent -48.2%. * ralph(eval): finalize report (final measurement, key findings, cost breakdown)
This commit is contained in:
parent
87441d5270
commit
b4a60aaf3a
5 changed files with 185 additions and 7 deletions
|
|
@ -81,6 +81,11 @@
|
|||
# Allow unfree packages
|
||||
nixpkgs.config.allowUnfree = true;
|
||||
|
||||
# The NixOS options manual is rarely consulted locally (options are searched
|
||||
# online), but generating it evaluates the doc string of every option — a
|
||||
# measurable chunk of eval time. Package man pages (`man git`) are unaffected.
|
||||
documentation.nixos.enable = false;
|
||||
|
||||
# List packages installed in system profile. To search, run:
|
||||
# $ nix search wget
|
||||
environment.systemPackages = with pkgs; [
|
||||
|
|
|
|||
32
docs/eval-bench.sh
Executable file
32
docs/eval-bench.sh
Executable file
|
|
@ -0,0 +1,32 @@
|
|||
#!/usr/bin/env bash
|
||||
# Ralph eval-time benchmark — counter-based.
|
||||
# Usage: ralph-bench.sh [target] target = pureintent | sincereintent | (both)
|
||||
#
|
||||
# Primary metric: deterministic eval-work counters from NIX_SHOW_STATS:
|
||||
# nrThunks, nrFunctionCalls, nrPrimOpCalls.
|
||||
# These are byte-identical across runs (independent of CPU clock/load), so a
|
||||
# single eval gives an exact number. cpuTime is NOT used: this machine thermally
|
||||
# throttles, inflating CPU-seconds for identical work (11s..21s for one eval).
|
||||
# Eval cache disabled; fetcher cache warm via persistent $HOME.
|
||||
set -uo pipefail
|
||||
|
||||
export HOME=/tmp/ralph-evalhome
|
||||
mkdir -p "$HOME"
|
||||
FLAKE="${FLAKE:-/home/srid/code/nixos-config/.worktrees/ralph}"
|
||||
ONLY="${1:-}"
|
||||
|
||||
declare -A ATTR
|
||||
ATTR[pureintent]="$FLAKE#nixosConfigurations.pureintent.config.system.build.toplevel.drvPath"
|
||||
ATTR[sincereintent]="$FLAKE#homeConfigurations.\"srid@sincereintent\".activationPackage.drvPath"
|
||||
|
||||
num() { grep -o "\"$1\": [0-9.]*" /tmp/ralph-stats.json | grep -o '[0-9.]*' | head -1; }
|
||||
|
||||
for target in pureintent sincereintent; do
|
||||
[ -n "$ONLY" ] && [ "$ONLY" != "$target" ] && continue
|
||||
NIX_SHOW_STATS=1 nix eval --raw "${ATTR[$target]}" --option eval-cache false \
|
||||
>/dev/null 2>/tmp/ralph-stats.json
|
||||
if [ $? -ne 0 ]; then echo "$target FAILED"; tail -8 /tmp/ralph-stats.json; exit 1; fi
|
||||
printf '%-11s thunks=%-10s calls=%-10s primops=%-9s | values=%s sets=%s\n' \
|
||||
"$target" "$(num nrThunks)" "$(num nrFunctionCalls)" "$(num nrPrimOpCalls)" \
|
||||
"$(num number)" "$(grep -o '"sets": {[^}]*"number": [0-9]*' /tmp/ralph-stats.json | grep -o '[0-9]*$')"
|
||||
done
|
||||
143
docs/eval-time-ralph-report.md
Normal file
143
docs/eval-time-ralph-report.md
Normal file
|
|
@ -0,0 +1,143 @@
|
|||
# Nix Eval-Time Optimization (Ralph)
|
||||
|
||||
Iterative measurement-driven reduction of Nix **evaluation** time for the two
|
||||
configs that get evaluated most often on this machine:
|
||||
|
||||
- `pureintent` — the NixOS system (`nixosConfigurations.pureintent`)
|
||||
- `sincereintent` — the home-manager config (`homeConfigurations."srid@sincereintent"`)
|
||||
|
||||
## Methodology
|
||||
|
||||
Primary metric: **deterministic eval-work counters** from `NIX_SHOW_STATS=1` —
|
||||
`nrThunks`, `nrFunctionCalls`, `nrPrimOpCalls`. These count the actual evaluation
|
||||
work and are **byte-identical across runs** (independent of CPU clock, load, GC).
|
||||
|
||||
> **Why not `cpuTime`/wall?** Initially planned, but rejected after measuring:
|
||||
> this is a live, thermally-throttling laptop. The *same* eval (identical
|
||||
> nrThunks every run) reported cpuTime climbing 11s → 21s monotonically as the
|
||||
> CPU clock dropped under sustained load. Time is meaningless here; counters are
|
||||
> exact. A 1% drop in nrThunks is a real 1% less work the evaluator must do, on
|
||||
> any machine. Single eval per target — no median needed.
|
||||
|
||||
```sh
|
||||
# docs/eval-bench.sh [target] target = pureintent | zest | (both)
|
||||
# - persistent $HOME=/tmp/ralph-evalhome keeps the fetcher/git cache warm
|
||||
# - --option eval-cache false forces a full re-eval every run
|
||||
# - reports nrThunks / nrFunctionCalls / nrPrimOpCalls per target (deterministic)
|
||||
nix eval --raw .#nixosConfigurations.pureintent.config.system.build.toplevel.drvPath --option eval-cache false
|
||||
nix eval --raw '.#homeConfigurations."srid@sincereintent".activationPackage.drvPath' --option eval-cache false
|
||||
```
|
||||
|
||||
No `nix store gc` between runs (per request). The eval cache is disabled rather
|
||||
than cleared, so the warm fetcher cache removes input-fetch noise.
|
||||
|
||||
**Commit rule:** commit only if `nrThunks` improves >3% for a target with no
|
||||
regression on the other. (Counters have ~zero noise, so any real reduction
|
||||
counts; >3% is the bar for a "meaningful" cycle.) Behaviour may change *only* by
|
||||
dropping inputs/features first confirmed unused.
|
||||
|
||||
Profiling tool: `nix eval … --eval-profiler flamegraph --eval-profile-file f`
|
||||
then aggregate self/inclusive frames (collapsed-stack format).
|
||||
|
||||
## Baseline (nrThunks / nrFunctionCalls / nrPrimOpCalls)
|
||||
|
||||
| Target | nrThunks | nrFunctionCalls | nrPrimOpCalls |
|
||||
|------------|------------|-----------------|---------------|
|
||||
| pureintent | 20,400,114 | 12,975,268 | 6,507,025 |
|
||||
| sincereintent | 5,323,428 | 3,227,404 | 1,565,120 |
|
||||
|
||||
Env: nix 2.34.7, x86_64-linux.
|
||||
|
||||
## Optimization log
|
||||
|
||||
(Δ% is on nrThunks vs baseline for the affected target.)
|
||||
|
||||
| Cycle | Change | pureintent thunks | sincereintent thunks | Verdict |
|
||||
|-------|--------|-------------------|----------------------|---------|
|
||||
| 0 | baseline | 20,400,114 | 5,323,428 | — |
|
||||
| 1 | pureintent: `documentation.nixos.enable = false` (drop NixOS options manual; option doc-strings no longer evaluated) | **19,201,996 (−5.9%)** | 5,323,428 | ✅ commit |
|
||||
| 2 | home: `manual.manpages.enable = false` in shared `modules/home/default.nix` (drop home-manager options manpage; evaluated every HM option's doc) | **17,431,437 (−9.2%)** | **3,551,679 (−33.3%)** | ✅ commit |
|
||||
|
||||
> Note: `modules/home/default.nix` is also fed to every Linux system's *embedded*
|
||||
> home-manager via `modules/nixos/default.nix`, so cycle 2 reduced pureintent
|
||||
> too (its HM user's manpages were previously on).
|
||||
|
||||
| 3 | (profiling cycle — no change) investigated nixpkgs double-instantiation, unused inputs, `escapeShellArg` hotspot | 17,431,437 | 3,551,679 | — no behaviour-preserving win found |
|
||||
| 4 | home: drop heavy `home.packages` from shared `cli/terminal.nix` — `yt-dlp`, `lima`, `omnix`, `pandoc`, `google-cloud-sdk` (kept `hledger`). *Behaviour change, user-approved.* | **16,989,720 (−2.5%)** | **2,757,201 (−22.4%)** | ✅ commit |
|
||||
|
||||
## Dead ends
|
||||
|
||||
_(investigated, no improvement — recorded so we don't retry)_
|
||||
|
||||
- **nixpkgs is NOT double-instantiated.** `nixos-unified.lib.mkLinuxSystem` calls
|
||||
`nixpkgs.lib.nixosSystem` without pinning `nixpkgs.pkgs`, but the NixOS
|
||||
`nixpkgs` module imports the package set exactly once; the `pkgs/top-level/impure.nix`
|
||||
frame in the profile is that single import. `home-manager.useGlobalPkgs = true`
|
||||
(set by nixos-unified) already shares it with the embedded HM. No win available.
|
||||
- **Dropping unused flake inputs does not reduce eval counters.** Nix input
|
||||
evaluation is lazy: an input not referenced during a config's eval (e.g.
|
||||
`nixos-hardware`, `git-hooks` — 0 refs) contributes ~0 thunks to that config.
|
||||
Pruning them shrinks `flake.lock`/`nix flake` overhead only, not `nrThunks`.
|
||||
- **`escapeShellArg` (~72% inclusive on pureintent) is not a fixable hotspot.**
|
||||
It is driven by `home-manager` file-linking (`files.nix:442`, ~43% — one
|
||||
escaped command per managed dotfile) and NixOS `/etc` generation
|
||||
(`etc.nix:58`). Both scale with how much is managed; nixpkgs/HM own the
|
||||
implementation. Not patchable from this repo.
|
||||
- **`programs.command-not-found` already disabled** by the `nix-index-database`
|
||||
module; nothing to gain.
|
||||
|
||||
## Remaining cost is feature-bound
|
||||
|
||||
After the two documentation wins, the dominant `derivationStrict` targets in the
|
||||
pureintent eval profile are all *wanted* features (samples ≈ inclusive share):
|
||||
`home-manager-generation`/`home-manager-files` (~43%, embedded HM), `/etc`
|
||||
(~29%), `vira.service`+`vira-wrapped`+`vira-0.1.0.0` (Haskell app), `kolu.service`,
|
||||
`pipewire`, plus heavy packages in `home.packages` (`google-cloud-sdk`, `pandoc`,
|
||||
`omnix` — Haskell). Further reductions require trading a feature, not a free
|
||||
structural change.
|
||||
|
||||
## Final measurement (median irrelevant — counters are exact)
|
||||
|
||||
| Target | nrThunks (base → final) | Δ | nrFunctionCalls | nrPrimOpCalls |
|
||||
|--------|-------------------------|------|-----------------|---------------|
|
||||
| pureintent | 20,400,114 → **16,989,720** | **−16.7%** | 12,975,268 → 10,794,555 (−16.8%) | 6,507,025 → 5,327,976 (−18.1%) |
|
||||
| sincereintent | 5,323,428 → **2,757,201** | **−48.2%** | 3,227,404 → 1,631,213 (−49.5%) | 1,565,120 → 771,171 (−50.7%) |
|
||||
|
||||
Both configs still instantiate cleanly (`nix build --dry-run` on the system
|
||||
toplevel / home activationPackage succeeds).
|
||||
|
||||
## Key findings
|
||||
|
||||
1. **On a thermally-throttling machine, time is a lie — count work instead.**
|
||||
The same eval reported cpuTime from 11s to 21s as the CPU clock sagged, while
|
||||
`nrThunks` was byte-identical every run. Eval-work counters (`nrThunks`,
|
||||
`nrFunctionCalls`, `nrPrimOpCalls`) are deterministic, zero-noise, and
|
||||
machine-independent — a far better optimization metric than wall/cpu time.
|
||||
2. **Generated option documentation is the biggest free win.** The NixOS options
|
||||
manual (`documentation.nixos.enable`) and the home-manager options manpage
|
||||
(`manual.manpages.enable`) each evaluate the doc string of *every* option.
|
||||
Disabling both: pureintent −14.6%, sincereintent −33% — with zero behaviour
|
||||
cost (options are searched online).
|
||||
3. **A shared module multiplies.** `manual.manpages.enable = false` in
|
||||
`modules/home/default.nix` helped both the standalone home config *and* every
|
||||
Linux system's embedded home-manager (via `modules/nixos/default.nix`).
|
||||
4. **Eval is lazy — unused inputs are already free.** Pruning unreferenced flake
|
||||
inputs does not move the counters; it only trims `flake.lock`.
|
||||
5. **The remainder is feature-bound.** After the doc wins, eval cost is dominated
|
||||
by embedded home-manager file-linking, `/etc` generation, and the closures of
|
||||
wanted services/packages. Trimming 5 heavy `home.packages` (user-approved) took
|
||||
sincereintent to −48% total; the rest are features worth their eval cost.
|
||||
|
||||
## Cost breakdown (per-package eval cost, standalone `pkgs.<p>.drvPath` thunks)
|
||||
|
||||
Shared base floor ≈ 1.09M thunks; the excess is each package's own closure-eval:
|
||||
|
||||
| Package | thunks | excess | removed? |
|
||||
|---------|--------|--------|----------|
|
||||
| yt-dlp | 2.15M | +1.05M | ✅ |
|
||||
| lima | 1.78M | +0.69M | ✅ |
|
||||
| omnix | 1.76M | +0.67M | ✅ |
|
||||
| pandoc | 1.46M | +0.37M | ✅ |
|
||||
| google-cloud-sdk | 1.39M | +0.30M | ✅ |
|
||||
| hledger | 1.35M | +0.26M | kept (wanted) |
|
||||
| ripgrep / fd / just / television | ~1.09M | ~0 | kept (at floor) |
|
||||
|
|
@ -17,29 +17,22 @@ in
|
|||
gnumake
|
||||
killall
|
||||
television
|
||||
yt-dlp
|
||||
gh
|
||||
# Broken, https://github.com/NixOS/nixpkgs/issues/299680
|
||||
# ncdu
|
||||
|
||||
# Useful for Nix development
|
||||
ci
|
||||
omnix
|
||||
nixpkgs-fmt
|
||||
just
|
||||
watchexec
|
||||
fswatch
|
||||
|
||||
eternal-terminal
|
||||
lima
|
||||
|
||||
# AI
|
||||
google-cloud-sdk
|
||||
|
||||
# Publishing
|
||||
asciinema
|
||||
ispell
|
||||
pandoc
|
||||
|
||||
# Dev
|
||||
fuckport
|
||||
|
|
|
|||
|
|
@ -3,6 +3,11 @@
|
|||
home.sessionVariables = {
|
||||
DO_NOT_TRACK = "1";
|
||||
};
|
||||
|
||||
# Home-manager generates an options manpage (`home-configuration.nix(5)`) by
|
||||
# default, which evaluates the doc string of every HM option. Options are
|
||||
# searched online, not via `man`, so skip it — measurable eval-time win.
|
||||
manual.manpages.enable = false;
|
||||
imports = [
|
||||
./cli/tmux.nix
|
||||
./editors/neovim
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue