mirror of
https://github.com/srid/nixos-config.git
synced 2026-07-16 22:01:33 +08:00
ralph(eval) cycle 1: disable NixOS options manual on pureintent
documentation.nixos.enable = false. Generating the NixOS manual evaluates every option's doc string. nrThunks 20,400,114 -> 19,201,996 (-5.9%), nrFunctionCalls -6.2%, nrPrimOpCalls -7.5%. Package man pages unaffected. Metric pivoted to deterministic NIX_SHOW_STATS counters: cpuTime is unusable on this thermally-throttling machine (same eval = 11s..21s).
This commit is contained in:
Sridhar Ratnakumar
parent
a37a585af2
commit
7477e25b46
3 changed files with 53 additions and 43 deletions
|
|
@ -81,6 +81,11 @@
|
|||
# Allow unfree packages
|
||||
nixpkgs.config.allowUnfree = true;
|
||||
|
||||
# The NixOS options manual is rarely consulted locally (options are searched
|
||||
# online), but generating it evaluates the doc string of every option — a
|
||||
# measurable chunk of eval time. Package man pages (`man git`) are unaffected.
|
||||
documentation.nixos.enable = false;
|
||||
|
||||
# List packages installed in system profile. To search, run:
|
||||
# $ nix search wget
|
||||
environment.systemPackages = with pkgs; [
|
||||
|
|
|
|||
|
|
@ -1,39 +1,32 @@
|
|||
#!/usr/bin/env bash
|
||||
# Ralph eval-time benchmark.
|
||||
# Usage: ralph-bench.sh <runs>
|
||||
# Primary metric: median cpuTime (NIX_SHOW_STATS) — stable, isolates eval work.
|
||||
# Secondary: median wall-clock (noisy, IO-bound).
|
||||
# Ralph eval-time benchmark — counter-based.
|
||||
# Usage: ralph-bench.sh [target] target = pureintent | zest | (both)
|
||||
#
|
||||
# Primary metric: deterministic eval-work counters from NIX_SHOW_STATS:
|
||||
# nrThunks, nrFunctionCalls, nrPrimOpCalls.
|
||||
# These are byte-identical across runs (independent of CPU clock/load), so a
|
||||
# single eval gives an exact number. cpuTime is NOT used: this machine thermally
|
||||
# throttles, inflating CPU-seconds for identical work (11s..21s for one eval).
|
||||
# Eval cache disabled; fetcher cache warm via persistent $HOME.
|
||||
set -uo pipefail
|
||||
|
||||
RUNS="${1:-5}"
|
||||
export HOME=/tmp/ralph-evalhome
|
||||
mkdir -p "$HOME"
|
||||
FLAKE="${FLAKE:-/home/srid/code/nixos-config/.worktrees/ralph}"
|
||||
ONLY="${1:-}"
|
||||
|
||||
declare -A ATTR
|
||||
ATTR[pureintent]="$FLAKE#nixosConfigurations.pureintent.config.system.build.toplevel.drvPath"
|
||||
ATTR[zest]="$FLAKE#homeConfigurations.\"srid@zest\".activationPackage.drvPath"
|
||||
|
||||
median() { sort -n | awk '{a[NR]=$1} END{ if(NR%2){print a[(NR+1)/2]} else {print (a[NR/2]+a[NR/2+1])/2} }'; }
|
||||
now_ms() { date +%s%3N; }
|
||||
num() { grep -o "\"$1\": [0-9.]*" /tmp/ralph-stats.json | grep -o '[0-9.]*' | head -1; }
|
||||
|
||||
for target in pureintent zest; do
|
||||
attr="${ATTR[$target]}"
|
||||
cpus=(); walls=()
|
||||
for i in $(seq 1 "$RUNS"); do
|
||||
start=$(now_ms)
|
||||
NIX_SHOW_STATS=1 nix eval --raw "$attr" --option eval-cache false \
|
||||
>/dev/null 2>/tmp/ralph-stats.json
|
||||
rc=$?
|
||||
end=$(now_ms)
|
||||
if [ $rc -ne 0 ]; then echo "$target run $i FAILED (rc=$rc)"; tail -8 /tmp/ralph-stats.json; exit 1; fi
|
||||
cpu=$(grep -o '"cpuTime": [0-9.]*' /tmp/ralph-stats.json | head -1 | grep -o '[0-9.]*')
|
||||
cpus+=("$cpu"); walls+=("$((end - start))")
|
||||
done
|
||||
medcpu=$(printf '%s\n' "${cpus[@]}" | median)
|
||||
medwall=$(printf '%s\n' "${walls[@]}" | median)
|
||||
medwall_s=$(awk "BEGIN{printf \"%.2f\", $medwall/1000}")
|
||||
printf '%-11s cpuTime=%ss (median of %d) | wall=%ss | cpu=[%s] wall_ms=[%s]\n' \
|
||||
"$target" "$medcpu" "$RUNS" "$medwall_s" "${cpus[*]}" "${walls[*]}"
|
||||
[ -n "$ONLY" ] && [ "$ONLY" != "$target" ] && continue
|
||||
NIX_SHOW_STATS=1 nix eval --raw "${ATTR[$target]}" --option eval-cache false \
|
||||
>/dev/null 2>/tmp/ralph-stats.json
|
||||
if [ $? -ne 0 ]; then echo "$target FAILED"; tail -8 /tmp/ralph-stats.json; exit 1; fi
|
||||
printf '%-11s thunks=%-10s calls=%-10s primops=%-9s | values=%s sets=%s\n' \
|
||||
"$target" "$(num nrThunks)" "$(num nrFunctionCalls)" "$(num nrPrimOpCalls)" \
|
||||
"$(num number)" "$(grep -o '"sets": {[^}]*"number": [0-9]*' /tmp/ralph-stats.json | grep -o '[0-9]*$')"
|
||||
done
|
||||
|
|
|
|||
|
|
@ -8,16 +8,22 @@ configs that get evaluated most often on this machine:
|
|||
|
||||
## Methodology
|
||||
|
||||
Primary metric: **`cpuTime`** from `NIX_SHOW_STATS=1` — it isolates pure
|
||||
evaluation work and is stable run-to-run (~3% spread). Wall-clock is reported
|
||||
too but is IO/cache-bound and noisy (±15%), so it is *not* used for commit
|
||||
decisions.
|
||||
Primary metric: **deterministic eval-work counters** from `NIX_SHOW_STATS=1` —
|
||||
`nrThunks`, `nrFunctionCalls`, `nrPrimOpCalls`. These count the actual evaluation
|
||||
work and are **byte-identical across runs** (independent of CPU clock, load, GC).
|
||||
|
||||
> **Why not `cpuTime`/wall?** Initially planned, but rejected after measuring:
|
||||
> this is a live, thermally-throttling laptop. The *same* eval (identical
|
||||
> nrThunks every run) reported cpuTime climbing 11s → 21s monotonically as the
|
||||
> CPU clock dropped under sustained load. Time is meaningless here; counters are
|
||||
> exact. A 1% drop in nrThunks is a real 1% less work the evaluator must do, on
|
||||
> any machine. Single eval per target — no median needed.
|
||||
|
||||
```sh
|
||||
# docs/eval-bench.sh <runs> (default 5)
|
||||
# docs/eval-bench.sh [target] target = pureintent | zest | (both)
|
||||
# - persistent $HOME=/tmp/ralph-evalhome keeps the fetcher/git cache warm
|
||||
# - --option eval-cache false forces a full re-eval every run
|
||||
# - reports median cpuTime + median wall over N runs, per target
|
||||
# - reports nrThunks / nrFunctionCalls / nrPrimOpCalls per target (deterministic)
|
||||
nix eval --raw .#nixosConfigurations.pureintent.config.system.build.toplevel.drvPath --option eval-cache false
|
||||
nix eval --raw '.#homeConfigurations."srid@zest".activationPackage.drvPath' --option eval-cache false
|
||||
```
|
||||
|
|
@ -25,25 +31,31 @@ nix eval --raw '.#homeConfigurations."srid@zest".activationPackage.drvPath'
|
|||
No `nix store gc` between runs (per request). The eval cache is disabled rather
|
||||
than cleared, so the warm fetcher cache removes input-fetch noise.
|
||||
|
||||
**Commit rule:** commit only if median cpuTime improves >3% for a target with no
|
||||
regression on the other. Behaviour may change *only* by dropping inputs/features
|
||||
first confirmed unused.
|
||||
**Commit rule:** commit only if `nrThunks` improves >3% for a target with no
|
||||
regression on the other. (Counters have ~zero noise, so any real reduction
|
||||
counts; >3% is the bar for a "meaningful" cycle.) Behaviour may change *only* by
|
||||
dropping inputs/features first confirmed unused.
|
||||
|
||||
## Baseline (median of 5)
|
||||
Profiling tool: `nix eval … --eval-profiler flamegraph --eval-profile-file f`
|
||||
then aggregate self/inclusive frames (collapsed-stack format).
|
||||
|
||||
| Target | cpuTime (s) | wall (s) | cpuTime runs |
|
||||
|------------|-------------|----------|--------------|
|
||||
| pureintent | **23.17** | 38.91 | 21.99 / 23.45 / 23.76 / 23.17 / 22.74 |
|
||||
| zest | **14.93** | 22.09 | 14.93 / 14.50 / 13.86 / 15.49 / 14.96 |
|
||||
## Baseline (nrThunks / nrFunctionCalls / nrPrimOpCalls)
|
||||
|
||||
Env: nix 2.34.7, x86_64-linux. Stats at baseline (pureintent): nrThunks 20.4M,
|
||||
nrFunctionCalls 13.0M, sets.bytes 868MB, values.bytes 503MB, gcFraction 0.058.
|
||||
| Target | nrThunks | nrFunctionCalls | nrPrimOpCalls |
|
||||
|------------|------------|-----------------|---------------|
|
||||
| pureintent | 20,400,114 | 12,975,268 | 6,507,025 |
|
||||
| zest | 11,066,680 | 7,091,428 | 3,587,586 |
|
||||
|
||||
Env: nix 2.34.7, x86_64-linux.
|
||||
|
||||
## Optimization log
|
||||
|
||||
| Cycle | Change | pureintent | zest | Verdict |
|
||||
|-------|--------|------------|------|---------|
|
||||
| 0 | baseline | 23.17 | 14.93 | — |
|
||||
(Δ% is on nrThunks vs baseline for the affected target.)
|
||||
|
||||
| Cycle | Change | pureintent thunks | zest thunks | Verdict |
|
||||
|-------|--------|-------------------|-------------|---------|
|
||||
| 0 | baseline | 20,400,114 | 11,066,680 | — |
|
||||
| 1 | pureintent: `documentation.nixos.enable = false` (drop NixOS options manual; option doc-strings no longer evaluated) | **19,201,996 (−5.9%)** | 11,066,680 | ✅ commit |
|
||||
|
||||
## Dead ends
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue