Panel benchmarks and evaluations

Use the Measurement lab (/rust/bench/lab) to start measured benchmark runs, calibrate noise, compare live endpoints, check endpoint behavior, review signed qualification evidence, measure context-depth curves, test held-out quality, plan model launches and run bounded engine jobs. The native navigation links to each workflow. Statistics (/rust/statistics) and Evaluate (/rust/evaluate) remain available for other reports; the old /rust/bench route is not required for the native controls.

Benchmark results are evidence, not a guarantee that a different workload will have the same performance. An idle host, known model artifact, exact execution backend, matching context and KV types, fixed seed, warmups, repeat counts, thermal and power conditions, and honest uncertainty are all part of a defensible comparison. Use at least three repeats, report the median and spread, include 95% confidence intervals when supported by the measurement owner, and explicitly report missing or inconclusive results. Never substitute zero for missing samples.

1. Live benchmark runs, calibration and traffic

Open Benchmark runs (/rust/bench/runs) to read the active bench job, saved run IDs, traffic noise and calibration measurements. The first owner snapshot is server-rendered. While the tab is visible, a small read-only module refreshes benchmark, traffic and calibration JSON about every six seconds; Refresh owner snapshots works without scripting. A refusal, missing instance or unmeasured sample remains an error or missing observation, never a fabricated speed number. Existing run records link to GET /api/bench/run?id=...; administrators can also enter a known run ID in Read saved run in Panel.

Administrators must check I confirm the host is ready and authorize this action on each mutation. Instance ID can be empty to select the active instance, or explicitly identify a live instance. Depth preset is quick or workload, Samples per depth is 2–10, Decode tokens is 16–1024, and Decision margin (%) is 0.2–50. The four actual owner jobs are:

• Profile current settings: measures without requesting a restart; optional Sampled-acceptance runs 0–20.
• Calibrate identical settings: 2–24 blocks; Restart between blocks explicitly allows an interruption and must be used only with an idle instance.
• Compare saved settings with a candidate: enter a separate KEY=value per line. This job requires an allowed server restart to switch settings and restore the saved configuration; it does not silently persist candidate settings.
• Measure concurrent request goodput: provide unique stream counts (1–32), a context depth (512–262144 tokens, or a whole-number k suffix), maximum first-token delay and minimum decode tokens/s.

Translate an existing launch command submits text only to the command-settings parser; it does not execute the supplied string. Stop active benchmark requests a safe stop. Permanently delete run requires an exact saved run ID and administrator confirmation. An incomplete run or a missing instance produces a real owner error. The backed routes are GET /api/bench, GET /api/bench/traffic, GET /api/bench/calibration, GET /api/bench/run, POST /api/bench/start, POST /api/bench/stop, POST /api/bench/delete, and POST /api/params/from-cmdline.

2. Endpoint behavior specification

Open Endpoint checks (/rust/bench/specification). Available endpoints uses GET /api/compare/endpoints; Latest specification checks reads GET /api/engines/spec. The administrator-only Run endpoint specification checks form populates checkbox IDs from the actual endpoint inventory. Select one or more eligible endpoints, approve the action and choose Start specification tests to reach POST /api/engines/spec. These check request, streaming and response behavior, not benchmark superiority. The owner rejects unavailable IDs and overlapping runs.

3. Qualification receipts and integrity

Open Qualification (/rust/bench/qualifications). Existing evidence and sealed exports displays the actual GET /api/qualify result with verification state. Export original JSON links to the owner's GET /api/qualify/export?id=... download, which only succeeds when a digest is present. Imported digest verification cannot independently prove who created a record, and is not a cryptographic signature or a release decision.

Administrators can Compare sealed records by entering two different IDs (GET /api/qualify/compare via the server-side handler), Import and verify digest by pasting a complete exported JSON object (POST /api/qualify/import, 2 MiB maximum), or Permanently delete record by exact ID (POST /api/qualify/delete). The owner checks imported seals and refuses invalid or edited receipts. No form repairs evidence or manufactures a passing qualification.

4. Fair benchmark comparison

1. Open Fair benchmark comparison (/rust/bench/compare). Review Available endpoint IDs and Recent measured comparisons.
2. In Builds, install the latest available upstream reference build and note its exact version. The comparison page does not download, update or attest the reference build for you. Match model bytes, quantization, backend, context window, cache policy and hardware before declaring a fair comparison.
3. Quiesce other workloads, use the same task set and output-token limit, and check both endpoints are reachable. Clear or control caches consistently. Keep the host power and temperature policy fixed.
4. Enter Candidate endpoint ID, Reference endpoint ID, Task set, Repeated rounds (3 to 5) and Maximum output tokens; check the authorization box and choose Start paired comparison.
5. For expanded tests, use Compare multiple endpoints with hard requirements. Choose up to eight live endpoint IDs; select the task set, score profile, 1–5 rounds, output limit, optional temperature, generated-task count and seed, or attention depths and tasks per depth. Optional hard limits reject endpoints below a minimum pass rate, decode speed or context capacity, or above a maximum first-token latency or error rate. No blank threshold is converted to a passing result.
6. Refresh Recent measured comparisons to view recent runs, or enter a saved ID in Inspect a saved run and choose Read saved comparison to inspect that run's JSON record. Use the owner-provided pass rates, latency, confidence intervals, identity fields, partial rows and errors. A nominal lead with overlapping uncertainty is inconclusive, not a win. Stop comparison takes a Run ID and stops at a safe boundary.

The live endpoint comparison routes are GET /api/compare/endpoints, POST /api/compare/start, GET /api/compare/run, and POST /api/compare/stop. The underlying endpoint owner pairs graded tasks and shuffles endpoint ordering across rounds. However, it does not independently attest every launch option or artifact identity. A same-model, same-quantization claim requires separately retained configuration evidence.

For a command-line head-to-head experiment, the engine also supports lexi-engine bench lx35-llama-compare with explicit model and binary paths, lane, context, tokens and depth settings. Its verification limits and output must be read separately from the Panel endpoint comparison; it is not automatically equivalent to it.

5. Decode depth curves

1. Start the desired instance and verify it is idle, including its active slots. Open Decode depth (/rust/bench/depth) and review Existing depth curves.
2. Set Depths in tokens, separated by commas (default 512,2048,8192), Repeated runs (3 to 10) (default 3), and Decode tokens per depth (default 128).
3. Check the authorization box and choose Start depth measurements. Refresh to follow owner progress and read individual depth buckets. The owner refuses conflicting work and enforces its corpus/context limits.
4. Use Stop current curve when necessary. Treat partial results as partial, and compare long contexts against the same reference depth.

The routes are GET /api/curve, POST /api/curve/start and POST /api/curve/stop. Live traffic and calibration are on Benchmark runs. Statistics reports observed prefill, decode and context buckets; these are not artificial benchmark numbers.

6. Evaluation quality

1. Open Evals (/rust/bench/evals), then inspect Evaluation capabilities and Evaluation history.
2. For held-out task sets, supply Candidate instance ID, Baseline instance ID, Held-out task JSON path, Repeat count, Seed, and Timeout (seconds); choose Run task evaluation. Both instances must be explicit and distinct. The form submits a background evaluation job and returns its job record; refresh history or use the existing jobs interface to check completion.
3. For perplexity and KL-divergence, supply Candidate GGUF path, Baseline GGUF path, Perplexity executable path, Held-out corpus path, and the same repeat/seed/timeout settings; choose Run perplexity evaluation. This path is CPU-bound and can require substantial disk for intermediate logits.
4. Look at the qualification result and interval before deciding whether a speed change is acceptable. A faster run with an unverified or regressed quality result must not be promoted solely for throughput.

Routes: GET /api/evals/capabilities, POST /api/evals/run, GET /api/evals/history.

7. Plan a model launch for this hardware

1. Open Launch planning (/rust/bench/planning) and read Installed inference capabilities. The capability inventory is GET /api/inference/capabilities.
2. Enter the Absolute model path in Read-only model probe and choose Probe model. This goes to GET /api/inference/probe?path=... and opens the JSON response. Check the recognized format, architecture, candidate runtimes and reasons a runtime may be unavailable.
3. With administrator access, enter Absolute model path, optional Runtime ID, Requested context tokens and optional GPU layers under Preview launch command; choose Preview launch. The result of POST /api/inference/plan is a proposed executable, argument array, environment and warnings. It does not start the model.
4. Under Estimate GGUF fit on this hardware, provide Absolute GGUF model path, Backend, GPU memory per device (MiB, comma separated), Available RAM (MiB), CPU threads, Context tokens, Concurrency, Reserve GPU memory (MiB) and KV cache type. Choose Estimate fit to call POST /api/inference/tune-gguf.
5. Read estimated KV size and VRAM headroom, warnings and launch parameters. Estimates use values supplied in the form, not an independently measured device reservation; verify actual available memory and start an instance through the normal instance workflow only after inspection.

These four inference routes have their original owners. The new forms neither invent a runtime nor bypass a route failure. This page works with no configured instance for capability and model inspection.

8. Engine command jobs

Open Engine jobs (/rust/bench/tools) as an administrator. The Panel runs typed arguments; it never accepts a shell command string or a user-supplied executable path. The service selects the installed engine executable. Each job has a persistent ID and status, a 15-minute execution limit, a combined 2 MiB output limit, and a readable output excerpt capped at 64 KiB per stream. If a job exits nonzero, times out, is cancelled or exceeds the output limit, the record reports that failure. Job outputs may contain sensitive information; they are not shown to viewers.

• Benchmark local model: Absolute GGUF path, Depths (tokens), Repeated runs (3–10), Decode tokens; Run engine benchmark starts lexi-engine bench --model ... --depths ... --n-gen ... --repetitions ... --output json with warmup enabled by default.
• Inspect topology: optional Refresh (yes or blank); Inspect topology starts lexi-engine topology --json with optional --refresh.
• Preview lexification placement: Absolute GGUF path, Backend, optional Device IDs; Plan placement starts lexi-engine lexify-plan --model ... --json --backend ... and optional --devices.
• Quantize an artifact: Absolute GGUF source path, Target type; Start quantization starts lexi-engine quantize INPUT OUTPUT TYPE. The destination is generated inside the private job folder; the source is unchanged. Verify quality before using the new artifact.
• Capture activations (sensitive): Local engine port, Absolute request JSON path, Absolute private operator-key file path; Start capture starts lexi-engine capture-activations --port ... --request ... --operator-key-file ... --output .... This requires a separately configured, authenticated local serving instance with activation collection enabled.

Every start needs explicit confirmation. Saved engine jobs and output lists recent results. To stop an active job, supply Engine job ID and choose Request job stop. Results persist on disk; a server restart can interrupt a running job, so verify the status before treating its output as complete. Engine-job routes: GET /api/ux/bench/tools, POST /api/ux/bench/tools/start, POST /api/ux/bench/tools/stop. The GET status route accepts an optional id query parameter.

Controls and help text

Every native Bench control uses its registered tooltip. The table below is drawn from the same labels and explanatory text as the Bench feature registry.

For guided step-by-step work available directly in this area, open /rust/bench/guides, then choose Profile and calibrate a running instance, Check behavior across endpoints, Review and compare qualification evidence, Benchmark fairly against the newest reference build, Measure a decode depth curve, Plan a model launch for this hardware, or Run a local engine benchmark. These guides require administrator privileges; each step confirms prerequisites, substitutes only inputs actually referenced by the step into a typed request, checks the owner response, and offers undo only where an owner exposes a reversible action. Read-only inventory and manual review steps can be exercised without a model; a measurement step needs a real idle instance or endpoint and may legitimately fail. The shared guides index (/rust/guides) may also link to them when its runner is installed. You can also review Panel Instances, Panel Builds and Engines, and Panel Fit Lab.