Benchmark protocols

Project overview · Running evaluations · Model integrations

Eight Jev-related benchmark families are implemented, exposed through ten TSV views. The Image/Video views are subdivisions of VisualDecisionBench, not additional benchmark families.

Family	Evaluation data	Scoring
VisualDecisionBench	Full released eval.jsonl, 1,696 shared states, 9,143 questions: 2,000 image and 7,143 video	Hard-label accuracy; soft-label NLL/Brier; ordinal MAE/RPS; modality/type/domain breakdowns
JevBench	231 frozen public tasks: 48 easy, 72 original, 111 hard	Vendored official task scorer, calibration and family/paraphrase reports
Decision Index	Edition 0.3 public suite, 140,178 requests after native subset/exclusion rules	Upstream edition-specific, chance-corrected weighted public index; 43 constituent datasets
Typed-Decisions	Pinned all/test: 400 shared states, 2,000 questions	Explicit modal-label accuracy; soft-reference KL, Brier and cross-entropy
DecisionBench	Pinned standard eval: 23,900 rows, three primitives	All-row accuracy; candidate-aligned NLL and calibration; original ordinal values
Procedural Typed Decisions	Pinned all/test: 18,000 states, 57,016 questions	Exact rule judgments; pure posterior questions use probability error without invented hard labels
zh-decision-bench	Pinned v0.2 public validation: 378 states, 467 questions	Chinese classification/rating; right-closed 15-bin reference ECE
JevAlt-Bench	Pinned English/Turkish/German test files: 3,116 states, 8,434 questions	Modal accuracy; hard and soft probability scores; language slices and equal-mass reference ECE

Each benchmark directory contains preparation, scoring and a manifest.json pinning TSV hashes, counts and protocols. Download the released data using the usage guide.

JevBench specifically means fstandhartinger/jevbench, revision bb05a335bc809e61b20c0f745d25499a82b326fc, used by Valen's public comparison. Other projects named JevBench use different tasks and cannot share these scores. The original task's custom true/false rubric is preserved for Noul inference.

Decision Index preserves question IDs, options, order, structured state, gold labels, grouping, official exclusions and the frozen suite's source metadata. Unlabeled diagnostic questions remain in the request and carry an explicit {"unscored": true} target. They do not contribute to probability or accuracy metrics. The official scorer is required for the headline weighted index; generic field accuracy is a separate diagnostic, not a replacement.

Inference never receives gold labels, rationales, evaluation IDs or source metadata. Native runtimes retain their published temperatures and context limits. Models which cannot represent a modality or question type report it as unsupported. Their failures remain visible in coverage and accuracy.

All probabilities must be finite, nonnegative, within [0,1], with complete candidate coverage. Generic metrics allow at most 0.01 rounding error in the sum, recording the raw answer and normalizing only for generic metrics. The official JevBench scorer retains its own published strict/rounded tolerances and lexical argmax tie rule. Its official aggregates are retained separately.

NLL floors probabilities at 1e-12; multiclass Brier is the sum of squared errors; ECE uses 15 equal-width confidence bins; ordinal RPS averages squared cumulative errors over the first K−1 levels. Soft targets without an explicit classification label have no hard-label accuracy. When a source provides both a label and a distribution, accuracy/ECE use the label while proper probability scores use the complete distribution. Per-request synchronized latency includes preprocessing and native inference, excludes model loading/warmup, and is counted once per multi-question request.

Dataset/provider licenses and usage restrictions remain in force. The toolkit uses the released evaluation snapshots and their documented source licenses; see third-party terms.

Additional decision-suite protocols

The TSV snapshots preserve complete shared requests and the source's candidate order. Hash checks reject changed data before inference.

• Typed-Decisions: decode JSON states, preserve all five questions in one request, and retain both explicit labels and normalized soft golds. Teacher annotations measure teacher agreement. General and workflow-finetuned models have different exposure; compare them with that distinction visible.
• DecisionBench: use the upstream System One projection: the first binary candidate maps to Noul true; Choice keeps original candidate IDs; Score sorts by original ordinal value and stores the inverse ID mapping. Scoring restores original candidate order for ties and original numeric values for ordinal errors. Compact fields are not used. The reference NLL floor is 1e-15; canonical generic NLL retains its 1e-12 floor.
• Procedural Typed Decisions: preserve every task, difficulty and question. Pure probabilistic Noul targets remain soft: they contribute probability losses and posterior MAE, not fabricated hard accuracy or empirical ECE. The pinned 18-task release differs from older RSI evaluation snapshots.
• zh-decision-bench: read the published v0.2 payload (323 routing and 55 business items), map Chinese Score labels to their supplied level positions, and retain the source's true,false tie order. The Hub card has stale count tables; verified payload counts are 378 items / 467 questions. This is its public validation set; no test-time temperature refit is performed.
• JevAlt-Bench: use file membership for the test split despite placeholder row fields saying train. The published input abstain flag adds an explicit unknown Choice candidate; gold labels never determine model input. This common interface is an explicitly tagged projection of the author's API abstention flag. Dictionary Score rubrics retain insertion order and label mappings. Extra upstream question fields are serialized into instructions. These interface choices must be matched before comparing author scores.

metrics contains common JevalKit measurements. For these five suites the historically named official field contains a JevalKit reference protocol report, identified by scorer=jevalkit-decision-protocol-v1; it does not imply execution by an upstream harness. Reports include task/domain/language/difficulty slices, label-only NLL/Brier, soft-target KL/Brier, and the applicable reference ECE. JevBench and Decision Index continue to invoke their benchmark-owned scorers.

Add a benchmark

Create jevalkit/datasets/<benchmark>/ with __init__.py and manifest.json. Implement prepare(root=None, source=None, **options) and, when required, evaluate(dataset, records, predictions, output) in that package. Register the package name, family and modality in jevalkit/datasets/__init__.py; the CLI and runner dispatch through that entry. Use _shared.commit_dataset for validated TSVs and add fixtures checking the source projection and scoring rules.

---

Synchronized from JevalKit / docs/benchmarks.md · Source blob f3f8916 · 2026-10-19.