Skip to main content
valen
GitHub
EN中
valen
FROM PROBABILITY TO ACTION

Automate the clear cases. Review the rest.

Evaluate Valen for ticket intent routing: how much work can you automate at an acceptable error rate?

Held-out replay

Rules are frozen on validation data. The observed test result is shown alongside.

11.5%automated
0.7%observed accepted error95% CI 0.1%–3.8%
88.5%sent to review

Accepted 147 / 1276 questions; 1 error. A validation target is not a guarantee for test or deployment.

BANKING77 · English · 77 candidate intents · 1276 test questions · retrospective public-test replay.

Automated coverage vs. error rate
Automated coverage vs. error rateGray: original. Blue: calibrated. Download the report for exact values.0%0%25%25%50%50%75%75%100%100%Selected rule: 11.5% coverage, 0.7% errorAutomated coverage
OriginalCalibrated
Stated probability vs. observed accuracy
Stated probability vs. observed accuracyGray: original. Blue: calibrated. Download the report for exact values.0%0%25%25%50%50%75%75%100%100%7 questions; 11.6% → 0.0%13 questions; 17.4% → 15.4%24 questions; 23.3% → 37.5%51 questions; 30.4% → 17.6%48 questions; 37.3% → 20.8%97 questions; 43.4% → 29.9%92 questions; 50.2% → 44.6%114 questions; 56.5% → 47.4%109 questions; 63.3% → 55.0%86 questions; 69.7% → 57.0%90 questions; 76.5% → 83.3%123 questions; 83.0% → 78.9%137 questions; 89.8% → 78.1%285 questions; 97.4% → 94.0%1 questions; 6.3% → 0.0%8 questions; 10.3% → 0.0%26 questions; 16.6% → 30.8%40 questions; 23.8% → 25.0%60 questions; 30.1% → 18.3%88 questions; 37.2% → 27.3%92 questions; 43.3% → 38.0%118 questions; 50.3% → 44.9%121 questions; 56.8% → 55.4%89 questions; 63.1% → 58.4%99 questions; 69.9% → 85.9%111 questions; 76.6% → 76.6%118 questions; 83.1% → 80.5%116 questions; 90.2% → 85.3%189 questions; 96.5% → 98.4%Stated probability
OriginalCalibratedDashed: ideal calibration
Why trust these numbers? Compare calibration and inspect the scope
Probability distribution: how many questions fall in each range?
Sample distribution of top candidate probabilitiesOriginal 0–7%: 0Calibrated 0–7%: 1Original 7–13%: 7Calibrated 7–13%: 8Original 13–20%: 13Calibrated 13–20%: 26Original 20–27%: 24Calibrated 20–27%: 40Original 27–33%: 51Calibrated 27–33%: 60Original 33–40%: 48Calibrated 33–40%: 88Original 40–47%: 97Calibrated 40–47%: 92Original 47–53%: 92Calibrated 47–53%: 118Original 53–60%: 114Calibrated 53–60%: 121Original 60–67%: 109Calibrated 60–67%: 89Original 67–73%: 86Calibrated 67–73%: 99Original 73–80%: 90Calibrated 73–80%: 111Original 80–87%: 123Calibrated 80–87%: 118Original 87–93%: 137Calibrated 87–93%: 116Original 93–100%: 285Calibrated 93–100%: 1890%100%

Top probability exactly 1: 0 original, 0 calibrated.

Same held-out test; lower is better except accuracy.
MetricOriginalCalibrated
ECE0.08190.0518
NLL1.35641.3240
Brier0.50620.4996
Accuracy63.5%63.5%

Valen-0.8B · c2c481fcb319 · 1276 test questions

BANKING77 · English · 77 candidate intents · retrospective public-test replay. Not a production or vision guarantee.

ECE uses 15 equal-width bins and top candidate probability. Error bars are pointwise 95% Wilson intervals; correlated questions can make them too narrow.

Full report JSON.gz · 551 KBStudy temperature artifactDataset sourceStudy protocolSelected routing policy

Downloaded parameters are bound to this historical model and recipe. Refit and validate for your own workflow.

Try ticket routingEvaluate on my data

From a report to your application

  1. Split calibration, policy validation and test by customer, conversation or media source.
  2. Reuse raw predictions offline, fit temperature, freeze rules, then inspect test results.
  3. Load parameters for the same model; route low-probability or failed requests to review.
python -m valen.calibrate pipeline \
  --calibration-cache calibration.json \
  --policy-cache policy.json --test-cache test.json \
  --output evidence

This command processes local caches only. It does not download or invoke a model.

Probability and concentration confidence are distinct. Temperature fitting does not guarantee better ranking or automation.