Calibrate ticket routing

Fit a review rule using your own labeled tickets, then return automatic or human_review for each request. This native-model example uses three candidate teams: billing, technical and account.

Prepare

Use a Valen source environment containing the calibration tools and an already downloaded native model. See native inference for environment setup. Install your local Valen source checkout with the command below (/path/to/Valen contains pyproject.toml); valen-sdk alone does not provide these native calibration tools. Download the Cookbook source bundle, then extract it:

Bash

python -m pip install -e /path/to/Valen
python -m zipfile -e valen-cookbook.zip .
cd valen-cookbook

The bundle contains the maintained script and sample JSON files, without weights or media. All commands below run in that directory, using the same Valen environment.

Prepare tickets/calibration.jsonl, tickets/policy.jsonl and tickets/test.jsonl in the labeled record format. Each record needs a stable source group_id, its request and labeled targets. Keep customers/conversations together and all three splits separate from training. Use the same candidate meanings and order as your application; calibration_protocol.json records the workflow versions.

Collect once, fit offline

Bash

for role in calibration policy test; do
  python -m valen.reliability_collect \
    --model-dir /models/Valen-0.8B --device cuda \
    --data tickets/$role.jsonl --role "$role" \
    --protocol calibration_protocol.json \
    --group-namespace support-customers-v1 --output cache/$role.json
done

python -m valen.calibrate pipeline \
  --calibration-cache cache/calibration.json \
  --policy-cache cache/policy.json --test-cache cache/test.json \
  --output output/reliability

Collection reads local weights and retains raw logits. Fitting and report generation run offline without model retraining. The output includes saved calibration, routing policies and a test report showing accepted work, errors and review volume. policy-2.json corresponds to the default 5% validation target; inspect the actual test result before adopting it.

Apply the saved rule

Bash

python calibrated_ticket_routing.py \
  --model-dir /models/Valen-0.8B --device cuda \
  --request calibration_request.json \
  --protocol calibration_protocol.json \
  --calibration output/reliability/calibration.json \
  --policy output/reliability/policy-2.json

The script uses valen.routing.route_choice, checks the model/calibration/policy association and proposes a route. Low probabilities, inference failures or an accept-nothing policy send the ticket to review. It does not execute account changes. Fit parameters for this workflow; the published BANKING77 artifact belongs to a different 77-intent task.

Complete maintained script

calibrated_ticket_routing.py

"""Route tickets with fitted parameters and a frozen review policy. Apache-2.0."""
import argparse
import json


def main(argv=None):
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--model-dir", required=True)
    parser.add_argument("--model-id")
    parser.add_argument("--revision")
    parser.add_argument("--device", default="cuda")
    parser.add_argument("--request", required=True)
    parser.add_argument("--protocol", required=True)
    parser.add_argument("--calibration", required=True)
    parser.add_argument("--policy", required=True)
    args = parser.parse_args(argv)
    from valen.evaluation.calibration import load_calibration, read_json
    from valen.reliability_collect import LocalRelease
    from valen.routing import route_choice
    model = LocalRelease(args.model_dir, read_json(args.protocol), model_id=args.model_id,
                         revision=args.revision, device=args.device)
    calibration = load_calibration(args.calibration, model.identity)
    policy = read_json(args.policy)
    request = read_json(args.request)
    questions = request["questions"]
    if len(questions) != 1 or next(iter(questions.values()))["type"] != "choice":
        raise ValueError("This example expects one Choice question")
    qid = next(iter(questions))
    try:
        keys, logits = model.logits(request)[qid]
    except Exception as exc:
        result = {"route": "human_review", "reason": "inference_failed", "error_type": type(exc).__name__}
    else:
        result = route_choice(keys, logits, calibration, policy)
    print(json.dumps(result, ensure_ascii=False, allow_nan=False))


if __name__ == "__main__":
    main()

Workflow declaration and sample request

calibration_protocol.json

{
  "application": "ticket-intent-v1",
  "candidate_schema": "billing-technical-account-v1",
  "prompt_version": "ticket-routing-v1",
  "label_policy": "human-adjudicated-intent-v1",
  "media_budget": "text-only",
  "question_grouping": "one Choice question per ticket"
}

calibration_request.json

{
  "state": "I was charged twice for the same subscription.",
  "questions": {
    "intent": {
      "type": "choice",
      "instructions": "Which team should review this ticket?",
      "criteria": {
        "billing": "Payments, charges, refunds and invoices",
        "technical": "Software errors and service availability",
        "account": "Profile, sign-in and account access"
      }
    }
  }
}

The request is a schema example, not an evaluation dataset. The script and both JSON files above are imported from the same files distributed in the Cookbook bundle.

Published-model evidence · Calibration overview · Metric and cache protocol