Build a support router with the agent skill

Fastbite, a food-delivery app, needs every in-app support message routed to one of six queues, with a safety flag and a compensation flag. We gave a coding agent the Valen skill, a one-page brief and ten labeled examples, and nothing else. This page shows what it built and how it scored on 60 messages it never saw.

The brief and the prompt

The agent's folder held only task.md (queues, flags and the predict(item) interface), labeling.md (twelve business rules, such as "a message that raises two problems goes to human review") and dev.json (ten examples). The skill was installed at .claude/skills/valen/. The prompt:

Text

Read task.md and labeling.md in this folder and build the deliverable they describe.
1. Write your first complete version and freeze it in v1/ before running it on any labeled example.
2. Improve it using dev.json and cases you write yourself; leave the final version in solution.py.
3. Finish with NOTES.md: what changed between v1 and the final version, and why.
Use Valen's free hosted API at https://api.valen-team.workers.dev (no key needed).

To try it yourself, install the skill and give your agent the same kind of brief:

Bash

claude plugin marketplace add Valen-Team/skills
claude plugin install valen@valen-team

What the agent built

One System One request per message on valen-4b, with every judgment asked as a question and every business rule kept in code:

• State says what the text is: a context field ("In-app support message from a customer of Fastbite, a food-delivery app in Singapore…"), plus order_status and message.
• Queue: a Choice over the five queues plus a described none; none goes to human_review.
• Two problems: one Noul, "reports problems of two or more of these kinds", whose false criterion says that one problem told at length, or one problem plus rudeness, does not count.
• Flags: safety and compensation as Nouls with concrete true / false criteria.
• Guards: a describes_event Noul so near-empty messages ("???") cannot raise safety, and a delete_account Noul because rule 5 sends account deletion to review.
• Order and thresholds: the multi-problem check sits last so rewording it leaves earlier answers alone; safety uses 0.8 because messages with no problem sat at 0.5–0.6 while real reports scored above 0.95.

The agent tested on its own labeled cases and a held-out set it wrote after tuning, and recorded each change and its reason in NOTES.md.

Results on unseen messages

Scored on the 60 hidden test messages of the skill benchmark, using balanced accuracy (mean per-class recall) per field:

Version	Queue	Safety	Compensation	Mean	All three right
First version (v1/, before any labeled example)	94%	98%	100%	97.2%	–
Final version (this page's code)	95%	100%	99%	98.1%	56 / 60

A constant answer scores 39% on this task. The four messages it still gets wrong:

• "wow, only 70 minutes for a 2km delivery. amazing service as always": sarcasm sent to human_review instead of late_order.
• Peanuts in noodles ordered without them, with an allergic reaction: food_quality instead of missing_or_wrong_item (safety is right).
• "Cancelled the order within a minute but I was still charged": compensation raised, though rule 12 says an overcharge report without a request is false.
• A hacked account used for three card orders: account_access instead of human_review for two problems.

Across all three benchmark tasks (text routing, image sorting, video tagging), first versions written with the skill averaged 92.7% with Claude Code and 91.2% with Codex; without the skill they averaged 80.2% and 63.3%. With the skill, no first version scored below 86.8%. Without it, each agent produced one first version that failed outright: Claude Code thresholded a vaguely worded usable question at 0.5 and marked every image unusable, and Codex sent eight video frames in a request that accepts one. The benchmark README keeps the current numbers.

Run it

Python 3.11+ and the standard library only. Download the bundle or copy the code below:

Bash

curl -fLO https://valen-team.github.io/cookbook/valen-cookbook.zip
python -m zipfile -e valen-cookbook.zip .
cd valen-cookbook
python skill_support_router.py --message "Order was super late AND they forgot my drink" --order-status delivered

The answer for this message is recorded in caches/skill_support_router.json, so the command runs without network access. A new message is sent to the free hosted API and recorded; --replay off always calls it.

The script prints the route, the returned model_id, the question-set version and every answer. predict(item) is the function the benchmark grader calls.

Complete code

skill_support_router.py

"""Fastbite support router, built by a coding agent with the Valen skill.

The agent wrote questions.py, valen_client.py and solution.py from a one-page brief; this file joins
them unchanged and adds a command line. Valen makes the judgments; decide() applies the labeling rules.

python skill_support_router.py --message "Order was late AND they forgot my drink" --order-status delivered
"""

import argparse
import contextlib
import json
import time
import urllib.error
import urllib.request

# questions.py: all Valen question wording, versioned with the model ID.

MODEL = "valen-4b"
QUESTION_SET_VERSION = "v3"

CONTEXT = ("In-app support message from a customer of Fastbite, a food-delivery app in Singapore. "
           "Customers write in English or Chinese.")

QUEUE_CRITERIA = {
    "late_order": "The order has not arrived yet or arrived late, and the wait is the complaint or question",
    "missing_or_wrong_item": "Something ordered is missing, a different item came, or the order never arrived although marked delivered",
    "food_quality": "The food arrived but is cold, spilled, spoiled, undercooked, contains something it should not, or tastes wrong",
    "payment_refund": "Charges, double charges, a refund already promised, payment methods, or a voucher that failed at checkout",
    "account_access": "Login, verification codes or OTP, a locked or hacked account, changing phone number or email",
    "none": ("None of these: compliments or feedback about a restaurant or rider with nothing to fix, partnership or job "
             "enquiries, deleting an account, dangerous rider behaviour alone, greetings or messages with no clear request"),
}

# Order matters: earlier answers are most stable, so the multi-problem check goes last.
QUESTIONS = {
    "queue": {
        "type": "choice",
        "instructions": "Which problem does the customer's `message` report?",
        "criteria": QUEUE_CRITERIA,
    },
    "safety": {
        "type": "noul",
        "instructions": "The `message` reports a health or safety problem.",
        "criteria": {
            "true": "An allergic reaction, feeling sick after eating, a foreign object such as hair, plastic or metal in the food, raw or spoiled food, or a rider behaving dangerously",
            "false": "Food that is only cold, spilled, late, missing or wrong, a payment or account issue, or no problem",
        },
    },
    "compensation": {
        "type": "noul",
        "instructions": "The customer asks for money back, credit or a voucher.",
        "criteria": {
            "true": "The customer asks for a refund, money back, credit or a voucher, or chases a refund already promised",
            "false": "The customer only reports a problem, asks a question or asks for something else, without asking for money, credit or a voucher",
        },
    },
    "multi": {
        "type": "noul",
        "instructions": ("The `message` reports problems of two or more of these kinds: a late order, a missing or wrong item, "
                         "poor food quality, a payment problem, an account access problem."),
        "criteria": {
            "true": "Two or more of those kinds, such as a late order and a missing item, or a double charge and a cold meal",
            "false": ("Only one of those kinds, even if described in several sentences, with several affected items, with a refund "
                      "request for it, or alongside complaints about rudeness or remarks about the rider or restaurant; or none of them"),
        },
    },
    "delete_account": {
        "type": "noul",
        "instructions": "The customer wants to delete or permanently close their Fastbite account.",
        "criteria": {
            "true": "The customer asks to delete, close or deactivate their account for good",
            "false": "The customer wants to keep using the account, or the message is about something else",
        },
    },
    "describes_event": {
        "type": "noul",
        "instructions": "The `message` describes something specific that happened or is happening.",
        "criteria": {
            "true": "The customer tells about a specific event, situation or problem",
            "false": "Only a greeting, a single word, punctuation, or a message too short to say anything specific",
        },
    },
}

MULTI_THRESHOLD = 0.5
# A weaker multi signal still sends the message to review when the queue choice itself is split.
MULTI_BAND = 0.3
QUEUE_SPLIT_GAP = 0.2
# Safety drifts to 0.5-0.6 on messages with no problem at all (greetings, login issues); real reports score >0.95.
SAFETY_THRESHOLD = 0.8
DELETE_ACCOUNT_THRESHOLD = 0.5
# Safety needs a message that actually describes something; near-empty messages drift high on safety.
DESCRIBES_EVENT_THRESHOLD = 0.5
COMPENSATION_THRESHOLD = 0.5

# valen_client.py: minimal HTTP client for Valen's System One endpoint.

BASE_URL = "https://api.valen-team.workers.dev/v1/systemone"
HEADERS = {"Content-Type": "application/json", "User-Agent": "fastbite-support-router/1.0"}


def ask(model, state, questions, timeout=25, retries=3):
    payload = {"model": model, "state": state, "questions": questions, "temperature": 1.0}
    data = json.dumps(payload, ensure_ascii=False).encode("utf-8")
    last = None
    for attempt in range(retries):
        req = urllib.request.Request(BASE_URL, data=data, headers=HEADERS, method="POST")
        try:
            with urllib.request.urlopen(req, timeout=timeout) as resp:
                return json.loads(resp.read().decode("utf-8"))
        except urllib.error.HTTPError as e:
            last = RuntimeError(f"Valen HTTP {e.code}: {e.read()[:300]!r}")
            if e.code not in (429, 500, 502, 503, 504):
                raise last
        except (urllib.error.URLError, TimeoutError) as e:
            last = RuntimeError(f"Valen request failed: {e}")
        time.sleep(2 * (attempt + 1))
    raise last

# solution.py: the decision logic.


def build_state(item):
    return {
        "context": CONTEXT,
        "order_status": item.get("order_status", "none"),
        "message": item.get("message", ""),
    }


def has_words(message):
    """True when the message contains any letter, digit or CJK character (not only punctuation)."""
    return any(ch.isalnum() for ch in message)


def decide(answers, message=""):
    queue = answers["queue"]["choice"]
    multi = answers["multi"]["noul"]
    safety = (answers["safety"]["noul"] >= SAFETY_THRESHOLD
              and answers["describes_event"]["noul"] >= DESCRIBES_EVENT_THRESHOLD
              and has_words(message))
    compensation = answers["compensation"]["noul"] >= COMPENSATION_THRESHOLD
    top, second = sorted(answers["queue"]["probabilities"].values(), reverse=True)[:2]
    split = multi >= MULTI_BAND and top - second < QUEUE_SPLIT_GAP
    deleting = answers["delete_account"]["noul"] >= DELETE_ACCOUNT_THRESHOLD
    if queue == "none" or multi >= MULTI_THRESHOLD or split or deleting:
        queue = "human_review"
    return {"queue": queue, "safety": bool(safety), "compensation": bool(compensation)}


def predict(item: dict) -> dict:
    body = ask(MODEL, build_state(item), QUESTIONS)
    return decide(body["answers"], item.get("message", ""))


def main(argv=None):
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--message", required=True)
    parser.add_argument("--order-status", default="none", choices=["delivered", "on_the_way", "cancelled", "none"])
    parser.add_argument("--replay", choices=["auto", "only", "off"], default="auto",
                        help="auto: reuse recorded answers and record new ones; only: recorded answers only; "
                             "off: always call the model")
    args = parser.parse_args(argv)
    item = {"message": args.message, "order_status": args.order_status}
    request = {"state": build_state(item), "questions": QUESTIONS}
    if args.replay == "off":
        body = ask(MODEL, request["state"], QUESTIONS)
    else:
        from common import recorded
        live = lambda: contextlib.nullcontext(lambda r: ask(MODEL, r["state"], r["questions"]))
        with recorded(live, args.replay, MODEL, BASE_URL) as predict:
            body = predict(request)
    print(json.dumps({"route": decide(body["answers"], args.message), "model_id": body.get("model_id"),
                      "question_set": QUESTION_SET_VERSION, "answers": body["answers"]}, ensure_ascii=False, indent=2))


if __name__ == "__main__":
    main()

The agent wrote it as three files: questions.py (all wording and thresholds), valen_client.py and solution.py. They are joined here with a command line added; the logic is unchanged and scores the same on the hidden set.

Code uses Apache-2.0. Messages and labels are fictional and written for the benchmark.

Agent skill · Write questions that work · System One API · Route support tickets · When no option fits