用 agent skill 做客服分流

外卖应用 Fastbite 需要把每条应用内客服消息分到六个队列之一，并打上 safety（安全）和 compensation（补偿）两个标记。我们只给了编程 agent 三样东西：Valen skill、一页需求说明和 10 条标注样本。本页展示它做出了什么，以及在 60 条它从没见过的消息上的得分。

需求和提示词

agent 的工作目录里只有 task.md（队列、标记和 predict(item) 接口）、labeling.md（12 条业务规则，例如"同时提出两个问题的消息转人工"）和 dev.json（10 条样本）。skill 装在 .claude/skills/valen/。提示词如下：

Text
Read task.md and labeling.md in this folder and build the deliverable they describe.
1. Write your first complete version and freeze it in v1/ before running it on any labeled example.
2. Improve it using dev.json and cases you write yourself; leave the final version in solution.py.
3. Finish with NOTES.md: what changed between v1 and the final version, and why.
Use Valen's free hosted API at https://api.valen-team.workers.dev (no key needed).

自己试一试：安装 skill，再给你的 agent 一份类似的需求说明。

Bash
claude plugin marketplace add Valen-Team/skills
claude plugin install valen@valen-team

agent 做了什么

每条消息在 valen-4b 上发一次 System One 请求。所有判断都写成问题，所有业务规则都留在代码里：

state 说明文本是什么：一个 context 字段（"新加坡外卖应用 Fastbite 的应用内客服消息……"），再加上 order_status 和 message。
队列：五个队列加一个有描述的 none 组成 Choice；none 转为 human_review。
两个问题：一个 Noul，问"是否提到两类及以上的问题"；它的 false 描述写明，一个问题讲了很多句，或者一个问题加上抱怨态度，都不算两个问题。
标记：safety 和 compensation 都是 Noul，true / false 写成具体情形。
防护：加一个 describes_event Noul，让"???"这类几乎为空的消息不会触发 safety；再加一个 delete_account Noul，因为规则 5 要求删号转人工。
顺序和阈值：多问题检查放在最后，改它的措辞不会影响前面的回答；safety 用 0.8，因为没有问题的消息落在 0.5–0.6，真正的安全报告都在 0.95 以上。

agent 用自己标注的样本，以及调参后才写的留出集做测试，并在 NOTES.md 里记下每次修改和原因。

在没见过的消息上的结果

在 skill 评测集的 60 条隐藏测试消息上打分，每个字段用 balanced accuracy（各类召回率的平均）：

版本	队列	安全	补偿	平均	三项全对
首版（`v1/`，看任何标注样本之前）	94%	98%	100%	97.2%	–
终版（本页代码）	95%	100%	99%	98.1%	56 / 60

常数答案在这个任务上得 39%。终版仍然判错的 4 条：

"wow, only 70 minutes for a 2km delivery. amazing service as always"：讽刺句被转到 human_review，应为 late_order。
要求不放花生却放了花生、并出现过敏：判成 food_quality，应为 missing_or_wrong_item（safety 判对了）。
"Cancelled the order within a minute but I was still charged"：compensation 判为 true，但按规则 12，只报告多扣款、没有提出要求时应为 false。
账号被盗并被用来下了三单：判成 account_access，按"两个问题"规则应为 human_review。

在评测集的三个任务上（文本分流、图片分拣、视频打标签），带 skill 写出的首版平均分：Claude Code 92.7%，Codex 91.2%；不带 skill 时分别是 80.2% 和 63.3%。带 skill 的首版最低也有 86.8%。不带 skill 时，两个 agent 各有一次首版彻底失败：Claude Code 对措辞含糊的 usable 问题直接用 0.5 做阈值，把所有图片判成不可用；Codex 在一个请求里塞了 8 帧视频，而接口每次只收一张图。最新数字见评测集 README。

运行

只需 Python 3.11+ 和标准库。下载示例包，或直接复制下面的代码：

Bash
curl -fLO https://valen-team.github.io/cookbook/valen-cookbook.zip
python -m zipfile -e valen-cookbook.zip .
cd valen-cookbook
python skill_support_router.py --message "晚了一个小时，而且鸡翅少了两个" --order-status delivered

这条消息的回答已录制在 caches/skill_support_router.json，不联网也能运行。换成新消息时会调用免费托管 API 并录制下来；加 --replay off 则每次都调用。

脚本会输出分流结果、返回的 model_id、问题集版本和每个问题的回答。predict(item) 就是评测打分时调用的函数。

完整代码

skill_support_router.py
"""Fastbite support router, built by a coding agent with the Valen skill.

The agent wrote questions.py, valen_client.py and solution.py from a one-page brief; this file joins
them unchanged and adds a command line. Valen makes the judgments; decide() applies the labeling rules.

python skill_support_router.py --message "Order was late AND they forgot my drink" --order-status delivered
"""

import argparse
import contextlib
import json
import time
import urllib.error
import urllib.request

# questions.py: all Valen question wording, versioned with the model ID.

MODEL = "valen-4b"
QUESTION_SET_VERSION = "v3"

CONTEXT = ("In-app support message from a customer of Fastbite, a food-delivery app in Singapore. "
           "Customers write in English or Chinese.")

QUEUE_CRITERIA = {
    "late_order": "The order has not arrived yet or arrived late, and the wait is the complaint or question",
    "missing_or_wrong_item": "Something ordered is missing, a different item came, or the order never arrived although marked delivered",
    "food_quality": "The food arrived but is cold, spilled, spoiled, undercooked, contains something it should not, or tastes wrong",
    "payment_refund": "Charges, double charges, a refund already promised, payment methods, or a voucher that failed at checkout",
    "account_access": "Login, verification codes or OTP, a locked or hacked account, changing phone number or email",
    "none": ("None of these: compliments or feedback about a restaurant or rider with nothing to fix, partnership or job "
             "enquiries, deleting an account, dangerous rider behaviour alone, greetings or messages with no clear request"),
}

# Order matters: earlier answers are most stable, so the multi-problem check goes last.
QUESTIONS = {
    "queue": {
        "type": "choice",
        "instructions": "Which problem does the customer's `message` report?",
        "criteria": QUEUE_CRITERIA,
    },
    "safety": {
        "type": "noul",
        "instructions": "The `message` reports a health or safety problem.",
        "criteria": {
            "true": "An allergic reaction, feeling sick after eating, a foreign object such as hair, plastic or metal in the food, raw or spoiled food, or a rider behaving dangerously",
            "false": "Food that is only cold, spilled, late, missing or wrong, a payment or account issue, or no problem",
        },
    },
    "compensation": {
        "type": "noul",
        "instructions": "The customer asks for money back, credit or a voucher.",
        "criteria": {
            "true": "The customer asks for a refund, money back, credit or a voucher, or chases a refund already promised",
            "false": "The customer only reports a problem, asks a question or asks for something else, without asking for money, credit or a voucher",
        },
    },
    "multi": {
        "type": "noul",
        "instructions": ("The `message` reports problems of two or more of these kinds: a late order, a missing or wrong item, "
                         "poor food quality, a payment problem, an account access problem."),
        "criteria": {
            "true": "Two or more of those kinds, such as a late order and a missing item, or a double charge and a cold meal",
            "false": ("Only one of those kinds, even if described in several sentences, with several affected items, with a refund "
                      "request for it, or alongside complaints about rudeness or remarks about the rider or restaurant; or none of them"),
        },
    },
    "delete_account": {
        "type": "noul",
        "instructions": "The customer wants to delete or permanently close their Fastbite account.",
        "criteria": {
            "true": "The customer asks to delete, close or deactivate their account for good",
            "false": "The customer wants to keep using the account, or the message is about something else",
        },
    },
    "describes_event": {
        "type": "noul",
        "instructions": "The `message` describes something specific that happened or is happening.",
        "criteria": {
            "true": "The customer tells about a specific event, situation or problem",
            "false": "Only a greeting, a single word, punctuation, or a message too short to say anything specific",
        },
    },
}

MULTI_THRESHOLD = 0.5
# A weaker multi signal still sends the message to review when the queue choice itself is split.
MULTI_BAND = 0.3
QUEUE_SPLIT_GAP = 0.2
# Safety drifts to 0.5-0.6 on messages with no problem at all (greetings, login issues); real reports score >0.95.
SAFETY_THRESHOLD = 0.8
DELETE_ACCOUNT_THRESHOLD = 0.5
# Safety needs a message that actually describes something; near-empty messages drift high on safety.
DESCRIBES_EVENT_THRESHOLD = 0.5
COMPENSATION_THRESHOLD = 0.5

# valen_client.py: minimal HTTP client for Valen's System One endpoint.

BASE_URL = "https://api.valen-team.workers.dev/v1/systemone"
HEADERS = {"Content-Type": "application/json", "User-Agent": "fastbite-support-router/1.0"}


def ask(model, state, questions, timeout=25, retries=3):
    payload = {"model": model, "state": state, "questions": questions, "temperature": 1.0}
    data = json.dumps(payload, ensure_ascii=False).encode("utf-8")
    last = None
    for attempt in range(retries):
        req = urllib.request.Request(BASE_URL, data=data, headers=HEADERS, method="POST")
        try:
            with urllib.request.urlopen(req, timeout=timeout) as resp:
                return json.loads(resp.read().decode("utf-8"))
        except urllib.error.HTTPError as e:
            last = RuntimeError(f"Valen HTTP {e.code}: {e.read()[:300]!r}")
            if e.code not in (429, 500, 502, 503, 504):
                raise last
        except (urllib.error.URLError, TimeoutError) as e:
            last = RuntimeError(f"Valen request failed: {e}")
        time.sleep(2 * (attempt + 1))
    raise last

# solution.py: the decision logic.


def build_state(item):
    return {
        "context": CONTEXT,
        "order_status": item.get("order_status", "none"),
        "message": item.get("message", ""),
    }


def has_words(message):
    """True when the message contains any letter, digit or CJK character (not only punctuation)."""
    return any(ch.isalnum() for ch in message)


def decide(answers, message=""):
    queue = answers["queue"]["choice"]
    multi = answers["multi"]["noul"]
    safety = (answers["safety"]["noul"] >= SAFETY_THRESHOLD
              and answers["describes_event"]["noul"] >= DESCRIBES_EVENT_THRESHOLD
              and has_words(message))
    compensation = answers["compensation"]["noul"] >= COMPENSATION_THRESHOLD
    top, second = sorted(answers["queue"]["probabilities"].values(), reverse=True)[:2]
    split = multi >= MULTI_BAND and top - second < QUEUE_SPLIT_GAP
    deleting = answers["delete_account"]["noul"] >= DELETE_ACCOUNT_THRESHOLD
    if queue == "none" or multi >= MULTI_THRESHOLD or split or deleting:
        queue = "human_review"
    return {"queue": queue, "safety": bool(safety), "compensation": bool(compensation)}


def predict(item: dict) -> dict:
    body = ask(MODEL, build_state(item), QUESTIONS)
    return decide(body["answers"], item.get("message", ""))


def main(argv=None):
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--message", required=True)
    parser.add_argument("--order-status", default="none", choices=["delivered", "on_the_way", "cancelled", "none"])
    parser.add_argument("--replay", choices=["auto", "only", "off"], default="auto",
                        help="auto: reuse recorded answers and record new ones; only: recorded answers only; "
                             "off: always call the model")
    args = parser.parse_args(argv)
    item = {"message": args.message, "order_status": args.order_status}
    request = {"state": build_state(item), "questions": QUESTIONS}
    if args.replay == "off":
        body = ask(MODEL, request["state"], QUESTIONS)
    else:
        from common import recorded
        live = lambda: contextlib.nullcontext(lambda r: ask(MODEL, r["state"], r["questions"]))
        with recorded(live, args.replay, MODEL, BASE_URL) as predict:
            body = predict(request)
    print(json.dumps({"route": decide(body["answers"], args.message), "model_id": body.get("model_id"),
                      "question_set": QUESTION_SET_VERSION, "answers": body["answers"]}, ensure_ascii=False, indent=2))


if __name__ == "__main__":
    main()

agent 原本写成三个文件：questions.py（全部措辞和阈值）、valen_client.py 和 solution.py。这里把它们合成一个文件并加上命令行入口；逻辑没有改动，在隐藏测试集上得分相同。

代码采用 Apache-2.0。消息和标签都是为评测编写的虚构内容。

Agent skill · 写好问题 · System One API · 工单分流 · 候选项都不对

需求和提示词​

agent 做了什么​

在没见过的消息上的结果​

运行​

完整代码​

需求和提示词

agent 做了什么

在没见过的消息上的结果

运行

完整代码