对检索到的段落重新排序
一个共享单车客服机器人用低成本的关键词搜索检索出五段帮助中心内容，然后要决定依据哪一段作答。对于 "I was charged twice"，关键词重合度把一段关于损坏报告的内容排在了第一。Valen 结合问题逐段阅读，把正确的段落移到最前面。
工作原理
一道判断题，对每个检索到的段落各问一次，每次单独发请求，这样任何段落都不会影响对其他段落的判断：
A support agent could answer the customer's question using only this passage.
（客服人员仅凭这段内容就能回答客户的问题。）
代码按概率排序，最高的段落达到 0.7 时就依据它作答；低于 0.7 时，机器人把问题交给人工。这个措辞是这样选出来的：把四种写法作为不同的问题，针对同一批段落一起发送。"answers the question" 让正确段落停留在 0.5，而 "a support agent could answer … using only this passage" 把正确和错误的段落区分开了。
运行
先下载一次示例包。caches/rerank.json 中录制的回答让脚本无需 API key 即可运行；加 --replay off 则改为调用免费托管 API。
Bash
curl -fLO https://valen-team.github.io/cookbook/valen-cookbook.zip
python -m zipfile -e valen-cookbook.zip .
cd valen-cookbook && python -m pip install -r requirements.txt
python rerank.py
python rerank.py --replay off --base-url https://api.valen-team.workers.dev/v1
代码
rerank.py
"""Rerank retrieved help-center passages by whether each one answers the question. Apache-2.0."""
import json
import re
from common import HERE, client, emit, parser, request
STOP = {"a", "an", "the", "i", "my", "is", "can", "do", "to", "of", "in", "on", "for", "it", "if", "and", "or",
"what", "how", "you", "your", "with", "be", "are", "at", "after", "from", "will", "does", "this"}
def words(text):
return set(re.findall(r"[a-z]+", text.lower())) - STOP
def retrieve(query, passages, k):
"""A cheap keyword retriever stands in for BM25 or embeddings."""
return sorted(passages, key=lambda p: len(words(query) & words(p["text"])), reverse=True)[:k]
def run(predict, query, passages, k=5):
retrieved = retrieve(query["text"], passages, k)
question = [{"name": "answers", "type": "predicate",
"instructions": "A support agent could answer the customer's question using only this passage."}]
# Each passage gets its own request so no passage can sway the judgment of another.
scored = [{"id": p["id"], "answers": request(predict, f"Customer question: {query['text']}\n\nHelp-center passage: "
f"{p['text']}", question)["answers"]["probability"]}
for p in retrieved]
reranked = sorted(scored, key=lambda x: x["answers"], reverse=True)
best = reranked[0]
return {"query": query["text"], "keyword_order": [p["id"] for p in retrieved],
"reranked": reranked, "keyword_top_correct": retrieved[0]["id"] in query["relevant"],
"reranked_top_correct": best["id"] in query["relevant"],
"answer_with": best["id"] if best["answers"] >= 0.7 else None}
def main():
args = parser(__doc__).parse_args()
data = json.loads((HERE / "rerank_data.json").read_text())
with client(args) as predict:
results = [run(predict, query, data["passages"]) for query in data["queries"]]
emit({"keyword_top1": sum(r["keyword_top_correct"] for r in results),
"reranked_top1": sum(r["reranked_top_correct"] for r in results), "queries": results}, args)
if __name__ == "__main__":
main()
rerank_data.json
rerank_data.json
{
"passages": [
{"id": "late-return", "text": "Returning a bike late: each extra 30 minutes after your booking ends is charged at the standard hourly rate. If the station is full, end the ride in the app and we waive the late fee."},
{"id": "station-full", "text": "If the return station is full, open the app, tap End ride and choose Station full. The app shows the nearest station with free docks and adds 15 free minutes."},
{"id": "damage-report", "text": "To report a damaged bike, tap the bike number in the app and choose Report a problem. Add a photo of the damage. You are not charged for damage that was there before your ride."},
{"id": "damage-fee", "text": "Damage fees: if a bike is damaged during your ride because of misuse, a repair fee of up to 80 euros may be charged. Normal wear such as worn brakes is never charged."},
{"id": "refund-time", "text": "Refunds go back to the card you paid with. Card refunds take 3 to 5 business days to appear; wallet top-ups are refunded within one day."},
{"id": "refund-request", "text": "You can request a refund for a ride within 14 days in the app under Ride history. Choose the ride, tap Request refund and tell us what went wrong."},
{"id": "helmet", "text": "Helmets are not included with standard bikes. E-bikes in Lisbon and Porto come with a helmet in the front basket; please return it with the bike."},
{"id": "e-bike-battery", "text": "An e-bike battery lasts about 60 km. The app shows the remaining range before you unlock. If the battery runs out, the bike still works as a normal bike."},
{"id": "card-charged-twice", "text": "Charged twice for one ride? The first charge is a temporary hold of 1 euro to check your card. It disappears within 7 days and is never collected."},
{"id": "student-pass", "text": "The student pass costs 9 euros a month and includes unlimited 45-minute rides. Verify your student email in the app to buy it."}
],
"queries": [
{"text": "I was charged twice for the same ride, can I get my money back?", "relevant": ["card-charged-twice", "refund-request"]},
{"text": "The bike's brakes were already worn when I took it. Will I have to pay for that?", "relevant": ["damage-fee", "damage-report"]},
{"text": "There were no free docks at the station so I kept riding. Will I pay a late fee?", "relevant": ["late-return", "station-full"]}
]
}
输出
按关键词排序，三个问题中有两个的正确段落排在第一；重新排序后三个都排在第一。对于站点停满的问题，最佳段落得分 0.64，所以机器人转人工，而不是直接作答。
录制的输出
{
"keyword_top1": 2,
"reranked_top1": 3,
"queries": [
{
"query": "I was charged twice for the same ride, can I get my money back?",
"keyword_order": [
"damage-report",
"card-charged-twice",
"late-return",
"damage-fee",
"station-full"
],
"reranked": [
{
"id": "card-charged-twice",
"answers": 0.8985652729227516
},
{
"id": "late-return",
"answers": 0.17800874742852527
},
{
"id": "damage-fee",
"answers": 0.05538949909100312
},
{
"id": "station-full",
"answers": 0.03725921031749856
},
{
"id": "damage-report",
"answers": 0.02945133769907817
}
],
"keyword_top_correct": false,
"reranked_top_correct": true,
"answer_with": "card-charged-twice"
},
{
"query": "The bike's brakes were already worn when I took it. Will I have to pay for that?",
"keyword_order": [
"damage-fee",
"damage-report",
"late-return",
"helmet",
"e-bike-battery"
],
"reranked": [
{
"id": "damage-report",
"answers": 0.9774886906340852
},
{
"id": "damage-fee",
"answers": 0.9611416781331417
},
{
"id": "late-return",
"answers": 0.09392517689858818
},
{
"id": "e-bike-battery",
"answers": 0.07181187558594625
},
{
"id": "helmet",
"answers": 0.06563972034513534
}
],
"keyword_top_correct": true,
"reranked_top_correct": true,
"answer_with": "damage-report"
},
{
"query": "There were no free docks at the station so I kept riding. Will I pay a late fee?",
"keyword_order": [
"late-return",
"station-full",
"damage-report",
"damage-fee",
"refund-time"
],
"reranked": [
{
"id": "late-return",
"answers": 0.6440917657586406
},
{
"id": "station-full",
"answers": 0.31263142824172974
},
{
"id": "damage-fee",
"answers": 0.17896146805641847
},
{
"id": "refund-time",
"answers": 0.09888336357126723
},
{
"id": "damage-report",
"answers": 0.03217982716574106
}
],
"keyword_top_correct": true,
"reranked_top_correct": true,
"answer_with": null
}
]
}
改成你自己的
- 把
retrieve换成你的 BM25 或向量检索；k保持较小，因为每个段落都是一次请求。 - Valen 只对检索返回的内容排序。正确段落没被检索到时，没有任何得分会达到 0.7，这正是转人工的信号。