Rerank retrieved passages
A bike-share help bot retrieves five help-center passages with a cheap keyword search, then has to decide which one to answer from. Keyword overlap ranks a passage about damage reports first for "I was charged twice". Valen reads each passage against the question and moves the right one to the top.
How it works
One predicate, asked once per retrieved passage, each in its own request so no passage sways the judgment of another:
A support agent could answer the customer's question using only this passage.
Code sorts by probability and answers from the top passage when it reaches 0.7. Below that, the bot hands the question to a person. The wording was chosen by sending four variants as separate questions over the same passages: "answers the question" left the right passage at 0.5, while "a support agent could answer … using only this passage" separated right from wrong.
Run it
Download the bundle once. The recorded answers in caches/rerank.json let the script run without an API key; --replay off calls the free hosted API instead.
Bash
curl -fLO https://valen-team.github.io/cookbook/valen-cookbook.zip
python -m zipfile -e valen-cookbook.zip .
cd valen-cookbook && python -m pip install -r requirements.txt
python rerank.py
python rerank.py --replay off --base-url https://api.valen-team.workers.dev/v1
Code
rerank.py
"""Rerank retrieved help-center passages by whether each one answers the question. Apache-2.0."""
import json
import re
from common import HERE, client, emit, parser, request
STOP = {"a", "an", "the", "i", "my", "is", "can", "do", "to", "of", "in", "on", "for", "it", "if", "and", "or",
"what", "how", "you", "your", "with", "be", "are", "at", "after", "from", "will", "does", "this"}
def words(text):
return set(re.findall(r"[a-z]+", text.lower())) - STOP
def retrieve(query, passages, k):
"""A cheap keyword retriever stands in for BM25 or embeddings."""
return sorted(passages, key=lambda p: len(words(query) & words(p["text"])), reverse=True)[:k]
def run(predict, query, passages, k=5):
retrieved = retrieve(query["text"], passages, k)
question = [{"name": "answers", "type": "predicate",
"instructions": "A support agent could answer the customer's question using only this passage."}]
# Each passage gets its own request so no passage can sway the judgment of another.
scored = [{"id": p["id"], "answers": request(predict, f"Customer question: {query['text']}\n\nHelp-center passage: "
f"{p['text']}", question)["answers"]["probability"]}
for p in retrieved]
reranked = sorted(scored, key=lambda x: x["answers"], reverse=True)
best = reranked[0]
return {"query": query["text"], "keyword_order": [p["id"] for p in retrieved],
"reranked": reranked, "keyword_top_correct": retrieved[0]["id"] in query["relevant"],
"reranked_top_correct": best["id"] in query["relevant"],
"answer_with": best["id"] if best["answers"] >= 0.7 else None}
def main():
args = parser(__doc__).parse_args()
data = json.loads((HERE / "rerank_data.json").read_text())
with client(args) as predict:
results = [run(predict, query, data["passages"]) for query in data["queries"]]
emit({"keyword_top1": sum(r["keyword_top_correct"] for r in results),
"reranked_top1": sum(r["reranked_top_correct"] for r in results), "queries": results}, args)
if __name__ == "__main__":
main()
rerank_data.json
rerank_data.json
{
"passages": [
{"id": "late-return", "text": "Returning a bike late: each extra 30 minutes after your booking ends is charged at the standard hourly rate. If the station is full, end the ride in the app and we waive the late fee."},
{"id": "station-full", "text": "If the return station is full, open the app, tap End ride and choose Station full. The app shows the nearest station with free docks and adds 15 free minutes."},
{"id": "damage-report", "text": "To report a damaged bike, tap the bike number in the app and choose Report a problem. Add a photo of the damage. You are not charged for damage that was there before your ride."},
{"id": "damage-fee", "text": "Damage fees: if a bike is damaged during your ride because of misuse, a repair fee of up to 80 euros may be charged. Normal wear such as worn brakes is never charged."},
{"id": "refund-time", "text": "Refunds go back to the card you paid with. Card refunds take 3 to 5 business days to appear; wallet top-ups are refunded within one day."},
{"id": "refund-request", "text": "You can request a refund for a ride within 14 days in the app under Ride history. Choose the ride, tap Request refund and tell us what went wrong."},
{"id": "helmet", "text": "Helmets are not included with standard bikes. E-bikes in Lisbon and Porto come with a helmet in the front basket; please return it with the bike."},
{"id": "e-bike-battery", "text": "An e-bike battery lasts about 60 km. The app shows the remaining range before you unlock. If the battery runs out, the bike still works as a normal bike."},
{"id": "card-charged-twice", "text": "Charged twice for one ride? The first charge is a temporary hold of 1 euro to check your card. It disappears within 7 days and is never collected."},
{"id": "student-pass", "text": "The student pass costs 9 euros a month and includes unlimited 45-minute rides. Verify your student email in the app to buy it."}
],
"queries": [
{"text": "I was charged twice for the same ride, can I get my money back?", "relevant": ["card-charged-twice", "refund-request"]},
{"text": "The bike's brakes were already worn when I took it. Will I have to pay for that?", "relevant": ["damage-fee", "damage-report"]},
{"text": "There were no free docks at the station so I kept riding. Will I pay a late fee?", "relevant": ["late-return", "station-full"]}
]
}
Output
Keyword order puts the right passage first for two of three questions; after reranking, all three. For the station-full question the best passage scores 0.64, so the bot escalates instead of answering.
Recorded output
{
"keyword_top1": 2,
"reranked_top1": 3,
"queries": [
{
"query": "I was charged twice for the same ride, can I get my money back?",
"keyword_order": [
"damage-report",
"card-charged-twice",
"late-return",
"damage-fee",
"station-full"
],
"reranked": [
{
"id": "card-charged-twice",
"answers": 0.8985652729227516
},
{
"id": "late-return",
"answers": 0.17800874742852527
},
{
"id": "damage-fee",
"answers": 0.05538949909100312
},
{
"id": "station-full",
"answers": 0.03725921031749856
},
{
"id": "damage-report",
"answers": 0.02945133769907817
}
],
"keyword_top_correct": false,
"reranked_top_correct": true,
"answer_with": "card-charged-twice"
},
{
"query": "The bike's brakes were already worn when I took it. Will I have to pay for that?",
"keyword_order": [
"damage-fee",
"damage-report",
"late-return",
"helmet",
"e-bike-battery"
],
"reranked": [
{
"id": "damage-report",
"answers": 0.9774886906340852
},
{
"id": "damage-fee",
"answers": 0.9611416781331417
},
{
"id": "late-return",
"answers": 0.09392517689858818
},
{
"id": "e-bike-battery",
"answers": 0.07181187558594625
},
{
"id": "helmet",
"answers": 0.06563972034513534
}
],
"keyword_top_correct": true,
"reranked_top_correct": true,
"answer_with": "damage-report"
},
{
"query": "There were no free docks at the station so I kept riding. Will I pay a late fee?",
"keyword_order": [
"late-return",
"station-full",
"damage-report",
"damage-fee",
"refund-time"
],
"reranked": [
{
"id": "late-return",
"answers": 0.6440917657586406
},
{
"id": "station-full",
"answers": 0.31263142824172974
},
{
"id": "damage-fee",
"answers": 0.17896146805641847
},
{
"id": "refund-time",
"answers": 0.09888336357126723
},
{
"id": "damage-report",
"answers": 0.03217982716574106
}
],
"keyword_top_correct": true,
"reranked_top_correct": true,
"answer_with": null
}
]
}
Make it yours
- Replace
retrievewith your BM25 or embedding search; keepksmall, since every passage is one request. - Valen ranks only what retrieval returns. When the right passage is missing, no score reaches 0.7, which is the signal to escalate.