跳到主要内容
valen
GitHub
EN中
valen

对照引用来源核查论断

一篇关于某城市共享单车的文章草稿提出了七条论断，每条都引用了一份报告、市议会会议记录或一项调查。发布前，每条论断都要对照它引用的段落核查。来源明确支持的论断通过；被反驳和无依据的论断退回给作者；其余的交给编辑。

工作原理​

每条论断与其来源组成一个输入，配两道判断题：

问题类型
supported：来源陈述了论断所说的内容predicate
contradicted：来源陈述的内容与论断相冲突predicate

“没有依据”和“被反驳”会导向不同的动作，所以分成两道题来问。真值表：

contradictedsupported状态
0.7 及以上任意contradicted
低于 0.70.7 及以上supported
低于 0.3低于 0.3unsupported
其他其他escalate

运行​

先下载一次示例包。caches/citation_check.json 中录制的回答让脚本无需 API key 即可运行；加 --replay off 则改为调用免费托管 API。

Bash
curl -fLO https://valen-team.github.io/cookbook/valen-cookbook.zip
python -m zipfile -e valen-cookbook.zip .
cd valen-cookbook && python -m pip install -r requirements.txt
python citation_check.py
python citation_check.py --replay off --base-url https://api.valen-team.workers.dev/v1

代码​

citation_check.py
"""Check each claim in a draft against the source it cites; escalate what the source does not settle. Apache-2.0."""
import json

from common import HERE, client, emit, parser, request

QUESTIONS = [
    {"name": "supported", "type": "predicate", "instructions": "The source states what the claim says."},
    {"name": "contradicted", "type": "predicate", "instructions": "The source states something that conflicts with the claim."},
]


def run(predict, claim, sources):
    answers = request(predict, f"Claim: {claim['text']}\n\nSource [{claim['cites']}]: {sources[claim['cites']]}",
                      QUESTIONS)
    supported, contradicted = answers["supported"]["probability"], answers["contradicted"]["probability"]
    if contradicted >= 0.7:
        status = "contradicted"
    elif supported >= 0.7:
        status = "supported"
    elif supported < 0.3 and contradicted < 0.3:
        status = "unsupported"
    else:
        status = "escalate"
    return {"id": claim["id"], "cites": claim["cites"], "status": status,
            "supported": supported, "contradicted": contradicted}


def main():
    args = parser(__doc__).parse_args()
    data = json.loads((HERE / "citation_check_data.json").read_text())
    with client(args) as predict:
        checks = [run(predict, claim, data["sources"]) for claim in data["claims"]]
    emit({"publishable": all(c["status"] == "supported" for c in checks), "claims": checks}, args)


if __name__ == "__main__":
    main()
citation_check_data.json
citation_check_data.json
{
  "sources": {
    "1": "City Transport Annual Report 2025, p. 12: Bike-share trips rose to 4.1 million in 2025, up from 3.4 million in 2024. E-bikes made up 38% of trips. Average trip length was 14 minutes.",
    "2": "Council minutes, 3 February 2026: The council approved 120 new docking stations, to be installed in the eastern districts by the end of 2026. Funding comes from the regional mobility fund.",
    "3": "Rider survey, November 2025 (n = 2,300): 61% of riders said they use the service to commute. The most requested improvement was more stations near train stations (44%)."
  },
  "claims": [
    {"id": "c1", "cites": "1", "text": "Bike-share trips grew to 4.1 million in 2025."},
    {"id": "c2", "cites": "1", "text": "The average trip lasted 40 minutes."},
    {"id": "c3", "cites": "2", "text": "The council approved 120 new docking stations for the eastern districts."},
    {"id": "c4", "cites": "2", "text": "The new stations will be paid for by a fare increase."},
    {"id": "c5", "cites": "3", "text": "Most riders use bike share to get to work."},
    {"id": "c6", "cites": "3", "text": "Riders are happier with the service than they were a year ago."},
    {"id": "c7", "cites": "1", "text": "Bike-share trips fell in 2025 compared with 2024."}
  ]
}

输出​

"The average trip lasted 40 minutes" 被来源中的 "14 minutes" 反驳（0.89）。"Riders are happier than a year ago" 在调查中找不到任何依据，因此判为无依据。"Trips fell in 2025" 的 contradicted 得分为 0.62，交给编辑处理。

录制的输出
{
  "publishable": false,
  "claims": [
    {
      "id": "c1",
      "cites": "1",
      "status": "supported",
      "supported": 0.992473812984103,
      "contradicted": 0.060028307587119245
    },
    {
      "id": "c2",
      "cites": "1",
      "status": "contradicted",
      "supported": 0.11813096878127707,
      "contradicted": 0.8891227576554849
    },
    {
      "id": "c3",
      "cites": "2",
      "status": "supported",
      "supported": 0.9837899291028864,
      "contradicted": 0.09080726585680793
    },
    {
      "id": "c4",
      "cites": "2",
      "status": "contradicted",
      "supported": 0.012904512852205586,
      "contradicted": 0.7954747317872861
    },
    {
      "id": "c5",
      "cites": "3",
      "status": "supported",
      "supported": 0.9694863250698544,
      "contradicted": 0.06629886029506596
    },
    {
      "id": "c6",
      "cites": "3",
      "status": "unsupported",
      "supported": 0.19670646185735557,
      "contradicted": 0.10636692573403511
    },
    {
      "id": "c7",
      "cites": "1",
      "status": "escalate",
      "supported": 0.041889894149444565,
      "contradicted": 0.6167122309420178
    }
  ]
}

改成你自己的​

  • 把数量表述（"most"、"a third"、"doubled"）与来源中的数字作比较属于算术：在代码里提取数字并比较，Valen 只负责判断文本说了什么。
  • 把 escalate 的论断交给编辑或推理模型；其余的都在一次低成本的处理中定下来。