Check claims against their citations

A drafted article about a city's bike share makes seven claims, each citing a report, council minutes or a survey. Before publishing, every claim is checked against the passage it cites. Claims the source clearly backs pass; contradicted and unsupported claims go back to the writer; the rest go to an editor.

How it works

Each claim and its source form one input, with two predicates:

Question	Type
supported: the source states what the claim says	predicate
contradicted: the source states something that conflicts with the claim	predicate

"Not supported" and "contradicted" lead to different actions, so they are separate questions. The truth table:

contradicted	supported	Status
0.7 or above	any	contradicted
below 0.7	0.7 or above	supported
below 0.3	below 0.3	unsupported
otherwise	otherwise	escalate

Run it

Download the bundle once. The recorded answers in caches/citation_check.json let the script run without an API key; --replay off calls the free hosted API instead.

Bash

curl -fLO https://valen-team.github.io/cookbook/valen-cookbook.zip
python -m zipfile -e valen-cookbook.zip .
cd valen-cookbook && python -m pip install -r requirements.txt
python citation_check.py
python citation_check.py --replay off --base-url https://api.valen-team.workers.dev/v1

Code

citation_check.py

"""Check each claim in a draft against the source it cites; escalate what the source does not settle. Apache-2.0."""
import json

from common import HERE, client, emit, parser, request

QUESTIONS = [
    {"name": "supported", "type": "predicate", "instructions": "The source states what the claim says."},
    {"name": "contradicted", "type": "predicate", "instructions": "The source states something that conflicts with the claim."},
]


def run(predict, claim, sources):
    answers = request(predict, f"Claim: {claim['text']}\n\nSource [{claim['cites']}]: {sources[claim['cites']]}",
                      QUESTIONS)
    supported, contradicted = answers["supported"]["probability"], answers["contradicted"]["probability"]
    if contradicted >= 0.7:
        status = "contradicted"
    elif supported >= 0.7:
        status = "supported"
    elif supported < 0.3 and contradicted < 0.3:
        status = "unsupported"
    else:
        status = "escalate"
    return {"id": claim["id"], "cites": claim["cites"], "status": status,
            "supported": supported, "contradicted": contradicted}


def main():
    args = parser(__doc__).parse_args()
    data = json.loads((HERE / "citation_check_data.json").read_text())
    with client(args) as predict:
        checks = [run(predict, claim, data["sources"]) for claim in data["claims"]]
    emit({"publishable": all(c["status"] == "supported" for c in checks), "claims": checks}, args)


if __name__ == "__main__":
    main()

citation_check_data.json

citation_check_data.json

{
  "sources": {
    "1": "City Transport Annual Report 2025, p. 12: Bike-share trips rose to 4.1 million in 2025, up from 3.4 million in 2024. E-bikes made up 38% of trips. Average trip length was 14 minutes.",
    "2": "Council minutes, 3 February 2026: The council approved 120 new docking stations, to be installed in the eastern districts by the end of 2026. Funding comes from the regional mobility fund.",
    "3": "Rider survey, November 2025 (n = 2,300): 61% of riders said they use the service to commute. The most requested improvement was more stations near train stations (44%)."
  },
  "claims": [
    {"id": "c1", "cites": "1", "text": "Bike-share trips grew to 4.1 million in 2025."},
    {"id": "c2", "cites": "1", "text": "The average trip lasted 40 minutes."},
    {"id": "c3", "cites": "2", "text": "The council approved 120 new docking stations for the eastern districts."},
    {"id": "c4", "cites": "2", "text": "The new stations will be paid for by a fare increase."},
    {"id": "c5", "cites": "3", "text": "Most riders use bike share to get to work."},
    {"id": "c6", "cites": "3", "text": "Riders are happier with the service than they were a year ago."},
    {"id": "c7", "cites": "1", "text": "Bike-share trips fell in 2025 compared with 2024."}
  ]
}

Output

"The average trip lasted 40 minutes" is contradicted by "14 minutes" (0.89). "Riders are happier than a year ago" has nothing in the survey to back it, so it is unsupported. "Trips fell in 2025" scores 0.62 contradicted and goes to an editor.

Recorded output

{
  "publishable": false,
  "claims": [
    {
      "id": "c1",
      "cites": "1",
      "status": "supported",
      "supported": 0.992473812984103,
      "contradicted": 0.060028307587119245
    },
    {
      "id": "c2",
      "cites": "1",
      "status": "contradicted",
      "supported": 0.11813096878127707,
      "contradicted": 0.8891227576554849
    },
    {
      "id": "c3",
      "cites": "2",
      "status": "supported",
      "supported": 0.9837899291028864,
      "contradicted": 0.09080726585680793
    },
    {
      "id": "c4",
      "cites": "2",
      "status": "contradicted",
      "supported": 0.012904512852205586,
      "contradicted": 0.7954747317872861
    },
    {
      "id": "c5",
      "cites": "3",
      "status": "supported",
      "supported": 0.9694863250698544,
      "contradicted": 0.06629886029506596
    },
    {
      "id": "c6",
      "cites": "3",
      "status": "unsupported",
      "supported": 0.19670646185735557,
      "contradicted": 0.10636692573403511
    },
    {
      "id": "c7",
      "cites": "1",
      "status": "escalate",
      "supported": 0.041889894149444565,
      "contradicted": 0.6167122309420178
    }
  ]
}

Make it yours

• Comparing quantities ("most", "a third", "doubled") with figures in the source is arithmetic: extract the numbers in code and compare them there, and keep Valen for what the text says.
• Feed escalate claims to an editor or a reasoning model; everything else is settled in one cheap pass.