Find when events happen in a clip

A food-content team wants chapter marks for short cooking clips: when is the dish served, and when does someone break a dumpling open? The script samples twelve evenly spaced frames from a 6.5-second clip, asks the same two questions of each frame, and finds when each event starts.

How it works

Question	Type
plate_served: a plate of dumplings sits in the middle of the frame	predicate
dumpling_opened: two hands hold a single dumpling that is split in half	predicate

Code turns twelve probabilities per event into one time: the event starts at the first of two consecutive frames at 0.7 or above, so one noisy frame does not move a chapter mark. The first wording, "someone's hands are pulling a dumpling apart", also scored high on frames where a hand only held a dumpling; describing exactly what the frame shows fixed it.

Run it

Download the bundle once. The recorded answers in caches/event_monitoring.json let the script run without an API key; --replay off calls the free hosted API instead.

Bash

curl -fLO https://valen-team.github.io/cookbook/valen-cookbook.zip
python -m zipfile -e valen-cookbook.zip .
cd valen-cookbook && python -m pip install -r requirements.txt
python event_monitoring.py --asset-dir .
python event_monitoring.py --asset-dir . --replay off --base-url https://api.valen-team.workers.dev/v1

The clip momos.mp4 (≈395 KB) is not in the bundle; fetch it next to the scripts:

Bash

curl -fL https://raw.githubusercontent.com/Valen-Team/Valen/main/assets/cookbook/momos.mp4 -o momos.mp4

Code

event_monitoring.py

"""Find when things happen in a clip: ask the same questions of evenly spaced frames, then smooth in code. Apache-2.0."""
from common import client, emit, frame_image, image_input, parser, png, request

EVENTS = {
    "plate_served": "A plate of dumplings sits in the middle of the frame.",
    "dumpling_opened": "Two hands hold a single dumpling that is split in half.",
}


def frames(path, count):
    """Evenly spaced frames as PNG bytes with their timestamps; the clip is decoded once."""
    import av
    with av.open(str(path)) as video:
        decoded = [(frame.time, frame_image(frame)) for frame in video.decode(video=0)]
    for i in range(count):
        time, image = decoded[round(i * (len(decoded) - 1) / (count - 1))]
        yield round(float(time), 2), png(image)


def first_seen(timeline, event, hold=2):
    """An event starts at the first of `hold` consecutive frames above 0.7, so one noisy frame is ignored."""
    for i in range(len(timeline) - hold + 1):
        if all(timeline[i + k][event] >= 0.7 for k in range(hold)):
            return timeline[i]["t"]
    return None


def run(predict, path, count=12):
    questions = [{"name": name, "type": "predicate", "instructions": text} for name, text in EVENTS.items()]
    timeline = []
    for t, payload in frames(path, count):
        answers = request(predict, image_input(payload), questions)
        timeline.append({"t": t, **{name: answers[name]["probability"] for name in EVENTS}})
    return {"clip": path.name, "events": {name: first_seen(timeline, name) for name in EVENTS}, "timeline": timeline}


def main():
    args = parser(__doc__).parse_args()
    with client(args) as predict:
        emit(run(predict, args.asset_dir / "momos.mp4"), args)


if __name__ == "__main__":
    main()

Output

The plate is served at 1.17 s and the dumpling is opened at 3.53 s.

Recorded output

{
  "clip": "momos.mp4",
  "events": {
    "plate_served": 1.17,
    "dumpling_opened": 3.53
  },
  "timeline": [
    {
      "t": 0.0,
      "plate_served": 0.28772332406868567,
      "dumpling_opened": 0.6869076285687795
    },
    {
      "t": 0.6,
      "plate_served": 0.2841475392624003,
      "dumpling_opened": 0.5378598731118402
    },
    {
      "t": 1.17,
      "plate_served": 0.9506918546070104,
      "dumpling_opened": 0.49909529084535187
    },
    {
      "t": 1.77,
      "plate_served": 0.9882011871570555,
      "dumpling_opened": 0.3844224214553833
    },
    {
      "t": 2.37,
      "plate_served": 0.9939650620129246,
      "dumpling_opened": 0.47693486580446764
    },
    {
      "t": 2.97,
      "plate_served": 0.9921591161768245,
      "dumpling_opened": 0.22409593879461745
    },
    {
      "t": 3.53,
      "plate_served": 0.9933772363965263,
      "dumpling_opened": 0.9733315472800861
    },
    {
      "t": 4.13,
      "plate_served": 0.9892245749400858,
      "dumpling_opened": 0.9821471985029578
    },
    {
      "t": 4.73,
      "plate_served": 0.9875626683592001,
      "dumpling_opened": 0.9937460755695204
    },
    {
      "t": 5.33,
      "plate_served": 0.9818190756846941,
      "dumpling_opened": 0.9975682870097096
    },
    {
      "t": 5.9,
      "plate_served": 0.9723710906766717,
      "dumpling_opened": 0.998138256710976
    },
    {
      "t": 6.5,
      "plate_served": 0.9833420918016248,
      "dumpling_opened": 0.997343602582841
    }
  ]
}

Make it yours

• Raise count for longer clips, or sample one frame per second; every frame is one request.
• Video models like Valen's local backends can also take the whole clip; per-frame questions give you a timeline instead of one answer.