跳到主要内容
valen
GitHub
EN中
valen

找出片段中事件发生的时间

一个美食内容团队想给烹饪短片打章节标记：菜什么时候上桌，什么时候有人把饺子掰开？脚本从一段 6.5 秒的片段中均匀抽取十二帧，对每一帧问同样的两个问题，找出每个事件的开始时间。

工作原理​

问题类型
plate_served：一盘饺子位于画面中央predicate
dumpling_opened：两只手拿着一个被掰成两半的饺子predicate

代码把每个事件的十二个概率转换成一个时间点：事件从连续两帧都达到 0.7 及以上的第一帧开始，这样单独一帧的噪声不会移动章节标记。最初的措辞 "someone's hands are pulling a dumpling apart" 在只是用手拿着饺子的帧上也得分很高；准确描述画面中显示的内容后，问题就解决了。

运行​

先下载一次示例包。caches/event_monitoring.json 中录制的回答让脚本无需 API key 即可运行；加 --replay off 则改为调用免费托管 API。

Bash
curl -fLO https://valen-team.github.io/cookbook/valen-cookbook.zip
python -m zipfile -e valen-cookbook.zip .
cd valen-cookbook && python -m pip install -r requirements.txt
python event_monitoring.py --asset-dir .
python event_monitoring.py --asset-dir . --replay off --base-url https://api.valen-team.workers.dev/v1

视频 momos.mp4（约 395 KB）不在示例包中；把它下载到脚本所在目录：

Bash
curl -fL https://raw.githubusercontent.com/Valen-Team/Valen/main/assets/cookbook/momos.mp4 -o momos.mp4

代码​

event_monitoring.py
"""Find when things happen in a clip: ask the same questions of evenly spaced frames, then smooth in code. Apache-2.0."""
from common import client, emit, frame_image, image_input, parser, png, request

EVENTS = {
    "plate_served": "A plate of dumplings sits in the middle of the frame.",
    "dumpling_opened": "Two hands hold a single dumpling that is split in half.",
}


def frames(path, count):
    """Evenly spaced frames as PNG bytes with their timestamps; the clip is decoded once."""
    import av
    with av.open(str(path)) as video:
        decoded = [(frame.time, frame_image(frame)) for frame in video.decode(video=0)]
    for i in range(count):
        time, image = decoded[round(i * (len(decoded) - 1) / (count - 1))]
        yield round(float(time), 2), png(image)


def first_seen(timeline, event, hold=2):
    """An event starts at the first of `hold` consecutive frames above 0.7, so one noisy frame is ignored."""
    for i in range(len(timeline) - hold + 1):
        if all(timeline[i + k][event] >= 0.7 for k in range(hold)):
            return timeline[i]["t"]
    return None


def run(predict, path, count=12):
    questions = [{"name": name, "type": "predicate", "instructions": text} for name, text in EVENTS.items()]
    timeline = []
    for t, payload in frames(path, count):
        answers = request(predict, image_input(payload), questions)
        timeline.append({"t": t, **{name: answers[name]["probability"] for name in EVENTS}})
    return {"clip": path.name, "events": {name: first_seen(timeline, name) for name in EVENTS}, "timeline": timeline}


def main():
    args = parser(__doc__).parse_args()
    with client(args) as predict:
        emit(run(predict, args.asset_dir / "momos.mp4"), args)


if __name__ == "__main__":
    main()

输出​

盘子在 1.17 秒上桌，饺子在 3.53 秒被掰开。

录制的输出
{
  "clip": "momos.mp4",
  "events": {
    "plate_served": 1.17,
    "dumpling_opened": 3.53
  },
  "timeline": [
    {
      "t": 0.0,
      "plate_served": 0.28772332406868567,
      "dumpling_opened": 0.6869076285687795
    },
    {
      "t": 0.6,
      "plate_served": 0.2841475392624003,
      "dumpling_opened": 0.5378598731118402
    },
    {
      "t": 1.17,
      "plate_served": 0.9506918546070104,
      "dumpling_opened": 0.49909529084535187
    },
    {
      "t": 1.77,
      "plate_served": 0.9882011871570555,
      "dumpling_opened": 0.3844224214553833
    },
    {
      "t": 2.37,
      "plate_served": 0.9939650620129246,
      "dumpling_opened": 0.47693486580446764
    },
    {
      "t": 2.97,
      "plate_served": 0.9921591161768245,
      "dumpling_opened": 0.22409593879461745
    },
    {
      "t": 3.53,
      "plate_served": 0.9933772363965263,
      "dumpling_opened": 0.9733315472800861
    },
    {
      "t": 4.13,
      "plate_served": 0.9892245749400858,
      "dumpling_opened": 0.9821471985029578
    },
    {
      "t": 4.73,
      "plate_served": 0.9875626683592001,
      "dumpling_opened": 0.9937460755695204
    },
    {
      "t": 5.33,
      "plate_served": 0.9818190756846941,
      "dumpling_opened": 0.9975682870097096
    },
    {
      "t": 5.9,
      "plate_served": 0.9723710906766717,
      "dumpling_opened": 0.998138256710976
    },
    {
      "t": 6.5,
      "plate_served": 0.9833420918016248,
      "dumpling_opened": 0.997343602582841
    }
  ]
}

改成你自己的​

  • 片段较长时调大 count，或每秒抽一帧；每一帧都是一次请求。
  • 像 Valen 本地后端这样的视频模型也可以直接接收整段视频；逐帧提问得到的是一条时间线，而不是单个回答。