Lecture Video Chaptering on VideoPoints

The problem

Lecture recordings are often an hour long, and students rarely need all of it. Chapters let them find a topic quickly, and they also give the platform's summarizer clean units to work with, one summary per chapter.

Instructors upload lectures continuously, so chaptering had to happen automatically for every video, soon after it is uploaded.

How it works

1. Extract the audio. ffmpeg pulls a mono 16 kHz track for speech recognition.
2. Clean up the slide frames. A lecture produces many near-identical frames of the same slide. A three-stage filter removes them: a grid comparison catches consecutive duplicates, an edge comparison drops frames caught halfway through a slide transition, and a sliding window catches slides that reappear later.
3. Get the text. Whisper transcribes the audio, and Tesseract OCR reads the text on each slide. Chaptering uses the transcript; the slide text is kept for later steps.
4. Find the chapters. The transcript is split slide by slide, with timestamps, and Gemini 2.5 Flash groups the slides into titled chapters.

Chapters that follow the slides

The model never picks raw timestamps. It sees the lecture as a numbered sequence of slides and answers with slide ranges, so every chapter begins and ends on a slide change.

The response is constrained by a JSON schema, so each chapter always arrives with a number, a title, and a start and end slide:

[
  {
    "section_number": 1,
    "section_title": "...",
    "slide_index_start": 1,
    "slide_index_end": 4
  },
  {
    "section_number": 2,
    "section_title": "...",
    "slide_index_start": 5,
    "slide_index_end": 9
  }
]

Shape of the output. Values are illustrative.

The prompt scales the number of chapters to the lecture's length, about six to ten per hour, and asks the model to split any chapter that runs very long at its strongest topic shift.

Checking every result

A language model can return chapters that skip a slide or overlap each other, and the output goes straight to the platform. So every response is checked before it is saved: every slide must belong to exactly one chapter, and chapters must be contiguous, with no gaps and no overlaps.

If a check fails, the service generates the chapters again, up to two attempts in total. If both fail, rule-based correction fills the gaps and resolves the overlaps, so every video still gets valid chapters.

Running in production

The service runs in the background on the VideoPoints server. Every few minutes it looks for newly uploaded lectures, processes each one, and writes the chapters back to the platform, where they become the units the summarizer works on.

Each video is processed independently, so one failure doesn't hold up the rest. Every step is logged, and the service pauses each night for a scheduled restart. A batch mode handles one-off reprocessing.

Tools