Video Auto-Labeling Pipeline

Step 1: Black Frame Detection (Worker)

Each 60-second chunk is analyzed for black frame runs using color histogram comparison:

1. Compute 32-bin color histogram for each frame (RGB channels)
2. Calculate chi-squared distance from a pure-black reference histogram
3. If distance < 0.3 → confirmed black frame
4. If distance < 8.0 → suspicious (may be very dark scene)
5. Refine coarse detections to frame-accurate start/end timestamps

For normal chunks, a coarse 1fps scan finds candidates, then a frame-by-frame refinement pass in ±2s windows determines precise boundaries. Short probe windows (≤15s) use frame-by-frame scanning directly.

Step 2: Event Extraction (Merger)

Black frame runs from all chunks are merged and converted into discrete "events":

1. Merge adjacent runs across chunk boundaries (gap < 0.1s)
2. Filter runs shorter than 0.1s (noise)
3. Extract events: runs separated by ≥15 seconds become distinct events
4. Remove events in first 5 minutes and last 60 seconds (margins)

Step 3: Segment Pairing (Merger)

Events are paired into segments using duration heuristics:

1. For each unpaired event, find the best partner where:
   - Duration is between 90s and 300s (1.5 to 5 minutes)
   - Duration is closest to a standard multiple of 30s
2. Remove "interior" events (black frames within paired segments)
3. Re-pair after cleanup
4. Probe for missing partners at standard offsets (30s, 60s, ..., 300s)
5. Short segments (<90s) → attempt to extend from endpoints
6. Final cleanup: merge overlaps, remove contained segments

Key Design Decisions

• Configurable margin: Start/end margins are skippable (default: first 300s skipped for pre-content)
• Histogram-based detection: More robust than simple pixel averaging — handles noise, compression artifacts, and near-black scenes
• Standard duration matching: Segments are paired using expected duration multiples, providing a strong pairing heuristic
• Active probing: When events can't be paired from the initial scan, targeted Lambda invocations search for missing partners at expected offsets
• 60-second chunks: Balances Lambda execution time against parallelism and Step Functions event limits

The system uses Claude (via Amazon Bedrock) with a carefully crafted archival prompt that instructs the model to:

• Respect hard boundaries: [ BREAK ] markers are absolute — no segment spans across them
• Identify natural transitions: Speaker handoffs, topic changes, and shifts in focus
• Extract verbatim sentences: First and last sentences of each segment are extracted word-for-word for precise timestamp matching
• Maintain continuity: Panel discussions that shift topics stay as one segment; same-story coverage across speakers stays unified
• Complete coverage: Every part of the transcript must be assigned to a segment with no gaps

Vanderbilt Cloud Innovation Lab

The Vanderbilt Cloud Innovation Lab, housed at the Jean and Alexander Heard Libraries and powered by Amazon Web Services, provides immersive experiences for Vanderbilt University students who are interested in learning how to use artificial intelligence and cloud technologies to develop advancements in digital scholarship. Under the mentorship of AWS and library staff experts, students not only gain important career skills by collaborating on open-source technology but also have meaningful opportunities to enact lasting, positive change. Their innovative solutions to today's real-life challenges ensure that the world's cultural heritage is preserved and accessible for the benefit of future generations.

Students are involved in every stage of a Vanderbilt Cloud Innovation Lab project and learn a variety of career competencies in the process: experiential problem solving, critical thinking, project management, agile development, technical skills, and time management.