The Fatal Flaw of Voice-Only Transcribers

Traditional transcription tools like Otter.ai or TurboScribe share a fundamental limitation: they only listen to audio. If a speaker in a video meeting says "As you can see on this diagram, we migrated this service here" or "Let's copy this command from the terminal", a voice-only transcriber captures incomplete, ambiguous text without knowing what was actually presented on screen.

Technical terms, architecture flowcharts, code snippets, and slide bullet points shown during screen shares are completely lost. To solve this, we built a Multimodal Video Transcription & Visual OCR system for Transcribe Copilot.

Multimodal Video Stream Processing at 1 FPS

Instead of stripping the audio track and discarding the video, the app ingests the native video container (MP4, MOV, WebM) and streams it directly to Google's Gemini 3.7 Flash multimodal engine:

• Native 1-FPS Video Decoding: Gemini inspects the visual track continuously across the entire duration (over 7,000 frames for a 2-hour recording).
• Visual-to-Voice Context Enrichment: On-screen code variables, product names, URLs, and architecture labels are correlated with the spoken dialogue to guarantee 100% accurate transcription spelling.
• Structured Visual Intelligence: Gemini produces both spoken transcript turns and a dedicated visualContext object containing on-screen OCR highlights and presentation summaries.

The "30-Slide Trap": Fixed Interval vs. AI Scene Detection

Early iterations of visual frame extraction used a naive fixed timer: capturing 1 snapshot every 4 minutes (capped at 30 frames). This created two problems:

1. If a presenter stayed on the same slide for 15 minutes, it generated 3–4 duplicate thumbnails.
2. If a presenter showed a quick 1-minute architecture diagram between the 4-minute marks, the slide was completely missed.

We replaced this with AI-Driven Semantic Scene Detection. In the prompt, Gemini is instructed to detect every visual slide or screen transition and return precise timestamps: visualTransitions: [{ timeSeconds: 142.5, timestamp: "02:22", title: "Architecture Flowchart", onScreenText: "..." }].

The client browser then uses an offscreen HTML5 <canvas> to seek directly to those exact transition moments and extract pixel-perfect screenshots. In a 2-hour meeting, instead of 30 arbitrary snapshots, the app captures the exact 21 real, non-duplicate slide transitions that actually occurred.

Zero-Compute Client-Side Keyframe Extraction

Rather than running heavy FFmpeg server processes that cost money on Cloud Run, all image extraction happens client-side in the browser:

// Browser client-side keyframe snapshotting
export async function extractKeyframesAtTransitions(videoBlob, transitions) {
  const video = document.createElement("video");
  video.src = URL.createObjectURL(videoBlob);
  const canvas = document.createElement("canvas");
  const ctx = canvas.getContext("2d");

  const frames = [];
  for (const trans of transitions) {
    await seekVideoTo(video, trans.timeSeconds);
    ctx.drawImage(video, 0, 0, canvas.width, canvas.height);
    frames.push({
      timestamp: trans.timestamp,
      title: trans.title,
      dataUrl: canvas.toDataURL("image/jpeg", 0.82),
      ocrText: trans.onScreenText,
    });
  }
  return frames;
}

Clicking any slide card in the gallery immediately jumps the audio/video player to that exact timestamp in the meeting, synchronizing visual notes with spoken discussion.

Solving 503 "Deadline Expired" on 2-Hour Video Files

When processing a 700MB 2-hour video file in a standard synchronous HTTP POST request, Google's API gateway can cut the connection with a 503 Deadline expired before operation could complete because analyzing thousands of frames exceeds gateway idle timeouts.

We solved this by switching to Server-Sent Events (SSE) Streaming via streamGenerateContent?alt=sse combined with automated 3-attempt backoff retries. Because tokens stream back immediately in real-time, the connection remains active, and the browser displays a smooth progress bar as the 2-hour meeting transcript arrives.

Conclusion

The future of transcription is multimodal. By combining audio diarization with AI visual scene detection and client-side canvas keyframe extraction, meeting recordings transform from flat walls of text into rich, interactive knowledge bases with slides, on-screen code OCR, and verified action items.