We’d like agents and workflows to understand the full contents of a video file, not only its audio. Right now an MP4 is processed for its audio track only, so anything shown on screen (graphics, captions, text overlays, scenes) can’t be analyzed.

Example use case: checking a TV commercial against its approved script. The agent would compare a Word document with the script against the final MP4, check both the spoken audio and the on-screen graphics and text, and flag any mismatches. This needs to work in multiple languages (e.g., English and Spanish).

What would help:

Video files (MP4 and other common formats) accepted as inputs in Chat, Workflows and Agents

Models that can read the video frames and on-screen text along with the audio transcript

Answers that point to where in the video something appears (timestamps)

Users shouldn’t have to convert or split videos into separate audio and image files first.