MediaFM: The Multimodal AI Foundation for Media Understanding at Netflix

Netflix introduces MediaFM, an in-house tri-modal (audio, video, text) foundation model for deep media content understanding at scale.

• •Shots are extracted via shot boundary detection and embedded using SeqCLIP (video), wav2vec2 (audio), and text-embedding-3-large (subtitles), concatenated into a 2304-dimensional fused vector
• •The core architecture is a BERT-like transformer encoder trained with a Masked Shot Modeling (MSM) objective, predicting original embeddings for randomly masked shots (20%)
• •A [GLOBAL] token derived from title-level metadata (synopses, tags) is prepended to each sequence to provide contextual grounding
• •"Embedding in context" — extracting clip embeddings from within a full episode sequence — significantly outperforms embedding clips in isolation
• •