FFmpegVideoTooling

FFmpeg 101

Practical FFmpeg recipes for video and audio preprocessing in ML pipelines.

Ported from the Kaggle notebook source (narrative + code). Cell outputs are not re-executed here — run it on Kaggle for live results.

Preamble

I have tried to use torchcodec (a codec library that has good PyTorch API etc) at the end of this notebook but failed due to env issues. If I manage to succeed later, I will add a section.

In the meantime, you can give it a try by installing it:

!pip install torchcodec --index-url=https://download.pytorch.org/whl/cu126

Or checking the repo.

Introduction

I have been using ffmpeg over the last 5 years (or so) and much more recently (due to my new professional activities).

In this guide, we will explore some of the core concepts of this great piece of software work, with some detailed exemples.

Let's get started!

What is ffmpeg?

logo

Here is a high-level overview from Claude (with some personal modification, using Claude's Sonnet 3.5 model):

ffmpeg is a powerful open-source software suite for handling multimedia data.

It's a complete solution for recording, converting, and streaming audio and video content. The name comes from "FF" (Fast Forward) and MPEG (Moving Picture Experts Group).

At its core, FFmpeg provides several key capabilities:

Format Conversion (Transcoding): It can convert between virtually any media format, such as converting MP4 videos to WebM, or WAV audio files to MP3. This includes support for hundreds of different codecs and container formats.
Video/Audio Processing: beyond simple conversion, FFmpeg can:
- Resize videos
- Adjust video quality
- Change frame rates
- Modify audio sampling rates
- Apply filters and effects
- Extract audio from video files
- Combine multiple media files
Streaming: it supports streaming protocols like RTMP, HLS, and DASH, making it essential for live streaming applications and video platforms.
Screen Recording: can capture screen content, making it useful for creating tutorials or recording gameplay.

FFmpeg is used extensively behind the scenes in many popular applications and services. YouTube, VLC media player, and many video editing tools rely on FFmpeg for media processing. It can be used both through command-line interface (CLI) and as a library in software development.

To install the CLI on Linux, the easiest way is the following command:

apt install ffmpeg

For developers, FFmpeg provides several libraries:

libavcodec: For encoding and decoding audio/video
libavformat: For muxing and demuxing media containers
libavfilter: For adding effects and modifications
libavutil: For common utilities
libswscale: For image scaling and color space/pixel format conversion

Due to its versatility and robust performance, FFmpeg has become the de facto standard for media handling in both professional and open-source software.

Great, now that we know a bit more about ffmpeg, let's some adjacent core concepts before moving to some common commands.

What is really a video?

You might be wondering what a video could be? A video is a collection of the following elements:

A container Format (also called wrapper format):

This is like an envelope that holds everything together. It defines how to store the video, audio, subtitles, and metadata in a single file. Common container formats include:

MP4 (.mp4) AVI (.avi) MKV (.mkv) WebM (.webm) MOV (.mov)

Video Codec: This is the algorithm that compresses and decompresses the actual video data. The codec determines how individual frames are encoded. Popular video codecs include:

H.264/AVC (very common for streaming and general use) H.265/HEVC (newer, more efficient compression) VP9 (open-source, used by YouTube) AV1 (newest open-source codec)

Audio Codec: Similar to video codecs, these handle the sound data. Common audio codecs include:

AAC (Advanced Audio Coding) MP3 AC3 (Dolby Digital) Opus (open-source)

When these components come together, they form what we consider a video file. For example, a typical MP4 file might contain:

Container: MP4 Video: H.264 encoded video stream Audio: AAC encoded audio stream Additional metadata like title, creation date, etc.

The playback software needs to understand both the container format and the specific codecs used to properly decode and play the video.

About codecs

A (video) codec is short for compressor-decompressor. It is used to transform a raw video into a more mangeable format. Most often this transformation is lossy, i.e. some of the details of the original raw video are lost.

Some of the popular codecs are:

H.264/AVC (Advanced Video Coding) - Widely used for streaming, Blu-ray discs, and video sharing platforms due to its excellent balance of quality and compression
H.265/HEVC (High Efficiency Video Coding) - The successor to H.264, offering roughly 50% better compression at the same quality level
AV1 (AOMedia Video 1) - A newer royalty-free codec gaining adoption by major streaming platforms and tech companies, offering better compression than HEVC
VP9 - Google's codec used widely on YouTube and in browsers, serving as an alternative to HEVC
ProRes - Apple's professional codec focused on editing quality rather than file size, popular in video production workflows
MPEG-2 - Older but still used for broadcast television and DVD
VC-1 - Microsoft's video codec used in some Blu-ray discs and Windows Media
AVC1 - A specific implementation of H.264 often used in web browsers
VP8 - An older open codec from Google, predecessor to VP9
MJPEG (Motion JPEG) - Used in some cameras and for editing, prioritizing each frame quality over compression

YCBCR color space.
Luma and chroma signals
Often the luma signal is more important for (human) visual perception
Thus, the chroma signal is often subsampled to achieve higher compression
Common subsampling ratios are 4:2:1 and 4:1:1
For professional codecs, the ratio could be 4:4:4 (one example is Sony's HDCAM-SR)
A few codecs can sample on the RGB subspace and blue could be undersampled.
There is also subsampling in the spatial and temporal dimensions.
Often DCT is used (and some modern codecs use wavelet transformations)

Let's have a detour and explore a bit the concept of video compression picture types. For that, we have used again Claude Sonnet and adapted the summary below:

Video compression relies on different frame types (also called picture types) to efficiently reduce file size while maintaining quality. Here's a detailed look at the key frame types used in modern video codecs:

I-Frames (Intra-coded frames)

Complete, standalone images similar to JPEG photos
Don't depend on data from other frames
Contain all the data needed to display the entire picture
Serve as reference points for other frame types
Require the most storage space but provide high quality
Typically appear at scene changes or at regular intervals (e.g., every 12-24 frames)
Essential for seeking/scrubbing through video

P-Frames (Predicted frames)

Store only the differences (delta) from previous I-frames or P-frames
Use motion compensation to predict where elements moved from the reference frame
Much smaller than I-frames (typically 30-50% of the size)
Form a dependency chain (require preceding reference frames to be decoded)
Quality gradually decreases as you move further from an I-frame

B-Frames (Bi-directional predicted frames)

Reference both previous AND future frames to predict content
Most storage-efficient frame type (often 10-20% of an I-frame's size)
Create the highest compression ratio but require more processing power
Introduce some encoding/decoding latency due to looking ahead
Not used in low-latency applications like video conferencing
Can't be used as reference frames in some implementations

The specific arrangement of these frame types is called the Group of Pictures (GOP) structure, which significantly impacts compression efficiency, quality, random access capability, and error resilience of the compressed video.

Let's dig a bit deeper into the concept of GOP (group of picture)

IPBBPBBPBB: this is one possible GOP

About containers

A container is a file format that contains different data streams (i.e. think about it as a box that contains things):

video stream
audio stream
metadata
subtitles

Its puropse is to keep the different streams synchronized and compatible. Notice that the different streams could be each encoded using a different codec: h264 for the video and aac for the audio for example.

One popular example is the .mp4 video container.

Thus, a container and a codec make a video format.

Let's explore the .mp4 video format as an example.

MP4

MP4, short for MPEG4 part 14, is a very popular video format.

As we have seen above, it is a combination of a codec (compressor-decoder) and a container:

codec: for mp4, it is often h264 for video and aac for audio
container: it is .mp4 file format

Extracting metadata

One common task that ffmpeg is used for is to extract metadata from the video to analyze. For that, there is a handy CLI tool: ffprobe

Here is one command to do it:

ffprobe path/to/video

!ffprobe -h

Let's do it using this Python wrapper.

# To install python's ffmpeg wrapper:
!pip install ffmpeg-python

import ffmpeg
import torch
from torchvision.datasets.utils import download_url

# Download the sample video
download_url(
    "https://github.com/pytorch/vision/blob/main/test/assets/videos/WUzgd7C1pWA.mp4?raw=true",
    ".",
    "WUzgd7C1pWA.mp4"
)
video_path = "./WUzgd7C1pWA.mp4"
meta = ffmpeg.probe(video_path)

meta

We can extract some useful information from the metadata:

video_info = next(s for s in meta['streams'] if s['codec_type'] == 'video')
width = int(video_info['width'])
height = int(video_info['height'])
num_frames = int(video_info['nb_frames'])
fps = eval(video_info['r_frame_rate'])


print("width: ", width, "\nheight: ", height, "\nfps: ", fps)

Extracting frames

Let's extract frames using numpy and the ffmpeg wrapper

import os
import numpy as np

# Create output directory for frames
output_dir = "extracted_frames"
os.makedirs(output_dir, exist_ok=True)

# Extract frames using ffmpeg
def extract_frames(video_path, output_dir, frame_rate=None):
    """
    Extract frames from a video file.
    
    Args:
        video_path: Path to the video file
        output_dir: Directory to save extracted frames
        frame_rate: Optional frame rate for extraction (e.g., '1' for 1 frame per second)
    """
    output_pattern = os.path.join(output_dir, "frame_%04d.jpg")
    
    # Build the ffmpeg command
    stream = ffmpeg.input(video_path)
    
    if frame_rate:
        # Extract at specified frame rate
        stream = stream.filter('fps', fps=frame_rate)
    
    stream = stream.output(output_pattern)
    
    # Run the ffmpeg command
    print("Extracting frames...")
    ffmpeg.run(stream, capture_stdout=False)
    print(f"Frames extracted to {output_dir}")

# Extract all frames
extract_frames(video_path, output_dir)

# To read an extracted frame using numpy
from PIL import Image
from IPython.display import display

def read_frame(frame_path):
    """Read an image frame and return it as a numpy array."""
    out, _ = (
        ffmpeg
        .input(frame_path)
        .output('pipe:', format='rawvideo', pix_fmt='rgb24')
        .run(capture_stdout=True)
    )
    frame = np.frombuffer(out, np.uint8).reshape([-1, height, width, 3])
    return frame[0]

# Example: Read the first extracted frame
first_frame_path = os.path.join(output_dir, "frame_0001.jpg")
if os.path.exists(first_frame_path):
    frame = read_frame(first_frame_path)
    print(f"First frame shape: {frame.shape}")
    img = Image.fromarray(frame)
    display(img)

Extracting audio

Let's give another example of extracting the audio stream using the ffmpeg wrapper as well. We will encode the audio as an mp3 since it is a very popular choice.

from IPython.display import Audio


def extract_and_play_audio_mp3(video_path, output_audio_path="extracted_audio.mp3"):
    """Extract audio as MP3 and play it"""
    try:
        (
            ffmpeg
            .input(video_path)
            .output(output_audio_path, acodec='mp3')
            .overwrite_output()
            .run(quiet=True)
        )
        
        print(f"Audio extracted to: {output_audio_path}")
        
        audio = Audio(output_audio_path)
        display(audio)
        
        return output_audio_path
        
    except Exception as e:
        print(f"Error: {e}")
        return None

extract_and_play_audio_mp3(video_path)

From a video to tensors on GPU

We will try to get a video decoded directly on the GPU as tensors and then save it back as a video using GPU supported encoder.

First, let's check if we have the necessary decoder and encoder with NVIDIA support in the current env.

!ffmpeg -decoders | grep -i nvidia

!ffmpeg -encoders | grep -i nvidia

We can try h264_cuvid for decoding and nvenc_h264 for encoding later (notice that these choices are driven by the sample video we are using)

As stated in the preambule, we won't use torchcodec (have tried but failed with the current env) for now since it is a pain to install, instead we will use PyNvVideoCodec.

https://developer.nvidia.com/pynvvideocodec

This Python library isn't perfect and lacks some complete examples (though there are some in the official doc) but it works and should be enough for what we want to achieve here.

We will also heavily rely on the code from here.

!pip install PyNvVideoCodec

import PyNvVideoCodec as nvc 


class VideoDecoder:
    def __init__(self, codec=nvc.cudaVideoCodec.H264, gpuid=0, usedevicememory=True):
        self.codec = codec
        self.gpuid = gpuid
        self.usedevicememory = usedevicememory
        self.decoder = None
        self.demuxer = None
        self.frame_count = 0
        self.packet_iterator = None
        self.frame_iterator = None

    def initialize(self, input_file):
        self.frame_count = 0
        self.demuxer = nvc.CreateDemuxer(filename=input_file)
        self.decoder = nvc.CreateDecoder(
            gpuid=self.gpuid,
            codec=self.codec,
            cudacontext=0,
            cudastream=0,
            usedevicememory=self.usedevicememory
        )
        self.packet_iterator = iter(self.demuxer)
        self.frame_iterator = None

    def __iter__(self):
        return self

    def __next__(self):
        if self.frame_iterator is None:
            try:
                packet = next(self.packet_iterator)
                self.frame_iterator = iter(self.decoder.Decode(packet))
            except StopIteration:
                raise StopIteration
            except Exception as e:
                logging.error(f'Error decoding packet: {e}', exc_info=True)
                raise e
        
        try:
            decoded_frame = next(self.frame_iterator)
            self.frame_count += 1
            return self.process_frame(decoded_frame)
        except StopIteration:
            self.frame_iterator = None
            return self.__next__()
        except Exception as e:
            logging.error(f'Error decoding frame: {e}', exc_info=True)
            raise e

    @staticmethod
    def nv12_to_rgb(nv12_tensor, width, height):
        try:
            nv12_tensor = nv12_tensor.to(dtype=torch.float32)
            y_plane = nv12_tensor[:height, :width]
            uv_plane = nv12_tensor[height:height + height // 2, :].view(height // 2, width // 2, 2).repeat_interleave(2, dim=0).repeat_interleave(2, dim=1)
            u_plane = uv_plane[:, :, 0] - 128
            v_plane = uv_plane[:, :, 1] - 128
            r = y_plane + 1.402 * v_plane
            g = y_plane - 0.344136 * u_plane - 0.714136 * v_plane
            b = y_plane + 1.772 * u_plane
            rgb_frame = torch.stack((r, g, b), dim=2).clamp(0, 255).byte()
            return rgb_frame
        except Exception as e:
            logging.error(f'Error converting NV12 to RGB: {e}', exc_info=True)
            raise e

    def process_frame(self, frame):
        try:
            src_tensor = torch.from_dlpack(frame)
            (height, width) = frame.shape
            rgb_tensor = self.nv12_to_rgb(src_tensor, width, int(height / 1.5))
            return rgb_tensor
        except Exception as e:
            logging.error(f'Error processing frame: {e}', exc_info=True)
            raise e

decoder = VideoDecoder()
decoder.initialize(video_path)

mean_frames = []

for frame_id, frame in enumerate(decoder):
    # From 0-255 to 0-1
    frame = frame / 255.0
    if frame_id % 100 == 0:
        print(frame.shape, frame.dtype, frame.device)
    mean_frames.append(frame.mean())

As you can see, we have managed to decode a video directly as a sequence of tensors on GPU. Very nice!

import pandas as pd
pd.Series(torch.stack(mean_frames).cpu().numpy()).plot()

Let's wrap up by encoding transformed frames into a new video. For that, we will use the following code snippet:

import logging

class VideoEncoder:
    def __init__(self, width, height, format, use_cpu_input_buffer=False, **kwargs):
        self.width = width
        self.height = height
        self.format = format
        self.use_cpu_input_buffer = use_cpu_input_buffer
        self.encoder = nvc.CreateEncoder(width, height, format, use_cpu_input_buffer, **kwargs)
        logging.info(f'Encoder created with width: {width}, height: {height}, format: {format}, use_cpu_input_buffer: {use_cpu_input_buffer}')

    @staticmethod
    def rgb_to_yuv(rgb_tensor):
        rgb_tensor = rgb_tensor.to(dtype=torch.float32)
        r = rgb_tensor[:, :, 0]
        g = rgb_tensor[:, :, 1]
        b = rgb_tensor[:, :, 2]
        y = 0.299 * r + 0.587 * g + 0.114 * b
        u = -0.14713 * r - 0.28886 * g + 0.436 * b + 128
        v = 0.615 * r - 0.51499 * g - 0.10001 * b + 128
        height, width = rgb_tensor.shape[:2]
        y_plane = y
        u_plane = u[0::2, 0::2]
        v_plane = v[0::2, 0::2]
        uv_plane = torch.stack((u_plane, v_plane), dim=2).reshape(height // 2, width)
        tensor_yuv = torch.cat((y_plane, uv_plane), dim=0).clamp(0, 255).byte()
        return tensor_yuv

    def encode(self, input_data):
        try:
            bitstream = self.encoder.Encode(input_data)
            return bitstream
        except Exception as e:
            logging.error(f'Error encoding frame: {e}', exc_info=True)
            return None

    def end_encode(self):
        try:
            bitstream = self.encoder.EndEncode()
            logging.info('Encoder flushed successfully')
            return bitstream
        except Exception as e:
            logging.error(f'Error ending encode: {e}', exc_info=True)
            return None

    def reconfigure(self, params):
        try:
            self.encoder.Reconfigure(params)
            logging.info('Encoder reconfigured successfully')
        except Exception as e:
            logging.error(f'Error reconfiguring encoder: {e}', exc_info=True)

    def get_reconfigure_params(self):
        try:
            params = self.encoder.GetEncodeReconfigureParams()
            logging.info('Reconfigure parameters fetched successfully')
            return params
        except Exception as e:
            logging.error(f'Error fetching reconfigure parameters: {e}', exc_info=True)
            return None

decoder = VideoDecoder()
decoder.initialize(video_path)
encoder = VideoEncoder(width=width, height=height, format="NV12", 
                       use_cpu_input_buffer=False, codec="h264", 
                       fps=fps)

from tqdm import tqdm

def process(video_decoder, video_encoder, output_file):
    with open(output_file, 'wb') as f:
        for frame_number, rgb_tensor in tqdm(enumerate(video_decoder, start=1), desc="Processing"):
            # You can do various transformation.
            # Here we simply invert the RGB to BGR space.
            rgb_tensor = rgb_tensor[:, :, [2, 1, 0]]
            input_tensor = video_encoder.rgb_to_yuv(rgb_tensor)
            bitstream = video_encoder.encode(input_tensor)
            if bitstream:
                f.write(bytearray(bitstream))
        remaining_bitstream = video_encoder.end_encode()
        if remaining_bitstream:
            f.write(bytearray(remaining_bitstream))

process(decoder, encoder, "raw_output.mp4")

As you can see, the processing is extremly fast.

Notice that for some reasons that are outside my knowledge for now, we need to remux the output of the process function (i.e. move from one container to another).

That's what we do in the next cell.

input_video = ffmpeg.input('raw_output.mp4')
output = ffmpeg.output(input_video, 'output.mp4', vcodec='copy', acodec='copy')
ffmpeg.run(output, overwrite_output=True)

from IPython.display import Video

# Doesn't display before remuxing.
Video("raw_output.mp4")

Video(video_path)

Video("output.mp4")

Now the output.mp4 can be displayed (even though there is some "jittering", need more exploration to understand the root cause, maybe it is already in the original video, need to try on other videos).

Conclusion

That's it for this introduciton to ffmpeg, I hope you have learned something out of it.