Measuring What the Eye Sees: PSNR, SSIM, and VMAF

If you work in video streaming long enough, you will inevitably face this question: how do you measure whether one encode looks better than another?

The naive answer is “just watch it.” But when you have hundreds of encoding configurations across dozens of test clips — different codecs, bitrates, resolutions, and preset levels — subjective evaluation does not scale. You need a number. The question is which number to trust.

This post walks through the three most important objective quality metrics in the streaming industry: PSNR, SSIM, and VMAF. We will look at how each one works, where it fails, and how they are used in practice to make real encoding decisions.

Why Objective Metrics Exist

The gold standard for video quality evaluation is a subjective test: you gather a panel of viewers, show them clips under controlled conditions, and collect their ratings. The result is a MOS (Mean Opinion Score) — typically on a 1-5 or 0-100 scale.

The problem is that subjective tests are expensive, slow, and not repeatable in a CI pipeline. An objective metric attempts to predict MOS computationally. The value of a metric is measured by how well its scores correlate with actual human ratings — typically evaluated via PCC (Pearson Correlation Coefficient) and SRCC (Spearman Rank-Order Correlation Coefficient) against subjective datasets.

Every metric discussed below is a full-reference (FR) metric, meaning it requires access to both the original uncompressed source and the compressed output. This is in contrast to no-reference (NR) metrics that evaluate quality from the compressed signal alone.

PSNR: The Simplest Baseline

How It Works

PSNR (Peak Signal-to-Noise Ratio) is the oldest and simplest video quality metric. It measures the ratio between the maximum possible signal power and the power of the distortion (noise) introduced by compression.

The computation is straightforward. First, calculate the MSE (Mean Squared Error) between corresponding pixels of the reference and distorted frames:

Then derive PSNR in decibels:

where $MAX$ is the maximum pixel value (255 for 8-bit content, 1023 for 10-bit).

For video, PSNR is computed per-frame and then averaged (arithmetic mean) across all frames.

Typical Score Ranges

Why It Is Still Used

Despite being a purely mathematical error measure with no perceptual model, PSNR remains ubiquitous:

• It is trivially cheap to compute — a single pass over the pixel data.
• It is universally understood — every video engineer knows what “38 dB” means.
• For a fixed codec and content type, relative PSNR comparisons are often directionally correct. If encode A has 2 dB higher PSNR than encode B at the same bitrate, it almost always looks better.
• Netflix still uses PSNR internally as one of their evaluation signals. As Jan Ozer put it: “if it’s good enough for Netflix, it’s good enough for me.”

Where It Fails

The fundamental limitation of PSNR is illustrated below: two images with the same PSNR score can have drastically different perceptual quality, because PSNR is blind to the type of distortion — it only measures the amount.

PSNR measures pixel-level error, not perceptual quality. This leads to well-known failure modes:

1. Texture masking: A complex, high-motion scene can have low PSNR but look subjectively fine because the eye cannot track individual pixel errors in textured regions. PSNR penalizes these errors just as heavily as errors in smooth gradients where they are actually visible.

2. Structural distortion insensitivity: PSNR treats all pixel errors equally. A spatially correlated blur (which humans find very objectionable) can produce the same MSE as uniformly distributed noise (which is far less noticeable).

3. Cross-codec comparison: PSNR is unreliable when comparing fundamentally different codecs. The Moscow State University (MSU) codec comparisons historically showed x264 far ahead of MainConcept by PSNR, but subjective evaluations consistently placed them much closer. Different codecs introduce different types of artifacts, and PSNR cannot distinguish between a blocking artifact and a ringing artifact that may be equally objectionable.

4. Resolution and content dependence: A PSNR of 35 dB on a 4K nature documentary means something very different from 35 dB on a 480p animation. The absolute number is not directly comparable across content or resolutions.

SSIM: Adding Structure

SSIM (Structural Similarity Index), proposed by Wang et al. in 2004, was the first widely adopted metric to incorporate aspects of human visual perception. Instead of measuring raw pixel error, it compares three properties between local patches of the reference and distorted images:

1. Luminance — comparison of mean pixel intensities
2. Contrast — comparison of standard deviations
3. Structure — comparison of normalized pixel patterns (correlation coefficient)

SSIM outputs a score between 0 and 1 (where 1 means identical). Its multi-scale variant, MS-SSIM, applies the computation at multiple resolutions and pools the results, which better captures distortions at different spatial frequencies.

SSIM correlates better with subjective ratings than PSNR, particularly for blur and structural distortions. However, it still struggles with temporal artifacts (since it is fundamentally a per-frame metric) and with content-adaptive quality assessment.

VMAF: Netflix’s Perceptual Fusion Metric

The Problem VMAF Solves

In 2016, Netflix published a metric that would change how the industry thinks about video quality. The motivation was practical: Netflix needed to optimize encoding across thousands of titles with wildly different visual characteristics — from animated shows to dark thrillers to high-motion sports. No single elementary metric predicted subjective quality well enough across all content types.

VMAF (Video Multi-Method Assessment Fusion) takes a fundamentally different approach: instead of designing a single better metric, it fuses multiple elementary metrics using machine learning and trains the fusion model against large-scale subjective data.

Architecture

VMAF follows a three-stage pipeline: feature extraction, ML fusion, and score output. The diagram below shows how the CPU-based feature extractors process each reference-distorted frame pair sequentially:

Stage 1: Feature Extraction

Three elementary feature extractors run on each frame pair:

This produces a feature vector of 9 values per frame (4 VIF scales + 4 ADM scales + 1 motion).

Stage 2: Machine Learning Fusion

The feature vector is fed into a Support Vector Machine (SVR) trained on a large corpus of subjective ratings. Netflix trained the default model (vmaf_v0.6.1) on HDTV viewing conditions (3 screen heights distance, 1080p display) using content spanning multiple genres and distortion levels.

The SVM learns non-linear relationships between the elementary features and subjective quality. For example, it learns that high motion reduces the perceptual impact of detail loss (because the eye cannot track fine details in fast-moving scenes), while low motion amplifies it.

Stage 3: Score Output

VMAF outputs a score per frame on a 0-100 scale, designed to correlate directly with MOS:

Available Models

Netflix ships several pre-trained models for different viewing scenarios:

The NEG model deserves special attention. Standard VMAF can be “gamed” by applying sharpening filters that boost VIF scores without genuinely improving quality. NEG mode disables this enhancement gain, making it essential for honest codec comparisons.

VMAF-CUDA: GPU Acceleration

As of libvmaf 3.0 (December 2023) and FFmpeg 6.1, VMAF computation can be offloaded to NVIDIA GPUs via libvmaf_cuda. The GPU implementation parallelizes the feature extraction stage across CUDA cores:

This achieves up to 4.4x throughput improvement and 37x lower latency at 4K resolution compared to CPU-only computation — which matters enormously when you are running quality analysis on thousands of encodes in a CI pipeline.

In practice, the throughput difference is dramatic — at 4K resolution, GPU-accelerated VMAF in FFmpeg achieves significantly higher frame rates:

Practical Usage with FFmpeg

Basic VMAF Measurement

ffmpeg -i distorted.mp4 -i reference.mp4 \
  -lavfi libvmaf="model=version=vmaf_v0.6.1:log_path=vmaf.json:log_fmt=json" \
  -f null -

VMAF with PSNR and SSIM (all at once)

ffmpeg -i distorted.mp4 -i reference.mp4 \
  -lavfi "libvmaf=model=version=vmaf_v0.6.1:\
  feature=name=psnr:\
  feature=name=float_ssim:\
  log_path=metrics.json:log_fmt=json" \
  -f null -

4K Model with NEG Mode

ffmpeg -i distorted_4k.mp4 -i reference_4k.mp4 \
  -lavfi "libvmaf=model=version=vmaf_4k_v0.6.1neg:\
  log_path=vmaf_4k.json:log_fmt=json:n_threads=8" \
  -f null -

GPU-Accelerated VMAF (NVIDIA)

ffmpeg -hwaccel cuda -i distorted.mp4 -hwaccel cuda -i reference.mp4 \
  -lavfi "[0:v]hwupload_cuda[dist];[1:v]hwupload_cuda[ref];\
  [dist][ref]libvmaf_cuda=log_path=vmaf_gpu.json:log_fmt=json" \
  -f null -

Reading the Output

The JSON output contains per-frame scores and pooled statistics:

{
  "pooled_metrics": {
    "vmaf": {
      "min": 72.41,
      "max": 99.12,
      "mean": 91.34,
      "harmonic_mean": 90.87
    }
  },
  "frames": [
    {"frameNum": 0, "metrics": {"vmaf": 94.21, "psnr_y": 42.3, "float_ssim": 0.987}},
    {"frameNum": 1, "metrics": {"vmaf": 93.87, "psnr_y": 41.8, "float_ssim": 0.985}}
  ]
}

The harmonic mean is generally preferred over arithmetic mean for VMAF pooling, as it is more sensitive to low-scoring frames (which disproportionately affect perceived quality). A single badly encoded scene can ruin the viewing experience even if the rest of the video scores high.

How These Metrics Shape Real Encoding Decisions

Per-Title Encoding

The most impactful application of VMAF in the industry is per-title encoding (or more precisely, per-shot encoding). The idea, pioneered by Netflix in 2015, is simple: instead of using a fixed encoding ladder for all content, analyze each title’s visual complexity and generate a custom bitrate ladder that achieves a target VMAF score.

An animated show like BoJack Horseman might achieve VMAF 93 at 750 kbps, while a dark, grainy thriller like Mindhunter might need 4,500 kbps for the same VMAF score. A fixed ladder wastes bandwidth on easy content and under-delivers on hard content.

The chart below shows this clearly: different content encoded with the same CRF values produces wildly different quality-bitrate relationships.

The typical workflow is to encode at multiple resolution-bitrate pairs, compute quality for each, and build rate-quality curves per resolution:

The convex hull is the envelope connecting the optimal (bitrate, quality) points across all resolutions. Points on or near this hull represent the most efficient encoding configurations:

ABR Quality Monitoring

In an adaptive bitrate player, VMAF scores can be used to validate that the encoding ladder provides consistent perceptual quality across renditions. If the 720p@2Mbps rendition scores VMAF 85 but the 1080p@3Mbps rendition only scores VMAF 82 (because the higher resolution exposes more compression artifacts at insufficient bitrate), the ladder needs adjustment.

Codec Evaluation

When evaluating whether to adopt a new codec (say, AV1 over H.264), VMAF provides the most reliable quality axis for BD-rate (Bjontegaard Delta rate) calculations. BD-rate tells you the bitrate savings at equivalent quality:

computed over the quality range where rate-quality curves overlap. A BD-rate of -30% with VMAF means the new codec achieves the same VMAF score at 30% lower bitrate.

Quality Gates in CI/CD

For a video SDK, VMAF enables automated quality regression testing:

# Pseudocode for a quality gate
- encode test clips with current SDK build
- compute VMAF against reference encodes
- assert VMAF_mean >= 90 and VMAF_min >= 75
- assert VMAF_delta vs baseline < 1.0 point

If an SDK update accidentally degrades encoding quality, the pipeline catches it before release.

Metric Comparison

Limitations and Open Questions

No metric is perfect. VMAF has its own blind spots:

• Training data bias: The default model was trained on specific viewing conditions and content types. It may not generalize perfectly to all scenarios (e.g., screen content, gaming, UGC with heavy filters).
• Temporal pooling: VMAF computes per-frame scores and pools them. It does not model temporal masking or sustained quality degradation over time.
• HDR content: Netflix has developed HDR-VMAF extensions, but HDR quality assessment remains an active research area. Standard VMAF models are trained on SDR content.
• Gaming artifacts: VMAF does not model latency-related quality factors. For cloud gaming and interactive streaming, metrics like VMAF alone are insufficient.
• Adversarial optimization: Encoders can be tuned to maximize VMAF at the expense of artifacts that VMAF does not penalize. The NEG model mitigates some of this, but it is not a complete solution.

The broader lesson is that no single number captures the full complexity of human visual perception. In practice, the best approach is to use VMAF as the primary metric, cross-check with PSNR for sanity, and always do spot-check subjective evaluation on the hardest scenes.

References

• Netflix/vmaf — GitHub Repository
• Toward a Better Quality Metric for the Video Community — Netflix Tech Blog
• VMAF-CUDA: GPU-Accelerated Quality Assessment — NVIDIA Blog
• Video Multimethod Assessment Fusion — Wikipedia
• Jan Ozer, Video Encoding by the Numbers — Introduction to objective quality benchmarks
• FFmpeg libvmaf Documentation