Android Video Frame Rendering: Scaling, Cropping, and Surface Management

Introduction

Video frame cropping occurs when video players need to display only a portion of video frames, or when the original video frame dimensions don’t match the render window dimensions. The Android framework provides several APIs and parameters that allow application developers to control how frames are cropped during playback. This document explains how these mechanisms work in multimedia scenarios.

Crop Rectangle Control

MediaFormat Parameters

Applications can specify which portion of video frames to display by configuring four parameters when setting up a video MediaCodec:

  • KEY_CROP_BOTTOM
  • KEY_CROP_LEFT
  • KEY_CROP_RIGHT
  • KEY_CROP_TOP

These four parameters define a visible rectangle (inclusive bounds). The following Java code demonstrates how to configure cropping to display only the top-left quarter of frames:

1
2
3
4
5

MediaFormat mediaFormat = new MediaFormat();    
mediaFormat.setInteger(MediaFormat.KEY_CROP_LEFT, 0);
mediaFormat.setInteger(MediaFormat.KEY_CROP_RIGHT, format.width / 2 - 1);
mediaFormat.setInteger(MediaFormat.KEY_CROP_TOP, 0);
mediaFormat.setInteger(MediaFormat.KEY_CROP_BOTTOM, format.height / 2 - 1);

Hardware Support Limitations: These parameters don’t work consistently across all devices, as they require explicit support declaration from C2 HAL implementations. Among FireTV products, MTK chips support these parameters (though they don’t function properly on Hubbell devices during testing), while Amlogic chips don’t support them at all.

SPS-Level Cropping

Video encoding standards like H.264 allow content creators to specify cropping areas within the Sequence Parameter Set (SPS) data. For example, FFmpeg provides a bitstream filter to modify these properties:

1

ffmpeg -i 1080.mp4 -c copy -bsf:v h264_metadata=crop_left=0:crop_right=960:crop_top=0:crop_bottom=540 1080_cropped.mp4 -y

The output video 1080_cropped.mp4 maintains its original 1080p resolution, but media players should respect the cropping properties embedded in the SPS during rendering.

Decoder Behavior: When decoding videos with SPS crop properties, C2 decoders trigger an output format change event to notify frameworks and applications once SPS data is successfully parsed. Example debug output:

1
2
3
4
5
6

08-29 15:26:08.253     0     0 I [3 T7759  ..] [23]: Parse from ucode, visible(960 x 548), coded(1920 x 1088), scan:P, bitdepth(0), dw(400)
08-29 15:26:08.121   505  8059 I VDA     : [0##23]"[GetVisibleSize] visible width 960, height 548\n"
08-29 15:26:08.248  6084  8049 D ccodecconfig: c2 config diff is   c2::u32 raw.crop.height = 548
08-29 15:26:08.248  6084  8049 D ccodecconfig:   c2::u32 raw.crop.left = 0
08-29 15:26:08.248  6084  8049 D ccodecconfig:   c2::u32 raw.crop.top = 0
08-29 15:26:08.248  6084  8049 D ccodecconfig:   c2::u32 raw.crop.width = 960

The following image consists of two screenshots taken from two ExoPlayer playback sessions playing the original video and the cropped video, respectively.

sps-crop-comparison

Video Scaling Modes

Public API Overview

For video playback scenarios, MediaCodec provides the setVideoScalingMode() API (also available in ExoPlayer) to control how video frames fit within render windows. The API accepts one of two scaling modes:

  1. VIDEO_SCALING_MODE_SCALE_TO_FIT (default)
  2. VIDEO_SCALING_MODE_SCALE_TO_FIT_WITH_CROPPING

Scale-to-Fit Mode (Stretch-to-Window)

This mode performs independent horizontal and vertical scaling to ensure all video frame content fits within render window boundaries. Video buffer dimensions are stretched or compressed as needed to match the window’s aspect ratio. This guarantees no video content is lost or cropped, but may introduce visual distortion when aspect ratios don’t match.

Characteristics:

  • Preserves all video content
  • May cause aspect ratio distortion
  • Can result in stretched or compressed appearance
  • Suitable for contents with non-square pixel aspect ratios

Scale-to-Fit-with-Cropping Mode (Aspect-Preserving Fill)

This mode applies uniform scaling to both dimensions, maintaining the video buffer’s original aspect ratio while maximizing render window utilization. The buffer is scaled so the smaller dimension fits the window exactly, while the larger dimension may extend beyond window boundaries and gets cropped. This ensures optimal visual quality without aspect ratio distortion, but may lose some video content.

Characteristics:

  • Maintains original aspect ratio
  • Maximizes screen area utilization
  • May crop content along the oversized dimension
  • Eliminates letterboxing/pillarboxing
  • Only suitable for pixel aspect ratio 1:1

Important Caveat: The applied scaling mode may be reset when video output buffers change, so applications should call setVideoScalingMode() every time the onOutputFormatChange event is triggered.

Visual Comparison

The following figure illustrates the visual effects of both scaling modes when rendering a 1920×1080 video in a 1080×1080 render window:

scaling-modes-comparison

Render Window Management

Surface Layout Considerations

The render window refers to the video output target, typically a Surface. The surface’s layout and size define the render window’s position and boundaries. To maintain video frame integrity and avoid visual distortion, media player applications should calculate and apply appropriate parameters through SurfaceView APIs. In practice, applications should ensure the output surface maintains the same aspect ratio as the current video and adjust surface dimensions when video properties change.

ExoPlayer Implementation Example

The ExoPlayer project demonstrates sophisticated surface layout management through the following flow:

  1. Resolution Detection: The MediaCodecVideoRenderer class monitors every video buffer for resolution changes and fires callbacks to PlayerView when changes occur.

  2. Layout Management: PlayerView uses an AspectRatioFrameLayout member specifically designed to handle video surface layout changes.

  3. Layout Request: AspectRatioFrameLayout calls requestLayout() to notify the parent View that UI rearrangement is needed.

  4. Measurement Calculation: The View class calls onMeasure(), implemented by AspectRatioFrameLayout, which calculates new surface dimensions based on the updated video size.

surface-resize-diagram

References

[0] MediaFormat.KEY_CROP_BOTTOM

[1] CCodecConfig.cpp

[2] FFmpeg H.264 Metadata Filter

[3] MediaCodec.setVideoScalingMode()

[4] ExoPlayer MediaCodecVideoRenderer

[5] BLASTBufferQueue.cpp