Handling FFmpeg Stream-Copy Failures on Corrupted MTS Files

by

Summary

A production pipeline designed to batch-process and re-containerize AVCHD (MTS) video files failed when encountering a subset of corrupted source files. The automated process, which utilized ffmpeg -i input.mts -c copy output.mts, crashed immediately upon encountering specific files with the error: “Invalid data found when processing input.” This failure highlights a fundamental misunderstanding of how demuxers interact with corrupted file headers and the limitations of stream copying for damaged media.

Root Cause

The failure stems from the way FFmpeg’s demuxer initializes a file. When the command is executed:

  • Header Integrity: FFmpeg first attempts to read the file header (MOOV atom in MP4 or specific metadata blocks in MTS) to understand the stream parameters (codec, bitrate, timestamp info).
  • Parsing Failure: If the file was interrupted during recording (e.g., camera power loss or SD card removal), the index or header is malformed or missing.
  • Fatal Error: Because the demuxer cannot find the structural metadata required to map the packets, it throws a fatal error and terminates the process before it can even attempt the -c copy (stream copy) operation.
  • Stream Copy Limitation: The -c copy flag instructs FFmpeg to bypass re-encoding, but it still requires a validly parsed input stream to identify which packets to copy. If the input cannot be parsed, there is no stream to copy.

Why This Happens in Real Systems

In high-throughput media ingestion pipelines, this is a common occurrence due to:

  • Non-Atomic Writes: Filesystem writes are not always completed. If a device is unplugged, the file remains on disk but lacks the EOF (End of File) markers or closing metadata.
  • Bit Rot/Hardware Failure: Physical degradation of storage media (SD cards, HDDs) can corrupt specific sectors containing critical file headers.
  • Interrupted Ingest: Network jitter or write-buffer overflows during high-bitrate recording can lead to frame synchronization errors or truncated files.

Real-World Impact

  • Pipeline Stalls: A single corrupted file in a batch of thousands can cause an entire automated ingestion job to fail if exception handling is not robust.
  • Data Loss: Without a recovery strategy, “unreadable” files are often flagged as junk and deleted, leading to the permanent loss of high-value raw footage.
  • Increased Latency: Manual intervention by engineers to “fix” files increases the Mean Time to Recovery (MTTR) for media availability.

Example or Code

When the standard stream copy fails, we must attempt to bypass the header or use tools that ignore stream errors. A common “brute force” approach is to use ffmpeg with specific flags to ignore errors, or move to a tool designed for bitstream reconstruction.

# Attempting to force recovery by ignoring errors and attempting to rebuild the container
ffmpeg -err_detect ignore_err -i corrupted_file.mts -c copy recovered_file.mts

# If the header is missing, using a tool like untrunc (for MP4/MOV) or 
# a specialized bitstream parser is often required.

How Senior Engineers Fix It

Senior engineers do not just try different flags; they implement a multi-tiered recovery architecture:

  • Header Reconstruction: Instead of relying on the corrupted file’s own metadata, use a “reference file” (a healthy file from the same camera settings) to rebuild the missing header information.
  • Bitstream Extraction: Use tools that treat the file as a raw bitstream rather than a structured container, ignoring the container’s broken index.
  • Robust Error Handling: In production scripts, implement try-except blocks that catch non-zero exit codes. When a file fails the -c copy test, the system automatically routes it to a “Deep Recovery” worker.
  • Checksum Validation: Implement MD5/SHA verification at the point of ingestion to identify corruption immediately, rather than discovering it during processing.

Why Juniors Miss It

  • Tool Dogmatism: Juniors often assume that if a tool (like FFmpeg) says “Invalid data,” the file is permanently unrecoverable.
  • Lack of Context: They focus on the command syntax (the “how”) rather than the file structure (the “why”).
  • Ignoring the Container: They treat the .mts file as a monolithic block of data rather than a container holding encapsulated bitstreams.
  • Missing the “Reference” Concept: They fail to realize that the metadata needed to fix a file often exists in a known-good sibling file recorded under identical parameters.

Leave a Comment