Pipeline selects wrong “latest” file after checkout

by

Summary

After a new .tar.gz file was committed to Azure DevOps, the CI pipeline failed to select the latest version using ls -t. Instead of picking the newest file (test8.6.tar.gz), the pipeline consistently selected an older version (test8.3.tar.gz), causing deployment of outdated artifacts.

Root Cause

The root cause was timestamp equality during checkout. Azure DevOps sets identical last modified timestamps for all files during repository checkout. When ls -t sorts files by modification time:

  • All files shared the same timestamp from the latest commit
  • With identical timestamps, ls defaults to alphabetical order (test8.3.tar.gz precedes test8.6.tar.gz)
  • Alphabetical ordering became the implicit secondary sort criterion

Why This Happens in Real Systems

  1. VCS timestamp behavior:
    • Version control systems reset file timestamps during checkout to maintain reproducibility
    • All files inherit commit time, removing original build timestamps
  2. Race conditions:
    • CI environments isolate file modifications via checkout, decoupling from real build times
    • Local builds retain chronological order via OS timestamps
  3. Filesystem limitations:
    • Storing sub-second precision varies across filesystems
    • Directory listing order is undefined when millisecond timestamps match

Real-World Impact

  • Outdated deployments: Customers receive old software versions despite new pushes
  • Silent failures: Pipelines succeed but deploy incorrect artifacts
  • Version drift: Production runs unsanctioned combinations of application assets
  • Debugging overhead:
    • Errors manifest later in workflow stages
    • Non-obvious link between deployment failures and source control interactions

Example or Code

Broken pipeline command: 

pkg=$(ls -t packages/*.tar.gz | head -n 1)

Fixed pipeline command (version-sort): 

pkg=$(ls packages/*.tar.gz | sort -V | tail -n 1)

How Senior Engineers Fix It

  1. Eliminate timestamp dependencies:
    • Use semantic sorting (sort -V) for versioned filenames
    • Store build artifacts with embedded timestamps/versions (e.g., artifact-20240522.1.tar.gz)
  2. Leverage pipeline metadata: 
    # Use triggered commit's file modifications  
changed_files=$(git diff --name-only $PREV