using ‘reput’ in sftp transfer in RHEL7

by

Summary

An application hosted on RHEL7 experienced critical failures during large file transfers via SFTP. When transfers stalled, the engineering team attempted to use the reput command to resume the interrupted upload. However, the reput operation failed to resume the transfer from the last byte, effectively failing to provide the intended resiliency. The investigation focused on whether this was an OS-level limitation or a protocol implementation issue.

Root Cause

The failure of reput in this specific context is rarely an OS limitation (RHEL7) and almost always a protocol mismatch or server-side configuration issue. The primary causes include:

  • Lack of Offset Support: The remote SFTP server may not support the SSH_FXP_WRITE command with a specific file offset. If the server only supports writing from the beginning of a file, resuming is impossible.
  • File Truncation/Overwrite Policies: Many SFTP server implementations default to overwriting the file upon a new write request rather than appending to the existing byte stream.
  • Filesystem Limitations: If the target filesystem does not support sparse files or efficient seeking, the reput command (which relies on seeking to a specific offset) will fail or restart from zero.
  • Client-Side Implementation: The specific SFTP client being used by the application may not be correctly calculating the local vs. remote byte offset before initiating the re-upload.

Why This Happens in Real Systems

In production environments, distributed systems introduce several layers of “silent” failure:

  • State Inconsistency: A network hiccup may leave a file on the remote server in a partially written state. If the client doesn’t verify the exact size of that partial file, it cannot “re-put” correctly.
  • Middleware Interference: Load balancers, firewalls, or Deep Packet Inspection (DPI) engines can terminate long-lived TCP connections, causing the “stall” that necessitates a reput in the first place.
  • Version Drift: Discrepancies between the OpenSSH version on the RHEL7 client and the version on the destination server can lead to unsupported feature sets during the SSH handshake.

Real-World Impact

  • Data Corruption: If reput fails and the application incorrectly assumes success, it may lead to incomplete datasets being processed by downstream systems.
  • Increased Latency: Repeatedly restarting large transfers from 0% instead of 90% consumes massive amounts of bandwidth and I/O.
  • Operational Overhead: SRE and Production teams are forced into manual intervention to clean up partial files and re-trigger jobs, increasing the Mean Time To Recovery (MTTR).

Example or Code

To debug whether the server supports seeking/offsetting, use a standard OpenSSH client to check the file size and attempt a manual append:

# 1. Check the size of the partially uploaded file on the remote server
ssh user@remote-server "stat -c%s /path/to/partial_file"

# 2. Use sftp in interactive mode to attempt to resume (if supported)
sftp user@remote-server
sftp> reput local_large_file.dat

How Senior Engineers Fix It

Senior engineers look past the “command failure” and address the architectural weakness:

  • Implementing Checksums: Instead of relying on reput, implement a sidecar checksum file (.md5 or .sha256). Only consider a transfer “complete” if the remote hash matches the local hash.
  • Moving to Object Storage: For large files, move away from SFTP toward S3-compatible object storage which handles multipart uploads and part-level retries natively.
  • Atomic Renames: Always upload files with a .tmp extension and use a rename operation only after the transfer and checksum are verified. This prevents downstream processes from consuming partial files.
  • Observability: Implement connection timeout monitoring to distinguish between a “stalled” transfer and a “dead” connection.

Why Juniors Miss It

  • Blaming the OS: Juniors often assume a “Legacy OS” like RHEL7 is the culprit, rather than investigating the application logic or the remote server’s capabilities.
  • Tool-Centric Thinking: They focus on making a specific command (reput) work, whereas seniors focus on the reliability of the data transfer lifecycle.
  • Ignoring Idempotency: Juniors often fail to design for idempotent operations—the ability to run the same transfer multiple times without causing errors or duplication.

Leave a Comment