Resolve Ghost Submissions: Enforce Atomic File Uploads

by

Summary

During a high-stakes hackathon hosted on the Unstop platform, the organizing team experienced intermittent data loss during the PPT submission phase. While participants received “success” confirmations, the corresponding files and metadata failed to appear in the administrative dashboard. This incident resulted in data inconsistency, broken submission links, and significant operational friction for the judging panel.

Root Cause

The failure stems from a distributed state mismatch between the client-side UI and the backend persistence layer, likely triggered by one of the following:

  • Asynchronous Race Conditions: The client-side application likely utilized AJAX to upload files, providing a “Success” message as soon as the request was sent, rather than waiting for the backend to confirm the file was successfully written to the Object Storage (S3/GCS) and the database record was committed.
  • Partial Upload Failures: Large PPT files may encounter network timeouts or packet loss. If the error handling logic only checks for the initiation of the request rather than the integrity of the completed write, the user sees a false positive.
  • Database Transaction Atomicity Issues: The system might upload the file to storage successfully but fail to write the metadata to the relational database. Without an atomic transaction spanning both storage and database, the file becomes “orphaned” and invisible to the dashboard.

Why This Happens in Real Systems

In high-concurrency environments like hackathons, systems face extreme traffic spikes that expose architectural flaws:

  • Non-Atomic Operations: Many web applications treat “File Upload” and “Database Update” as two separate, non-transactional events. If the second fails, the first is not rolled back.
  • Optimistic UI Updates: To make apps feel “snappy,” engineers often implement Optimistic UI, where the interface assumes success before the server responds. Under heavy load, this leads to a massive discrepancy between user perception and actual system state.
  • Resource Contention: Large file uploads consume I/O bandwidth and memory. Under load, the connection between the application server and the storage bucket may throttle, leading to silent failures in the middle of the request lifecycle.

Real-World Impact

  • Loss of Integrity: The “Source of Truth” (the dashboard) becomes unreliable, making it impossible to audit submissions.
  • Participant Frustration: Users experience “Ghost Submissions,” where they believe they have complied with rules, only to be disqualified later.
  • Operational Overhead: The organizing team must manually intervene, cross-referencing emails or support tickets to reconstruct the submission list.
  • Reputational Damage: For large-scale events, technical failures in the submission pipeline undermine the credibility of the entire platform.

Example or Code

// ANTI-PATTERN: Optimistic success without verifying backend persistence
async function handleUpload(file) {
  const formData = new FormData();
  formData.append('ppt', file);

  // The UI shows success immediately after the AJAX call is initiated
  // or if the request simply reaches the gateway, not the DB.
  fetch('/api/submit', {
    method: 'POST',
    body: formData
  }).then(() => {
    showSuccessMessage("Submission Successful!"); // FALSE POSITIVE RISK
  }).catch(err => {
    showErrorMessage("Upload failed.");
  });
}

// BEST PRACTICE: Verify completion and handle atomicity
async function handleUploadReliably(file) {
  try {
    const response = await fetch('/api/submit', {
      method: 'POST',
      body: file
    });

    if (!response.ok) {
      throw new Error(`Server responded with ${response.status}`);
    }

    const result = await response.json();

    // Verify that the backend explicitly confirms DB persistence
    if (result.status === 'COMMITTED') {
      showSuccessMessage("Submission Confirmed and Saved.");
    } else {
      throw new Error("File uploaded but metadata failed to save.");
    }
  } catch (error) {
    logErrorToSentry(error);
    showErrorMessage("Submission failed. Please try again.");
  }
}

How Senior Engineers Fix It

To prevent this, a senior engineer would implement a robust submission pipeline:

  • Implement Idempotency Keys: Ensure that if a user retries a failed upload, the system recognizes it as the same submission and doesn’t create duplicate or corrupted entries.
  • Use Two-Phase Commits or Sagas: Ensure that the file upload to storage and the database entry creation are treated as a single logical unit. If the DB write fails, the uploaded file is cleaned up (or vice versa).
  • Server-Side Validation (Checksums): The client should send an MD5 hash of the file. The server calculates the hash after the upload and compares them to ensure the file wasn’t corrupted in transit.
  • Webhook/Polling Architecture: Instead of a simple AJAX POST, use an architecture where the client receives a “Processing” status and polls a status endpoint until the backend confirms the record is fully indexed and visible.
  • Fallback Redundancy: For high-stakes events, provide a secondary, low-bandwidth fallback (e.g., a dedicated Google Form or a direct cloud storage link) as a contingency.

Why Juniors Miss It

  • Focus on the “Happy Path”: Juniors often write code that works perfectly under low load and perfect network conditions, failing to account for edge cases like partial timeouts.
  • Over-reliance on Client-Side Feedback: They often assume that if the fetch call doesn’t throw an error, the data is safely stored, neglecting the complex chain of events occurring on the backend.
  • Ignoring Observability: Juniors might notice an error in the console but fail to implement server-side logging (like ELK or Sentry) that would reveal the mismatch between the storage layer and the database.
  • Underestimating Concurrency: They often view a submission as a single user action rather than one of thousands of simultaneous requests competing for the same database connections and I/O throughput.

Leave a Comment