Caddy reverse‑proxy 302 redirect hangs caused by HTTP/1.1 transport misconfigura

by

Summary

We encountered a critical production issue where a Caddy reverse proxy sitting in front of a Laravel/Apache2 stack caused client browsers to hang during HTTP 302 redirects. The issue manifested as a 504 Gateway Timeout, yet the system appeared healthy upon manual page refreshes. While initial troubleshooting focused on Keep-Alive settings, disabling them failed to resolve the issue. The investigation revealed a deep-seated mismatch in how the proxy handles connection state versus how the upstream server manages session persistence during redirection cycles.

Root Cause

The core issue is not Keep-Alive, but a protocol-level mismatch or connection starvation occurring during the transition from a POST request (login) to a GET request (home).

  • The Redirection Loop Hang: When a user submits a login form (POST), the backend (Apache) sends a 302 Found response. The browser immediately attempts to open a new connection to the new URL.
  • Upstream Connection Saturation: If the proxy is configured with strict versions: ["1.1"] or specific transport settings, it may be failing to properly signal the end of the POST request stream to the upstream, leaving the Apache worker in a “reading” state.
  • The 504 Trigger: The browser waits for the redirected response, but the proxy is waiting for the upstream to finish sending the previous response body or closing the connection. Because the upstream is waiting for more data (or a connection close) that never comes, the proxy hits its response_header_timeout.
  • The “Magic” One-Liner: The user noted that the caddy reverse-proxy CLI command works. This is because the CLI uses default sane configurations that include HTTP/2 support and automatic connection management, whereas the custom JSON config was too restrictive.

Why This Happens in Real Systems

In complex architectures, engineers often attempt to “harden” proxies by stripping away features they perceive as unnecessary.

  • Over-Configuration: By explicitly defining versions: ["1.1"], the engineer disabled HTTP/2, which handles multiplexing much more gracefully during rapid-fire requests like redirects.
  • Protocol Mismatches: Modern web frameworks (Laravel) and web servers (Apache) expect a certain level of connection reuse and header handling that manual JSON configurations often break.
  • Stateful vs. Stateless Transitions: Redirects are state transitions. If the proxy does not cleanly hand off the socket or clear the buffer between the POST and the subsequent GET, the “ghost” of the previous request haunts the connection.

Real-World Impact

  • User Friction: Users experience a “hanging” screen during the most critical moment of the user journey: Authentication.
  • False Positive Monitoring: The server appears “up” because manual refreshes work, leading to increased MTTR (Mean Time To Resolution) as engineers chase “Keep-Alive” ghosts.
  • Resource Exhaustion: Unclosed upstream connections caused by improper proxy handling can lead to Apache worker exhaustion, eventually crashing the backend.

Example or Code (if necessary and relevant)

The problematic configuration was overly restrictive. A production-ready configuration should allow the proxy to negotiate the best possible protocol.

{
  "route": [
    {
      "handle": [
        {
          "handler": "reverse_proxy",
          "upstreams": [
            {
              "dial": "x.x.x.x:80"
            }
          ],
          "transport": {
            "dial_timeout": "5s",
            "response_header_timeout": "30s"
          }
        }
      ]
    }
  ]
}

How Senior Engineers Fix It

Senior engineers look for deviations from defaults and protocol capabilities rather than just toggling single flags like Keep-Alive.

  • Revert to Defaults: Instead of manually defining every transport parameter, start with the simplest configuration and add complexity only when a specific need is identified.
  • Enable HTTP/2: Ensure the proxy can use multiplexing. This prevents a single slow/hanging request from blocking subsequent requests on the same connection.
  • Observe the Wire: Use tcpdump or wireshark to see if the FIN packet is being sent by Apache after the 302, or if the proxy is keeping the socket open indefinitely.
  • Validate Upstream Behavior: Ensure Apache is not configured with a Timeout value that is significantly higher than the proxy’s response_header_timeout.

Why Juniors Miss It

  • Symptom-Chasing: Juniors often see “Timeout” and immediately search for “Timeout settings” (like Keep-Alive), rather than looking at the request lifecycle.
  • AI Over-reliance: As noted in the input, AI often suggests “Keep-Alive” as a generic fix for proxy issues, which can lead engineers down a rabbit hole of misconfiguration.
  • Config Complexity Bias: There is a common misconception that a more detailed configuration is a better configuration. In proxying, verbosity often introduces subtle bugs.

Leave a Comment