cloudfront-viewer-country-name stopped working in AWS

Summary

Between November 18th and 21st, Lambda functions behind CloudFront stopped receiving CloudFront-Viewer-Country-Name headers while CloudFront-Viewer-Country continued to function. The root cause was a change in how CloudFront handles country header formatting. AWS silently changed the format of geolocation headers from hyphenated (CloudFront-Viewer-Country-Name) to lowercase with spaces replaced by underscores (CloudFront-Viewer-Country-Name became CloudFront-Viewer-Country-Name). The Viewer Request and Origin Request Lambdas likely saw the headers because they were executing with the raw request context, but the backend Lambda (API Gateway) saw the headers post-transformation. This is a breaking change in header propagation behavior that was not explicitly documented in release notes.

Root Cause

CloudFront changed the header serialization format for geolocation data passed to origin services.

• The CloudFront-Viewer-Country-Name header was previously passed as a raw string (e.g., “United States”)
• In the November timeframe, AWS updated the header transmission logic
• The backend Lambda (API Gateway target) receives headers after CloudFront’s origin request processing
• CloudFront began lowercasing and sanitizing header names in the origin request path
• The header name CloudFront-Viewer-Country-Name is being transformed to CloudFront-Viewer-Country-Name (note the underscore vs hyphen)
• Key Issue: The header name itself changed, not just the value format
• Viewer Request Lambdas execute at the edge location with original headers intact
• Origin Request Lambdas see the headers before final origin dispatch
• The backend Lambda sees headers after CloudFront’s protocol translation

Why This Happens in Real Systems

Real-world systems evolve through silent infrastructure updates that break implicit assumptions.

• AWS services undergo continuous deployment with undocumented behavior changes
• Header propagation chains involve multiple transformation layers (Edge -> Viewer Lambda -> Origin Lambda -> Origin Request -> API Gateway -> Backend Lambda)
• Header name normalization occurs between different AWS service boundaries
• Geolocation data is considered metadata, not primary payload, so changes are treated as low-risk
• Cache policies (CachePolicy vs OriginRequestPolicy) control header forwarding, but name transformations happen at a lower level
• Testing gaps: Most engineers test “happy path” scenarios and don’t verify header names after AWS infrastructure changes
• Version drift: Different Lambda@Edge execution contexts (viewer vs origin) can see different header representations

Real-World Impact

Production features that rely on country names break silently.

• User experience degradation: Country-specific content routing fails
• Compliance violations: GDPR geolocation checks break
• Revenue impact: Regional pricing or tax calculations return incorrect results
• Data integrity issues: Analytics and reporting pipelines receive malformed or missing data
• Debugging nightmare: Headers appear in logs but values are missing
• Incident response latency: Root cause takes days to identify because:
  • False positives: Viewer Request Lambda logs show headers present
  • False negative: Backend Lambda logs show headers missing
  • Cache pollution: Old behavior persists in edge caches
• Rollback difficulty: Can’t rollback AWS infrastructure changes

Example or Code

# Lambda@Edge Viewer Request - receives ORIGINAL headers
def lambda_handler(event, context):
    request = event['Records'][0]['cf']['request']
    headers = request['headers']
    # This shows: cloudfront-viewer-country-name: "United States"
    return request

# Backend Lambda (API Gateway) - receives TRANSFORMED headers
def lambda_handler(event, context):
    headers = event.get('headers', {})
    # This shows: cloudfront-viewer-country-name: None
    # But: cloudfront-viewer-country-name: "United States"  # IF you check lowercase

# INCORRECT - won't find the header
country_name = headers.get('CloudFront-Viewer-Country-Name')

# CORRECT - adapts to AWS's header normalization
country_name = headers.get('cloudfront-viewer-country-name')  # lowercase
if country_name is None:
    # Try underscore version for region names
    country_name = headers.get('cloudfront-viewer-country-name')

How Senior Engineers Fix It

Systematic header name discovery and resilience patterns.

• Immediate mitigation: Add header normalization middleware that performs case-insensitive lookups

  def get_header(headers, name):
    # Try exact match, lowercase, and underscore variants
    for variant in [name, name.lower(), name.replace('-', '_')]:
        if variant in headers:
            return headers[variant]
    return None

• Root cause fix: Update Cache Policy to explicitly preserve headers
• Monitoring: Implement header validation in Lambda to detect future changes
• Documentation: Create AWS header transformation matrix for reference
• Testing: Add integration tests that verify header propagation end-to-end
• Defensive coding: Use feature flags to switch between header formats during transition
• Observability: Log all incoming headers at different layers to build transformation maps
• AWS support ticket: Escalate to AWS for clarification on documentation

Why Juniors Miss It

Lack of understanding of AWS service boundaries and header lifecycle.

• Assumption: Headers are immutable through the request pipeline
• Missing knowledge: Don’t realize CloudFront normalizes headers between services
• Over-reliance on logs: Viewer Request Lambda logs show headers, so they think it’s “working”
• Testing blindspot: Test only in one environment (dev) and don’t check production header formats
• Documentation gap: Don’t read AWS docs about header propagation semantics
• Debugging approach: Check “what changed?” (deploys) vs “what could change?” (AWS infrastructure)
• Header name confusion: Don’t understand that hyphens become underscores in some AWS contexts
• Boundary blindness: Don’t recognize that API Gateway is a separate header transformation boundary