Addressing eBPF SK_SKB Arithmetic Mistakes in BPF Verification

by

Summary

A production incident involving an eBPF SK_SKB program (specifically a parser/verdict program) revealed a critical misunderstanding of how context pointer arithmetic behaves within the BPF verifier and the kernel runtime. The developer observed that while skb->len returned the expected packet length, the expression data_end - data consistently evaluated to zero, leading to incorrect logic in packet inspection routines.

Root Cause

The issue stems from a fundamental misunderstanding of how pointer casting and arithmetic interact with the BPF verifier and the underlying __sk_buff structure.

  • Context vs. Memory Address: In SK_SKB programs, skb->data and skb->data_end are provided as offsets or hardware-specific pointers that are not guaranteed to be a direct, contiguous memory mapping in the way a standard userspace buffer is.
  • Verifier Optimization: The BPF verifier often treats skb->data and skb->data_end as opaque bounds. When performing arithmetic like data_end - data, the verifier cannot always guarantee that these two pointers belong to the same scalar range or that they represent a linear, contiguous memory segment without explicit bounds checking.
  • Lack of Bounds Validation: Without an explicit check (e.g., if (data + offset > data_end)), the verifier prevents the program from assuming the relationship between these two pointers to maintain memory safety.
  • The Zero Result: The “zero” result often occurs because the arithmetic operation is being performed on values that the verifier has constrained or because the underlying implementation of the SK_SKB context uses a relative offset model where the arithmetic doesn’t map to a standard integer difference unless properly cast and verified.

Why This Happens in Real Systems

In high-performance networking, we move away from standard socket buffers to highly optimized, often fragmented or non-contiguous memory layouts.

  • Memory Abstraction: The __sk_buff structure is an abstraction. The actual packet data might be spread across multiple skb fragments (frags).
  • Verifier Constraints: To ensure the kernel never crashes from an out-of-bounds access, the verifier enforces a “check-before-use” policy. If you do not prove that data + N is less than data_end, the verifier treats the range as invalid or zero-length to prevent exploitation.
  • Context Specialization: Different BPF program types (e.g., XDP vs SK_SKB) handle data pointers differently. XDP provides direct hardware DMA pointers, whereas SK_SKB works with the kernel’s networking stack abstraction, which is significantly more complex.

Real-World Impact

  • Silent Logic Failure: As seen in this case, a program might report a packet length of 0, causing firewall rules to skip inspection or load balancers to misroute traffic.
  • Security Vulnerabilities: If an engineer assumes data_end - data is safe to use for buffer allocation or loop bounds without proper verification, it can lead to out-of-bounds reads/writes in the kernel.
  • Performance Degradation: Attempting to bypass these checks using inefficient methods (like constant bpf_skb_load_bytes calls instead of direct pointer access) can increase CPU cycles per packet.

Example or Code

The correct way to handle this is to perform a bounds check that satisfies the verifier, allowing it to track the pointer range.

SEC("sk_skb/correct_parser")
int bpf_prog_parser(struct __sk_buff *skb)
{
    void *data = (void *)(long)skb->data;
    void *data_end = (void *)(long)skb->data_end;

    // The critical step: The Verifier needs this check to 
    // establish the relationship between data and data_end.
    if (data + skb->len > data_end) {
        return SK_PASS; 
    }

    // After the check, the verifier knows 'data' is within bounds.
    // Now, calculating the length is mathematically sound for the verifier.
    u32 actual_len = (u32)((void *)data_end - data);

    bpf_printk("PARSER, len: %d, diff: %d", skb->len, actual_len);

    return skb->len;
}

How Senior Engineers Fix It

Senior engineers do not fight the verifier; they work with it.

  • Explicit Bounds Checking: Always implement the if (ptr + offset > end) pattern immediately after extracting pointers. This is the “contract” with the verifier.
  • Using Helper Functions: When dealing with complex packet structures, use bpf_skb_pull_data() to ensure the entire header/data range you need is linearized in the kernel before attempting to access it.
  • Linearization Awareness: Understand that SK_SKB programs may need to call bpf_skb_pull_data to move fragments into a contiguous linear area if the packet is fragmented.
  • Type Safety: Use explicit casting to void * or specific struct pointers only after the bounds check has been validated by the verifier.

Why Juniors Miss It

  • Assuming Userspace Semantics: Juniors often treat __sk_buff like a standard C struct in a userspace application, forgetting that BPF is a restricted execution environment with a mathematical verifier.
  • Ignoring the Verifier Logs: When the verifier rejects a program or the logic fails, juniors often try to “tweak” the math rather than reading the verifier’s instruction trace to see exactly where the pointer range was lost.
  • Confusing XDP and SK_SKB: There is a tendency to apply XDP logic (where data is a simple, contiguous xdp_md buffer) to SK_SKB logic (where data is an abstracted, potentially fragmented kernel buffer).

Leave a Comment