CiliumIngressIPMissedinWireGuardAllowedIPsClausewithgrowingKubernetesCluster

by

Summary

An RKE2 cluster utilizing Cilium with WireGuard encryption enabled experienced intermittent connectivity failures (connection timeouts and resets) following the addition of a second node. The issue manifested as upstream connect error when accessing services like Rancher via Ingress. Investigation revealed that the WireGuard interface was dropping legitimate packets because the source IP of the Ingress traffic was not included in the peer’s AllowedIPs list.

Root Cause

The root cause is a mismatch between the Cilium networking model and the WireGuard cryptographic routing policy.

  • WireGuard’s AllowedIPs Mechanism: WireGuard uses a concept called “Cryptokey Routing.” It only accepts incoming packets if the source IP address of the packet matches an entry in the AllowedIPs list for that specific peer.
  • Cilium Ingress IP: When Cilium performs certain routing operations or handles Ingress traffic, it uses a specific Cilium Internal Ingress IP (e.g., fd00:0:1:1:1::d439).
  • Configuration Gap: The Cilium Helm deployment was configured to encrypt node-to-node traffic, but the automation responsible for updating the WireGuard AllowedIPs list only accounted for the Node Internal IP and the Pod CIDR ranges. It failed to include the specialized Cilium Ingress IP in the peer configuration.
  • Packet Rejection: When Node 2 attempted to send traffic to Node 1 using its Ingress IP, Node 1’s kernel inspected the packet, checked the cilium_wg0 interface, saw that fd00:0:1:1:1::d439 was not in the AllowedIPs for the peer, and dropped it with a Packet has unallowed src IP error.

Why This Happens in Real Systems

This issue occurs due to the complexity of multi-layered networking abstractions:

  • Overlay vs. Underlay: We often assume that if the “Underlay” (Node IPs) is connected, the “Overlay” (Cilium/Pods) will work. However, security protocols like WireGuard operate at a layer that requires explicit knowledge of every possible source IP.
  • Dynamic IP Allocation: Modern CNIs (Container Network Interfaces) like Cilium do not just use Node IPs; they inject specialized IPs for Ingress, LoadBalancers, and Service VIPs.
  • Protocol Strictness: WireGuard is a Zero Trust protocol by design. Unlike traditional routing where a packet is simply forwarded if a route exists, WireGuard adds a cryptographic validation step that acts as a strict firewall based on source IP.

Real-World Impact

  • Service Unavailability: Critical applications (like Rancher or Ingress Controllers) become unreachable despite the pods being “Running” and the network being “Up.”
  • Silent Failures: The application layer reports “Connection Timeout,” which often leads engineers to falsely blame the application code or the Load Balancer rather than the network encryption layer.
  • Scaling Blockers: The cluster may function perfectly with a single node, but horizontal scaling introduces new peer relationships and new IP addresses, causing the system to fail only when it grows.

Example or Code

To diagnose this, the engineer used the kernel dynamic debug interface to unmask WireGuard’s internal logic:

# Enable WireGuard debugging to see dropped packets due to AllowedIPs mismatch
echo "module wireguard +p" > /sys/kernel/debug/dynamic_debug/control

# Monitor the kernel logs for the specific error
dmesg -w | grep -i wireguard

The resulting error confirms the mismatch:
[12808.903589] wireguard: cilium_wg0: Packet has unallowed src IP (fd00:0:1:1:1::d439) from peer 1

How Senior Engineers Fix It

A senior engineer solves this by ensuring the IPAM (IP Address Management) and the Encryption Policy are synchronized.

  • Immediate Fix: Manually or via automation, ensure the CiliumInternalIP and Ingress IP ranges are included in the WireGuard peer configuration.
  • Long-term Fix (Cilium Configuration): In Cilium, this is often handled by ensuring the IPv6 Native Routing CIDR and the Pod CIDRs sufficiently encompass all possible internal IPs used by the agent.
  • Verification: Use wg show to verify that the allowed ips list for every peer is a superset of all IPs the node might use to communicate (Node IP, Pod IPs, Service VIPs, and Ingress IPs).
  • Infrastructure as Code (IaC): Adjust the Helm values.yaml to ensure that the ipv6NativeRoutingCIDR is wide enough to cover the specialized Cilium IP ranges, or ensure the CNI version is patched to correctly propagate these IPs to the WireGuard interface.

Why Juniors Miss It

  • Observability Gap: Juniors typically look at kubectl get pods or kubectl get svc. If those look “Green,” they assume the network is fine. They fail to look at kernel-level logs (dmesg).
  • Layer Confusion: They assume a “Network Error” is a routing issue (Layer 3) or a connectivity issue (Layer 1/2), whereas this is a cryptographic policy issue (Layer 4/Security).
  • Assumption of Transparency: They assume encryption is “transparent” and won’t change the fundamental behavior of how packets are validated, failing to realize that WireGuard turns routing into an identity-based system.

Leave a Comment