Pico 2W MbedTLS LwIP Crt verification fails on Supabase POST

by

Summary

A developer using a Raspberry Pi Pico 2W with MbedTLS and LwIP encountered certificate verification failures when connecting to a Supabase backend, while connections to other APIs functioned correctly. The specific error indicated an expired certificate despite the system time being correctly synchronized via SNTP. The issue was identified as failed certificate chain validation due to the absence of the correct Root Certificate Authority (CA) in the device’s trusted store, combined with potential server name indication (SNI) and intermediate certificate issues specific to Supabase’s TLS setup.

Root Cause

The primary failure is certificate chain trust validation in MbedTLS. While the device maintains correct system time (disproving simple clock skew), the TLS handshake fails because the device cannot verify the Supabase server certificate against its trusted root store.

  • Missing Root CA: Supabase typically uses certificates issued by Let’s Encrypt (or specific cloud providers). The Pico’s trusted certificate store likely contains standard browser/OS roots (DST Root CA X3, ISRG Root X1) but may lack the specific R13 or WE1 roots mentioned, or the chain requires an intermediate CA not present on the device.
  • Incomplete Certificate Chain: If the device only trusts the Root CA but the server sends an incomplete chain (missing intermediates), MbedTLS cannot build a path to the trusted root, resulting in a verification error.
  • Common Name (CN) / Subject Alternative Name (SAN) Mismatch: MbedTLS validates the server hostname strictly. If the certificate presented by Supabase does not match the hostname being requested (e.g., connecting to an IP or a mismatched subdomain), verification fails.
  • MbedTLS Configuration Limits: The default MbedTLS configuration on Pico SDK often has limited heap and stack space, and may not support large certificate chains or specific algorithms used by modern CAs if not explicitly enabled.

Why This Happens in Real Systems

This scenario is a classic “it works here, fails there” distributed systems problem.

  • Certificate Authority Rotation: Cloud providers like Supabase periodically rotate their intermediate certificates or root CAs. A certificate bundle generated months ago may be stale and no longer valid for current connections.
  • Embedded Constraints: Unlike desktop OSes which have massive trusted stores updated automatically, embedded devices (Pico) have static trust stores. Developers must manually curate and inject the exact CA chain required by the target server.
  • TLS Stack Complexity: MbedTLS is highly configurable. The default “reference” configuration provided by Raspberry Pi Pico SDK is often optimized for code size, not interoperability. Features like Server Name Indication (SNI) or OCSP stapling may be disabled by default, causing failures with strict servers like Supabase.

Real-World Impact

  • Service Outage: The application cannot communicate with the backend, rendering IoT devices non-functional or “offline.”
  • Security Risks: Attempting to debug this often leads developers to disable certificate verification (VERIFY_NONE) as a temporary fix. This exposes the device to Man-in-the-Middle (MitM) attacks and should never be used in production.
  • Resource Wastage: Hours are wasted debugging system time (NTP) when the actual issue is cryptographic trust.
  • Supply Chain Complexity: Managing and embedding specific CA certificates into firmware binaries increases build complexity and firmware size.

Example or Code

To debug and fix this, you must inspect the certificate chain presented by the server and configure MbedTLS correctly.

1. Inspecting the Certificate Chain (OpenSSL)
Run this command from a PC to see exactly what certificates Supabase sends:

openssl s_client -connect .supabase.co:443 -showcerts

Look for the -----BEGIN CERTIFICATE----- blocks. You need the Root CA and usually at least one Intermediate CA.

2. MbedTLS Certificate Configuration (C Code)
Do not rely on the default mbedtls_x509_crt_parse with a single cert. You often need to parse a “chain” file containing Root and Intermediates.

#include "mbedtls/ssl.h"
#include "mbedtls/x509_crt.h"

// Global TLS context and certificate structures
mbedtls_ssl_context ssl;
mbedtls_ssl_config conf;
mbedtls_x509_crt cacert;

// In your initialization function:
void setup_tls() {
    int ret;

    // 1. Initialize structures
    mbedtls_ssl_init(&ssl);
    mbedtls_ssl_config_init(&conf);
    mbedtls_x509_crt_init(&cacert);

    // 2. Load the CA Certificate Chain
    // The string `supabase_ca_chain_pem` must contain the Root and Intermediate certs
    // concatenated together (often obtained from `openssl s_client` or Supabase docs).
    ret = mbedtls_x509_crt_parse(&cacert, 
                                 (const unsigned char *)supabase_ca_chain_pem, 
                                 strlen(supabase_ca_chain_pem) + 1);
    if (ret < 0) {
        printf(" failed\n  ! mbedtls_x509_crt_parse returned -0x%x\n\n", -ret);
        return;
    }

    // 3. Setup SSL Configuration
    ret = mbedtls_ssl_config_defaults(&conf,
                                      MBEDTLS_SSL_IS_CLIENT,
                                      MBEDTLS_SSL_TRANSPORT_STREAM,
                                      MBEDTLS_SSL_PRESET_DEFAULT);
    if (ret != 0) {
        printf(" failed\n  ! mbedtls_ssl_config_defaults returned %d\n\n", ret);
        return;
    }

    // 4. CRITICAL: Set authentication mode to REQUIRED
    mbedtls_ssl_conf_authmode(&conf, MBEDTLS_SSL_VERIFY_REQUIRED);

    // 5. Set the trusted CA chain
    mbedtls_ssl_conf_ca_chain(&conf, &cacert, NULL);

    // 6. Set the RNG (Entropy/CTR_DRBG) - Required for TLS
    // (Assuming rng_ctx is initialized elsewhere)
    mbedtls_ssl_conf_rng(&conf, mbedtls_ctr_drbg_random, &ctr_drbg);

    // 7. Set the SSL Context
    ret = mbedtls_ssl_setup(&ssl, &conf);
    if (ret != 0) {
        printf(" failed\n  ! mbedtls_ssl_setup returned %d\n\n", ret);
        return;
    }

    // 8. Set SNI (Server Name Indication)
    // Supabase requires this to route to the correct virtual host
    mbedtls_ssl_set_hostname(&ssl, ".supabase.co");
}

3. Synchronization Check (System Time)
Ensure time is set before initializing the TLS stack.

#include "pico/util/datetime.h"
#include "hardware/rtc.h"

// Example check (simplified)
void check_time() {
    datetime_t t;
    rtc_get_datetime(&t);
    // Ensure year is > 2020. If 1970, SNTP failed.
    if (t.year < 2020) {
        printf("Error: Time not synchronized. Certificate validation will fail.\n");
    }
}

How Senior Engineers Fix It

Senior engineers approach this systematically rather than guessing which certificate to use.

  1. Capture the Exact Chain: Use openssl s_client to dump the full certificate chain presented by the target server. Do not trust generic “Root CA” files; get the specific chain used by the service at this moment.
  2. Embed the Chain in Firmware: Create a certs.h file containing the PEM strings of the Root and Intermediate certificates. Parse them using mbedtls_x509_crt_parse in a loop or concatenated string.
  3. Enable Debug Logging: Temporarily increase MbedTLS debug verbosity to trace the handshake. 
    mbedtls_ssl_conf_dbg(&conf, my_debug, NULL);
mbedtls_debug_set_threshold(4); // High verbosity
  4. Verify Hostname explicitly: Ensure mbedtls_ssl_set_hostname matches the exact URL used in the connection. If using a custom domain proxying to Supabase, ensure the SNI matches the SSL certificate on the proxy.
  5. Update MbedTLS Configuration: If using pico-sdk, ensure MBEDTLS_SSL_TLS_C, MBEDTLS_X509_CRT_PARSE_C, and MBEDTLS_SHA256_C are enabled in mbedtls_config.h.

Why Juniors Miss It

  • Symptom vs. Cause Confusion: The error message “Certificate Expired” or “Verification Failed” is interpreted literally as the server cert being expired, or the system time being wrong. Juniors often spend hours tweaking NTP logic when the issue is a missing Intermediate CA.
  • Assuming Defaults Work: Believing that the Pico SDK’s default TLS configuration is sufficient for all websites. It is not.
  • PEM vs. DER Confusion: Attempting to load binary DER certificates directly without base64 decoding, or failing to null-terminate the PEM string before parsing.
  • Ignoring SNI: Forgetting that modern cloud hosts (like Supabase) use SNI to serve multiple domains on one IP. Without setting the hostname in MbedTLS, the server may default to a generic certificate that doesn’t match your project URL.