User Safety: safe

by

Summary

A StarRocks cluster in a UAT environment experienced a critical failure where the Backend (BE) nodes reported zero total capacity despite being successfully registered with the Frontend (FE). This resulted in a cluster that appeared “healthy” from a connectivity standpoint but was functionally useless for any data ingestion or query execution, as the system believed it had no storage resources available.

Root Cause

The issue stems from a misalignment between the BE process’s view of the filesystem and the actual volume mounting strategy used in the Docker container. Specifically:

  • Filesystem Permission Mismatch: The BE process inside the Docker container runs under a specific UID/GID. When mounting a local data directory from the host (IP B and C), the containerized process often lacks the write permissions required to initialize the storage metadata.
  • Incorrect Metadata Initialization: StarRocks BE requires the ability to create and write to its storage directory to calculate capacity. If the process cannot write the metadata files or segment files, the internal capacity counter remains at zero.
  • Path Misconfiguration: The be.conf file inside the container may be pointing to a path that does not correctly map to the Docker volume mount point, causing the BE to look for data in a directory that is effectively empty or read-only.

Why This Happens in Real Systems

In distributed database deployments, this is a classic Environment Parity Gap.

  • Abstraction Leaks: Docker abstracts the OS, but it does not abstract Linux kernel permissions or Filesystem ownership.
  • Volume Mapping Complexity: When transitioning from “Native” (running on bare metal/VM) to “Containerized” (running in Docker), the pathing logic changes. A path like /home/starrocks/data on the host is not the same as /opt/starrocks/be/storage inside a container unless explicitly mapped.
  • Silent Failures in Distributed Systems: The FE (Frontend) only cares if the BE is “alive” via heartbeat. It does not inherently know if the BE’s internal storage engine has failed to initialize its disk capacity, leading to a Partial Success state where the node is “Up” but “Empty”.

Real-World Impact

  • Immediate Service Unavailability: Any INSERT or CREATE TABLE statement will fail immediately because the scheduler cannot find a backend with available capacity.
  • False Positives in Monitoring: Standard “Process Up/Down” checks will pass, making the issue harder to detect via automated heartbeat monitoring.
  • Deployment Delays: In CI/CD pipelines, this can lead to “Ghost Deployments” where the infrastructure is provisioned successfully, but the application layer is non-functional.

Example or Code

services:
  starrocks-be:
    image: starrocks/be-ubuntu:latest
    container_name: starrocks_be_1
    volumes:
      - /mnt/data/starrocks/be/storage:/opt/starrocks/be/storage
      - /mnt/data/starrocks/be/conf:/opt/starrocks/be/conf
    user: "1000:1000"
    environment:
      - BE_CONF_PATH=/opt/starrocks/be/conf/be.conf
    command: ["/opt/starrocks/be/bin/start_be.sh", "--daemon"]

How Senior Engineers Fix It

A senior engineer approaches this by validating the Data Plane rather than just the Control Plane:

  1. Permission Alignment: Ensure the host directory is pre-allocated with the correct ownership: chown -R 1000:1000 /mnt/data/starrocks/be/.
  2. Storage Verification: Exec into the running container and attempt a manual write: docker exec -it starrocks_be_1 touch /opt/starrocks/be/storage/test_file.
  3. Log Aggregation: Instead of checking show backend, they immediately inspect the be.out and be.log files inside the container to find IO errors or Permission Denied errors during the storage engine startup.
  4. Configuration Audit: Verify that the storage_root_path in be.conf matches the absolute path inside the container, not the host path.

Why Juniors Miss It

  • Focusing on Connectivity: Juniors often assume that if show backend shows the node as Alive, the problem must be with the SQL query or the FE.
  • Ignoring the Filesystem: They tend to treat Docker volumes as “magic buckets” that just work, neglecting the underlying Linux UID/GID permission model.
  • Surface-Level Debugging: They rely on the High-Level API (the SQL command) to diagnose a Low-Level System Error (filesystem mounting), rather than diving into the process logs.

Leave a Comment