Fixing Hybrid Rust/C++ Build System Artifacts Distribution Problem

by

Summary

During a routine build orchestration upgrade, we identified a critical failure in how the hybrid Rust/C++ build system handled artifact distribution. While the project compiled successfully, the cmake crate used in the Rust build script was overriding the standard CMake installation behavior. This resulted in the C++ shared library and its headers being trapped within the Cargo build directory instead of being installed to the system-wide /usr/local/lib and /usr/local/include paths. This prevents the resulting binary from being consumable by other system processes and breaks standard Linux distribution patterns.

Root Cause

The failure stems from a fundamental misunderstanding of how the cmake Rust crate interacts with the CMake CMAKE_INSTALL_PREFIX and the out_dir concept.

  • Override of Installation Prefix: The cmake crate in build.rs calls .out_dir(out_dir), which explicitly tells CMake to use a specific, local directory for all build artifacts, including installation targets.
  • Implicit Destination Redirection: When cmake::Config::build() is executed, the crate sets the CMAKE_INSTALL_PREFIX to the provided out_dir. Consequently, even if the cpp-lib/CMakeLists.txt uses GNUInstallDirs (like install(TARGETS ... DESTINATION ${CMAKE_INSTALL_LIBDIR})), the “install” destination is relative to the Cargo target directory, not the system root.
  • Build-time vs. Install-time: The build script is designed to facilitate link-time dependencies for the Rust compiler, but it inadvertently swallows the install-time logic required for system-level deployment.

Why This Happens in Real Systems

In complex polyglot monorepos, we often attempt to unify two different lifecycles:

  1. The Cargo Lifecycle: Optimized for hermetic, local builds where everything lives inside target/.
  2. The CMake/System Lifecycle: Optimized for global availability, following the Filesystem Hierarchy Standard (FHS).

When a build tool (like build.rs) invokes a secondary build system (CMake), it typically attempts to “sandbox” that secondary system to avoid polluting the developer’s machine. This sandboxing is achieved by redirecting the Installation Prefix. In a production environment, if the CI/CD pipeline expects a simple make install or cmake --install to distribute the package, it will fail because the artifacts are buried deep within the Cargo target folder, often in a non-obvious sub-directory.

Real-World Impact

  • Broken Packaging: Automated packaging tools (like rpmbuild or debbuild) fail to find the C++ headers and libraries, leading to empty or incomplete packages.
  • Deployment Failures: Production environments expecting a standard library installation will encounter libsup_lib.so not found errors because the library was never moved to /usr/local/lib.
  • Developer Friction: New engineers attempting to use the library in other projects via system paths will face “header not found” errors, despite the project “building fine” locally.

Example or Code

The error is located in the rust-bin/build.rs file. The out_dir is being set to the source directory of the C++ library, which forces all install() commands in CMake to land there.

// INCORRECT: This forces CMake to install everything into the source tree
let out_dir = format!("{manifest}/cpp-lib");
let dst = Config::new(cpp_lib_path)
    .pic(true)
    .out_dir(out_dir.clone()) // This is the culprit
    .build();

To fix the build-time linking while still allowing for a proper system installation, the CMakeLists.txt orchestration must be separated from the build.rs logic, or the installation step must be explicitly invoked via a separate command that targets the actual system prefix.

How Senior Engineers Fix It

Senior engineers decouple the Build-for-Link phase from the Build-for-Install phase.

  • Decouple Build Logic: Use build.rs only to provide the library for the current Rust compilation (linking via out_dir).

  • Explicit Installation Target: Instead of relying on cargo build to perform the installation, provide a top-level install target in the root CMakeLists.txt that calls the installation logic using the intended system prefix.

  • Standardized Prefixes: Use CMAKE_INSTALL_PREFIX explicitly in the CI/CD pipeline rather than letting the build script dictate the destination.

  • Implementation of a wrapper:

    # For development (Rust-centric)
cargo build

# For production deployment (System-centric)
mkdir build && cd build
cmake .. -DCMAKE_INSTALL_PREFIX=/usr/local
make install

Why Juniors Miss It

  • The “It Builds” Trap: Juniors often define “success” as “the compiler didn’t return an error.” If the Rust binary runs, they assume the build system is correct.
  • Mental Model Mismatch: They treat build.rs as a general-purpose setup script rather than a specialized tool meant only for link-time discovery.
  • Ignoring Installation Logic: They focus on add_library and target_link_libraries but overlook the install() commands, assuming that if a library exists in a folder, it is “installed.”
  • Lack of FHS Awareness: They may not be familiar with the importance of the Filesystem Hierarchy Standard and why libraries shouldn’t live in project source directories.

Leave a Comment