Inline Assembly Bug in gcc?

by

Summary

This incident analyzes a subtle inline assembly constraint bug that appears only under GCC with high optimization levels. The program prints garbage instead of "Hello, World!\n" when compiled with -O3, even though it works correctly with -O0 and under Clang. The failure stems from incorrect assumptions about how GCC allocates registers for inline assembly operands, causing the compiler to reuse registers in ways the developer did not expect.

Root Cause

The root cause is the use of the "r" constraint for the msg pointer combined with a clobber list that does not fully describe all registers the assembly actually depends on.

Key issues:

  • GCC is allowed to place "r" operands in any general-purpose register, including those you later overwrite.
  • The inline assembly block clobbers registers that GCC believes are still valid, because the constraints do not tell GCC that the assembly depends on the original value of msg.
  • Under -O3, GCC aggressively reuses registers and performs register allocation differently, causing %rsi to receive a corrupted or unintended value.

In short: the inline assembly constraints are incomplete, so GCC is free to optimize in ways that break the code.

Why This Happens in Real Systems

Real compilers aggressively optimize, and inline assembly is treated as a black box. If constraints are wrong, the compiler will make assumptions that break your code.

Common reasons:

  • Incorrect operand constraints allow the compiler to choose registers that conflict with the assembly.
  • Missing clobbers cause the compiler to assume registers remain unchanged.
  • Optimizers reorder or eliminate instructions when the assembly block does not declare its side effects.
  • Inline assembly is not a stable ABI boundary unless fully constrained.

Real-World Impact

Incorrect inline assembly constraints can lead to:

  • Silent data corruption
  • Incorrect system calls
  • Crashes due to invalid pointers
  • Security vulnerabilities (e.g., leaking or overwriting memory)
  • Heisenbugs that appear only under certain optimization levels or compilers

Example or Code (if necessary and relevant)

A corrected version uses "r"(msg) but forces it into a register that will not be overwritten, or better, uses "m" to reference memory directly:

__asm__ volatile(
    "mov $1, %%rax\n"
    "mov $1, %%rdi\n"
    "mov %0, %%rsi\n"
    "mov $14, %%rdx\n"
    "syscall\n"
    :
    : "r"(msg)
    : "rax", "rdi", "rsi", "rdx"
);

Or safer:

__asm__ volatile(
    "mov $1, %%rax\n"
    "mov $1, %%rdi\n"
    "lea %0, %%rsi\n"
    "mov $14, %%rdx\n"
    "syscall\n"
    :
    : "m"(msg)
    : "rax", "rdi", "rsi", "rdx"
);

How Senior Engineers Fix It

Experienced engineers avoid these pitfalls by:

  • Using correct operand constraints ("m", "r", "g", "i", etc.)
  • Declaring all clobbered registers
  • Avoiding inline assembly unless absolutely necessary
  • Using extended asm with explicit input/output semantics
  • Letting the compiler handle calling conventions and register allocation

They also:

  • Validate assembly with multiple compilers and optimization levels
  • Use tools like objdump and Compiler Explorer to inspect generated code

Why Juniors Miss It

Juniors often:

  • Assume inline assembly behaves like standalone assembly
  • Believe "r" means “put it in a safe register”
  • Forget that the compiler is free to reuse registers aggressively
  • Do not understand how operand constraints drive register allocation
  • Do not realize that missing clobbers cause undefined behavior
  • Test only with -O0, where register allocation is minimal

Inline assembly is a contract with the compiler. If the contract is incomplete, the compiler is allowed to break your assumptions.

Leave a Comment