Strange stack trace from memory profile

by

Summary

A developer observed a mysterious stack trace while profiling a large C++ service with AddressSanitizer (ASan). The trace showed massive memory consumption (259 MB) attributed solely to malloc called from operator new, with no visible user code in the call stack. This indicates that standard stack trace introspection is insufficient for allocations triggered by dynamic libraries or lower-level runtime components (like the C++ standard library). The root cause is usually that the compiler optimized away the frame pointer or the allocation is happening inside a shared library (like libstdc++) that was compiled without frame pointers, masking the true origin.

Root Cause

The immediate cause of the “invisible” stack trace is the lack of frame pointers in the build artifacts of the system libraries or the application itself.

  • Optimization: Compilers (GCC/Clang) optimize register usage by omitting the frame pointer (rbp on x86_64) when the -fomit-frame-pointer flag is used (often default in Release builds).
  • ASan Limitation: When __sanitizer_print_memory_profile unwinds the stack, it relies on standard stack walking techniques. Without frame pointers, it must rely on DWARF debug info (.debug_frame or .eh_frame). If these sections are stripped or if the unwinder hits a boundary between the instrumented app and a system library, it stops.
  • The “Hidden” Code: The operator new call is coming from the C++ standard library (e.g., libstdc++). If the standard library was compiled without frame pointers (or without sufficient unwind info), the stack walker sees the transition to libstdc++, executes the allocation, but cannot backtrack further to show who requested the allocation.

Why This Happens in Real Systems

  • Performance Trade-offs: System libraries and high-performance services are almost always compiled with -fomit-frame-pointer to save registers and reduce instruction cache pressure.
  • Mixed Builds: You are likely running a Release build of the application (which strips debug info) but using ASan (which requires debug info for useful traces).
  • Inline Aggressive Allocation: operator new is often inlined or devirtualized. If the compiler sees that no user-defined new handler is present, it may inline the call to malloc directly, making the “user code” frame disappear if the instruction pointer jumps into the middle of a library routine.

Real-World Impact

  • Blind Spots: Engineers cannot debug memory leaks or spikes because the root cause is obscured.
  • Wasted Engineering Time: Teams spend hours guessing which data structure is growing, rather than seeing the exact line of code.
  • False Positives: It looks like a “ghost” allocation—memory appearing from nowhere—which can lead to suspicion of memory corruption or compiler bugs.
  • Operational Risk: In production, if this 259 MB allocation grows to 2 GB, the OOM killer terminates the process, and post-mortem analysis yields no answers without the right build flags.

Example or Code

The stack trace provided in the report is:

259188664 bytes (38%) in 31679 allocations
#0 0x00000092b40b in malloc ../../.././libsanitizer/asan/asan_malloc_linux.cpp:67
#1 0x000002447f0b in operator new(unsigned long) ../../.././libstdc++-v3/libsupc++/new_op.cc:50

This is the “truncated” trace. A correct trace would look like:

#0 0x... in malloc
#1 0x... in operator new
#2 0x... in std::vector::emplace_back(...)
#3 0x... in MyClass::processData()
#4 0x... in main()

How Senior Engineers Fix It

  • Rebuild with Frame Pointers: The most reliable fix is to compile your application and the C++ standard library (if possible) with -fno-omit-frame-pointer. This ensures the linked list of stack frames exists in memory.
  • Enable Full Unwind Info: Ensure -gdwarf-4 or -gdwarf-5 is used and that debug symbols are not stripped from the binary (or are available in a separate file).
  • Use Allocation Hooks: Instead of relying on the profiler’s stack unwinder, implement a custom operator new or override malloc using LD_PRELOAD (hooking) to capture the stack trace at the moment of allocation using backtrace() or libunwind. This bypasses the ASan internal unwinder limitations.
  • Use External Profilers: Tools like Heaptrack or Massif (Valgrind) often handle stack unwinding differently and might capture the missing frames where ASan fails.

Why Juniors Miss It

  • “It Works on My Machine”: Junior developers often test on debug builds where frame pointers are preserved by default, so they never encounter the truncated stack traces seen in production-like release builds.
  • Blind Trust in Tools: They assume that if ASan prints a stack trace, it must be the complete and correct trace. They don’t realize that the tool is only as good as the binary metadata it reads.
  • Misunderstanding Inlining: They might not understand that std::vector or other containers are heavy users of operator new, and that the compiler might optimize the call chain to look like it’s just “standard library internals” rather than user code accessing the vector.
  • Lack of Build Flags Knowledge: They may not know the impact of -fomit-frame-pointer on debugging capabilities.