Coredump generated on crash (segfault, abort), but not when sending corresponding signal (kill -11)

by

Summary

The issue at hand is related to coredump generation on an old embedded Linux platform. When a program crashes naturally, a coredump is generated and sent to a remote server using netcat. However, when the program is manually terminated using kill -11, no coredump is generated. This behavior is unexpected, given that ulimit -c unlimited has been set to allow for unlimited coredump size.

Root Cause

The root cause of this issue lies in the way Linux kernel 2.6.32 handles signal delivery and coredump generation. Some possible causes include:

  • Signal handling: The program may have registered signal handlers that prevent coredump generation when the signal is delivered manually.
  • Kernel configuration: The kernel may be configured to ignore coredump generation for certain signals or under specific conditions.
  • Busybox limitations: The busybox implementation of ulimit and kill may have limitations or quirks that affect coredump generation.

Why This Happens in Real Systems

This issue can occur in real systems due to:

  • Inconsistent signal handling: Programs may handle signals differently, leading to inconsistent behavior when it comes to coredump generation.
  • Kernel version differences: Different kernel versions may handle coredump generation and signal delivery differently, leading to unexpected behavior.
  • Embedded system constraints: Embedded systems often have limited resources and customized configurations, which can lead to unique issues like this one.

Real-World Impact

The impact of this issue includes:

  • Difficulty in debugging: Without coredumps, debugging program crashes becomes much more challenging.
  • Inconsistent system behavior: The inconsistent behavior of coredump generation can lead to confusion and difficulties in reproducing and fixing issues.
  • Reliability concerns: The inability to generate coredumps in certain situations can raise concerns about the overall reliability of the system.

Example or Code

#include 
#include 
#include 

void sigsegv_handler(int sig) {
    // Signal handler for SIGSEGV
    printf("Received SIGSEGV\n");
    exit(1);
}

int main() {
    // Register signal handler for SIGSEGV
    signal(SIGSEGV, sigsegv_handler);
    // Simulate a segmentation fault
    int* ptr = NULL;
    *ptr = 1;
    return 0;
}

How Senior Engineers Fix It

Senior engineers can fix this issue by:

  • Checking kernel configuration: Verifying that the kernel is configured to generate coredumps for the relevant signals.
  • Reviewing signal handling: Ensuring that signal handlers are not interfering with coredump generation.
  • Testing with different signals: Trying different signals to see if the issue is specific to SIGSEGV or occurs with other signals as well.
  • Upgrading to a newer kernel: If possible, upgrading to a newer kernel version that may have improved coredump generation and signal handling.

Why Juniors Miss It

Junior engineers may miss this issue due to:

  • Lack of experience with embedded systems: Inexperience with the unique constraints and configurations of embedded systems.
  • Insufficient knowledge of signal handling: Limited understanding of how signal handling works and how it can affect coredump generation.
  • Overlooking kernel version differences: Failing to consider the potential differences in behavior between different kernel versions.
  • Inadequate testing: Not thoroughly testing the system with different signals and scenarios to identify the issue.

Leave a Comment