Serilog Tracing - Azure Monitor with Traces and mismatched spans

Summary

The issue at hand involves mismatched spans in Azure Monitor when using SerilogTracing in an ASP.NET Core 8 Web API. This results in inconsistent tracing and incorrect filtering of activities in the App Insights operation view. The root cause of this issue lies in the way activities are started and managed within the application.

Root Cause

The root cause of this issue can be attributed to the following factors:

Incorrect activity nesting: Activities are not properly nested, leading to mismatched spans.
Insufficient context: Activities lack sufficient context, making it difficult to filter and associate them correctly.
Inconsistent logging configuration: The logging configuration may not be consistent across the application, leading to inconsistencies in activity tracing.

Why This Happens in Real Systems

This issue can occur in real systems due to:

Complexity of distributed systems: Distributed systems can be complex, making it challenging to manage activities and spans correctly.
Inadequate logging configuration: Inadequate logging configuration can lead to inconsistencies in activity tracing.
Lack of standardization: Lack of standardization in logging and tracing practices can contribute to this issue.

Real-World Impact

The real-world impact of this issue includes:

Inconsistent monitoring and debugging: Inconsistent tracing and logging can make it challenging to monitor and debug applications.
Difficulty in identifying performance bottlenecks: Incorrect activity tracing can make it difficult to identify performance bottlenecks.
Inaccurate analytics and reporting: Inconsistent logging and tracing can lead to inaccurate analytics and reporting.

Example or Code (if necessary and relevant)

using Serilog;
using Serilog.Context;

public class RateController
{
    private readonly ILogger _logger;
    private readonly Service _service;

    public RateController(ILogger seriLogger, IService service)
    {
        _logger = seriLogger.ForContext();
        _service = service;
    }

    [HttpPost]
    public async Task Submit(Message request)
    {
        using (LogContext.PushProperty("Request", request))
        {
            using LoggerActivity activity = _logger.StartActivity("Rating {@number} from message {@MessageID}", request.Number, request.MessageID);
            _service.MethodCallThatAlsoCallsStartActivity();
        }
    }
}

How Senior Engineers Fix It

Senior engineers can fix this issue by:

Standardizing logging and tracing practices: Ensuring consistent logging and tracing practices across the application.
Implementing correct activity nesting: Ensuring that activities are properly nested to maintain correct spans.
Providing sufficient context: Ensuring that activities have sufficient context to facilitate correct filtering and association.
Configuring logging correctly: Configuring logging correctly to ensure consistent activity tracing.

Why Juniors Miss It

Junior engineers may miss this issue due to:

Lack of experience with distributed systems: Limited experience with distributed systems can make it challenging to understand the complexities of activity tracing.
Inadequate understanding of logging and tracing: Limited understanding of logging and tracing practices can lead to inconsistencies in activity tracing.
Insufficient attention to detail: Insufficient attention to detail can result in incorrect activity nesting and insufficient context.

Serilog Tracing – Azure Monitor with Traces and mismatched spans