In a monolithic application, when a request is slow, you can fire up a profiler and find the bottleneck. In a microservices architecture, it’s not so simple. A single user request might travel through five, ten, or even more services before a response is returned.

If that request is slow, where is the problem? Is it Service A’s database query? Is it the network call from Service B to Service C? Is Service D waiting on an external API?

This is the problem that distributed tracing solves. It allows you to visualize the entire lifecycle of a request as it flows through your system, giving you a detailed breakdown of how much time was spent in each service.

The industry standard for implementing tracing is OpenTelemetry (OTel), a vendor-neutral, open-source observability framework.

The Core Concepts of Distributed Tracing

#

1. Trace: Represents the entire journey of a request. A trace is a collection of spans.
2. Span: Represents a single unit of work within a trace, like an HTTP call, a database query, or a specific method execution. Spans have a start time, a duration, and can be nested.
3. Trace Context: A set of unique identifiers (TraceId, SpanId) that are passed between services with each request (usually as HTTP headers). This context is what allows the system to stitch the individual spans together into a single, coherent trace.

Implementing OpenTelemetry in .NET

#

Let’s imagine we have two services: an OrderService (an ASP.NET Core Web API) that receives the initial request, and a ProductService that it calls to get product details.

Step 1: Add the NuGet Packages

#

You’ll need a few packages in both of your service projects.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# Core OTel packages
dotnet add package OpenTelemetry
dotnet add package OpenTelemetry.Extensions.Hosting

# Instrumentation for ASP.NET Core and HTTP calls
dotnet add package OpenTelemetry.Instrumentation.AspNetCore
dotnet add package OpenTelemetry.Instrumentation.Http

# An exporter to send the data somewhere (we'll use the console for this example)
dotnet add package OpenTelemetry.Exporter.Console

Other exporters are available for systems like Jaeger, Zipkin, or Application Insights.

Step 2: Configure OpenTelemetry in Program.cs

#

In both OrderService and ProductService, you need to configure OpenTelemetry.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

using OpenTelemetry.Trace;
using OpenTelemetry.Resources;

var builder = WebApplication.CreateBuilder(args);
var serviceName = "MyWebApp.OrderService"; // CHANGE THIS FOR EACH SERVICE

builder.Services.AddOpenTelemetry()
    .ConfigureResource(resource => resource.AddService(serviceName: serviceName))
    .WithTracing(tracing => tracing
        .AddAspNetCoreInstrumentation() // Automatically traces incoming ASP.NET Core requests
        .AddHttpClientInstrumentation() // Automatically traces outgoing HttpClient requests
        .AddSource(serviceName) // <--- Add this line to capture custom spans!
        .AddConsoleExporter()); // Sends traces to the console

// You also need to add an HttpClient for the service to call others
builder.Services.AddHttpClient();

// IMPORTANT: Register the ActivitySource so OTel knows to listen to it
builder.Services.AddSingleton(new ActivitySource(serviceName));

var app = builder.Build();

// ... your app's endpoints
app.Run();

That’s the basic setup. With just this code, OpenTelemetry will automatically:

• Start a new trace and a root span for every incoming request to OrderService.
• When OrderService makes an HttpClient call to ProductService, OTel will automatically inject the trace context headers (traceparent, tracestate) into that outgoing request.
• When ProductService receives the request, it will see the trace context headers and continue the same trace, creating a new child span.

Step 3: The Service Code

#

Let’s look at the code for the two services.

OrderService’s Program.cs Endpoints:

1
2
3
4
5
6
7
8
9

app.MapGet("/order/{id}", async (int id, IHttpClientFactory clientFactory) =>
{
    var httpClient = clientFactory.CreateClient();

    // OTel's HttpClientInstrumentation will automatically add trace headers here
    var productInfo = await httpClient.GetStringAsync($"http://localhost:5001/product/123"); // Assuming ProductService runs on port 5001

    return $"Order {id} contains: {productInfo}";
});

ProductService’s Program.cs Endpoints:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

// OTel configuration is the same, just with a different service name
// ...

app.MapGet("/product/{id}", async (int id) =>
{
    // OTel's AspNetCoreInstrumentation has already started a child span
    // for this incoming request.
    await Task.Delay(150); // Simulate some work
    return $"Product {id} - 'Super Widget'";
});

Step 4: Run and Observe

#

When you run both services and make a request to http://localhost:5000/order/1, you’ll see output in the console of both applications from the ConsoleExporter.

OrderService Console Output:

1
2
3
4
5
6
7

Activity.TraceId:            a1b2c3d4...
Activity.SpanId:             e5f6g7h8...
Activity.DisplayName:        /order/{id}
Activity.StartTimeUtc:       ...
Activity.Duration:           00:00:00.200
Resource associated with Activity:
    service.name: MyWebApp.OrderService

This is the root span.

ProductService Console Output:

1
2
3
4
5
6
7
8

Activity.TraceId:            a1b2c3d4...  <-- SAME TRACE ID
Activity.ParentSpanId:       e5f6g7h8...  <-- Parent is the OrderService span
Activity.SpanId:             i9j0k1l2...
Activity.DisplayName:        /product/{id}
Activity.StartTimeUtc:       ...
Activity.Duration:           00:00:00.150
Resource associated with Activity:
    service.name: MyWebApp.ProductService

This is the child span. A backend like Jaeger or Zipkin would visualize this as a waterfall diagram, showing that the OrderService span took 200ms, and 150ms of that was spent waiting for the ProductService span to complete.

Adding Custom Spans

#

Automatic instrumentation is great, but sometimes you want to trace a specific piece of work inside a method. You can create custom spans for this.

First, you need to define an ActivitySource. A common pattern is to hold this in a static field or inject it.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

using System.Diagnostics;

// At the top of your class or in a shared location
public static readonly ActivitySource MyActivitySource = new ActivitySource("MyWebApp.OrderService");

// Inside your endpoint
app.MapGet("/order/{id}", async (int id, IHttpClientFactory clientFactory) =>
{
    // Create a custom span
    using (var activity = MyActivitySource.StartActivity("CalculatingTax"))
    {
        // Add custom attributes (called "tags") to your span
        activity?.SetTag("order.id", id);
        await Task.Delay(50); // Simulate tax calculation
    }
    // ... rest of the code

Crucial Step: For these custom spans to appear in your trace, you must tell the OpenTelemetry builder to listen to this specific ActivitySource. I’ve updated the configuration code above to include .AddSource(serviceName). Without that line, StartActivity returns null!

Conclusion

#

Distributed tracing is no longer a “nice-to-have”—it’s a necessity for understanding and debugging modern, distributed systems. OpenTelemetry provides a powerful, standardized, and easy-to-use framework for adding this capability to your .NET applications. With just a few lines of configuration, you can gain deep insights into the performance of your entire system, making it faster and more reliable.

Further Reading

#

• OpenTelemetry .NET Documentation - Official OpenTelemetry documentation
• .NET Observability with OpenTelemetry - Microsoft’s guide
• Jaeger Documentation - Learn about the Jaeger tracing backend
• OpenTelemetry on GitHub - The .NET SDK repository