Distributed Tracing

1---
2name: distributed-tracing
3description: Implement distributed tracing with Jaeger and Tempo to track requests across microservices and identify performance bottlenecks. Use when debugging microservices, analyzing request flows, or implementing observability for distributed systems.
4---
5 
6# Distributed Tracing
7 
8Implement distributed tracing with Jaeger and Tempo for request flow visibility across microservices.
9 
10## Do not use this skill when
11 
12- The task is unrelated to distributed tracing
13- You need a different domain or tool outside this scope
14 
15## Instructions
16 
17- Clarify goals, constraints, and required inputs.
18- Apply relevant best practices and validate outcomes.
19- Provide actionable steps and verification.
20- If detailed examples are required, open `resources/implementation-playbook.md`.
21 
22## Purpose
23 
24Track requests across distributed systems to understand latency, dependencies, and failure points.
25 
26## Use this skill when
27 
28- Debug latency issues
29- Understand service dependencies
30- Identify bottlenecks
31- Trace error propagation
32- Analyze request paths
33 
34## Distributed Tracing Concepts
35 
36### Trace Structure
37```
38Trace (Request ID: abc123)
39  ↓
40Span (frontend) [100ms]
41  ↓
42Span (api-gateway) [80ms]
43  ├→ Span (auth-service) [10ms]
44  └→ Span (user-service) [60ms]
45      └→ Span (database) [40ms]
46```
47 
48### Key Components
49- **Trace** - End-to-end request journey
50- **Span** - Single operation within a trace
51- **Context** - Metadata propagated between services
52- **Tags** - Key-value pairs for filtering
53- **Logs** - Timestamped events within a span
54 
55## Jaeger Setup
56 
57### Kubernetes Deployment
58 
59```bash
60# Deploy Jaeger Operator
61kubectl create namespace observability
62kubectl create -f https://github.com/jaegertracing/jaeger-operator/releases/download/v1.51.0/jaeger-operator.yaml -n observability
63 
64# Deploy Jaeger instance
65kubectl apply -f - <<EOF
66apiVersion: jaegertracing.io/v1
67kind: Jaeger
68metadata:
69  name: jaeger
70  namespace: observability
71spec:
72  strategy: production
73  storage:
74    type: elasticsearch
75    options:
76      es:
77        server-urls: http://elasticsearch:9200
78  ingress:
79    enabled: true
80EOF
81```
82 
83### Docker Compose
84 
85```yaml
86version: '3.8'
87services:
88  jaeger:
89    image: jaegertracing/all-in-one:latest
90    ports:
91      - "5775:5775/udp"
92      - "6831:6831/udp"
93      - "6832:6832/udp"
94      - "5778:5778"
95      - "16686:16686"  # UI
96      - "14268:14268"  # Collector
97      - "14250:14250"  # gRPC
98      - "9411:9411"    # Zipkin
99    environment:
100      - COLLECTOR_ZIPKIN_HOST_PORT=:9411
101```
102 
103**Reference:** See `references/jaeger-setup.md`
104 
105## Application Instrumentation
106 
107### OpenTelemetry (Recommended)
108 
109#### Python (Flask)
110```python
111from opentelemetry import trace
112from opentelemetry.exporter.jaeger.thrift import JaegerExporter
113from opentelemetry.sdk.resources import SERVICE_NAME, Resource
114from opentelemetry.sdk.trace import TracerProvider
115from opentelemetry.sdk.trace.export import BatchSpanProcessor
116from opentelemetry.instrumentation.flask import FlaskInstrumentor
117from flask import Flask
118 
119# Initialize tracer
120resource = Resource(attributes={SERVICE_NAME: "my-service"})
121provider = TracerProvider(resource=resource)
122processor = BatchSpanProcessor(JaegerExporter(
123    agent_host_name="jaeger",
124    agent_port=6831,
125))
126provider.add_span_processor(processor)
127trace.set_tracer_provider(provider)
128 
129# Instrument Flask
130app = Flask(__name__)
131FlaskInstrumentor().instrument_app(app)
132 
133@app.route('/api/users')
134def get_users():
135    tracer = trace.get_tracer(__name__)
136 
137    with tracer.start_as_current_span("get_users") as span:
138        span.set_attribute("user.count", 100)
139        # Business logic
140        users = fetch_users_from_db()
141        return {"users": users}
142 
143def fetch_users_from_db():
144    tracer = trace.get_tracer(__name__)
145 
146    with tracer.start_as_current_span("database_query") as span:
147        span.set_attribute("db.system", "postgresql")
148        span.set_attribute("db.statement", "SELECT * FROM users")
149        # Database query
150        return query_database()
151```
152 
153#### Node.js (Express)
154```javascript
155const { NodeTracerProvider } = require('@opentelemetry/sdk-trace-node');
156const { JaegerExporter } = require('@opentelemetry/exporter-jaeger');
157const { BatchSpanProcessor } = require('@opentelemetry/sdk-trace-base');
158const { registerInstrumentations } = require('@opentelemetry/instrumentation');
159const { HttpInstrumentation } = require('@opentelemetry/instrumentation-http');
160const { ExpressInstrumentation } = require('@opentelemetry/instrumentation-express');
161 
162// Initialize tracer
163const provider = new NodeTracerProvider({
164  resource: { attributes: { 'service.name': 'my-service' } }
165});
166 
167const exporter = new JaegerExporter({
168  endpoint: 'http://jaeger:14268/api/traces'
169});
170 
171provider.addSpanProcessor(new BatchSpanProcessor(exporter));
172provider.register();
173 
174// Instrument libraries
175registerInstrumentations({
176  instrumentations: [
177    new HttpInstrumentation(),
178    new ExpressInstrumentation(),
179  ],
180});
181 
182const express = require('express');
183const app = express();
184 
185app.get('/api/users', async (req, res) => {
186  const tracer = trace.getTracer('my-service');
187  const span = tracer.startSpan('get_users');
188 
189  try {
190    const users = await fetchUsers();
191    span.setAttributes({ 'user.count': users.length });
192    res.json({ users });
193  } finally {
194    span.end();
195  }
196});
197```
198 
199#### Go
200```go
201package main
202 
203import (
204    "context"
205    "go.opentelemetry.io/otel"
206    "go.opentelemetry.io/otel/exporters/jaeger"
207    "go.opentelemetry.io/otel/sdk/resource"
208    sdktrace "go.opentelemetry.io/otel/sdk/trace"
209    semconv "go.opentelemetry.io/otel/semconv/v1.4.0"
210)
211 
212func initTracer() (*sdktrace.TracerProvider, error) {
213    exporter, err := jaeger.New(jaeger.WithCollectorEndpoint(
214        jaeger.WithEndpoint("http://jaeger:14268/api/traces"),
215    ))
216    if err != nil {
217        return nil, err
218    }
219 
220    tp := sdktrace.NewTracerProvider(
221        sdktrace.WithBatcher(exporter),
222        sdktrace.WithResource(resource.NewWithAttributes(
223            semconv.SchemaURL,
224            semconv.ServiceNameKey.String("my-service"),
225        )),
226    )
227 
228    otel.SetTracerProvider(tp)
229    return tp, nil
230}
231 
232func getUsers(ctx context.Context) ([]User, error) {
233    tracer := otel.Tracer("my-service")
234    ctx, span := tracer.Start(ctx, "get_users")
235    defer span.End()
236 
237    span.SetAttributes(attribute.String("user.filter", "active"))
238 
239    users, err := fetchUsersFromDB(ctx)
240    if err != nil {
241        span.RecordError(err)
242        return nil, err
243    }
244 
245    span.SetAttributes(attribute.Int("user.count", len(users)))
246    return users, nil
247}
248```
249 
250**Reference:** See `references/instrumentation.md`
251 
252## Context Propagation
253 
254### HTTP Headers
255```
256traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
257tracestate: congo=t61rcWkgMzE
258```
259 
260### Propagation in HTTP Requests
261 
262#### Python
263```python
264from opentelemetry.propagate import inject
265 
266headers = {}
267inject(headers)  # Injects trace context
268 
269response = requests.get('http://downstream-service/api', headers=headers)
270```
271 
272#### Node.js
273```javascript
274const { propagation } = require('@opentelemetry/api');
275 
276const headers = {};
277propagation.inject(context.active(), headers);
278 
279axios.get('http://downstream-service/api', { headers });
280```
281 
282## Tempo Setup (Grafana)
283 
284### Kubernetes Deployment
285 
286```yaml
287apiVersion: v1
288kind: ConfigMap
289metadata:
290  name: tempo-config
291data:
292  tempo.yaml: |
293    server:
294      http_listen_port: 3200
295 
296    distributor:
297      receivers:
298        jaeger:
299          protocols:
300            thrift_http:
301            grpc:
302        otlp:
303          protocols:
304            http:
305            grpc:
306 
307    storage:
308      trace:
309        backend: s3
310        s3:
311          bucket: tempo-traces
312          endpoint: s3.amazonaws.com
313 
314    querier:
315      frontend_worker:
316        frontend_address: tempo-query-frontend:9095
317---
318apiVersion: apps/v1
319kind: Deployment
320metadata:
321  name: tempo
322spec:
323  replicas: 1
324  template:
325    spec:
326      containers:
327      - name: tempo
328        image: grafana/tempo:latest
329        args:
330          - -config.file=/etc/tempo/tempo.yaml
331        volumeMounts:
332        - name: config
333          mountPath: /etc/tempo
334      volumes:
335      - name: config
336        configMap:
337          name: tempo-config
338```
339 
340**Reference:** See `assets/jaeger-config.yaml.template`
341 
342## Sampling Strategies
343 
344### Probabilistic Sampling
345```yaml
346# Sample 1% of traces
347sampler:
348  type: probabilistic
349  param: 0.01
350```
351 
352### Rate Limiting Sampling
353```yaml
354# Sample max 100 traces per second
355sampler:
356  type: ratelimiting
357  param: 100
358```
359 
360### Adaptive Sampling
361```python
362from opentelemetry.sdk.trace.sampling import ParentBased, TraceIdRatioBased
363 
364# Sample based on trace ID (deterministic)
365sampler = ParentBased(root=TraceIdRatioBased(0.01))
366```
367 
368## Trace Analysis
369 
370### Finding Slow Requests
371 
372**Jaeger Query:**
373```
374service=my-service
375duration > 1s
376```
377 
378### Finding Errors
379 
380**Jaeger Query:**
381```
382service=my-service
383error=true
384tags.http.status_code >= 500
385```
386 
387### Service Dependency Graph
388 
389Jaeger automatically generates service dependency graphs showing:
390- Service relationships
391- Request rates
392- Error rates
393- Average latencies
394 
395## Best Practices
396 
3971. **Sample appropriately** (1-10% in production)
3982. **Add meaningful tags** (user_id, request_id)
3993. **Propagate context** across all service boundaries
4004. **Log exceptions** in spans
4015. **Use consistent naming** for operations
4026. **Monitor tracing overhead** (<1% CPU impact)
4037. **Set up alerts** for trace errors
4048. **Implement distributed context** (baggage)
4059. **Use span events** for important milestones
40610. **Document instrumentation** standards
407 
408## Integration with Logging
409 
410### Correlated Logs
411```python
412import logging
413from opentelemetry import trace
414 
415logger = logging.getLogger(__name__)
416 
417def process_request():
418    span = trace.get_current_span()
419    trace_id = span.get_span_context().trace_id
420 
421    logger.info(
422        "Processing request",
423        extra={"trace_id": format(trace_id, '032x')}
424    )
425```
426 
427## Troubleshooting
428 
429**No traces appearing:**
430- Check collector endpoint
431- Verify network connectivity
432- Check sampling configuration
433- Review application logs
434 
435**High latency overhead:**
436- Reduce sampling rate
437- Use batch span processor
438- Check exporter configuration
439 
440## Reference Files
441 
442- `references/jaeger-setup.md` - Jaeger installation
443- `references/instrumentation.md` - Instrumentation patterns
444- `assets/jaeger-config.yaml.template` - Jaeger configuration
445 
446## Related Skills
447 
448- `prometheus-configuration` - For metrics
449- `grafana-dashboards` - For visualization
450- `slo-implementation` - For latency SLOs
451