Service Mesh Observability

1---
2name: service-mesh-observability
3description: Implement comprehensive observability for service meshes including distributed tracing, metrics, and visualization. Use when setting up mesh monitoring, debugging latency issues, or implementing SLOs for service communication.
4---
5 
6# Service Mesh Observability
7 
8Complete guide to observability patterns for Istio, Linkerd, and service mesh deployments.
9 
10## Do not use this skill when
11 
12- The task is unrelated to service mesh observability
13- You need a different domain or tool outside this scope
14 
15## Instructions
16 
17- Clarify goals, constraints, and required inputs.
18- Apply relevant best practices and validate outcomes.
19- Provide actionable steps and verification.
20- If detailed examples are required, open `resources/implementation-playbook.md`.
21 
22## Use this skill when
23 
24- Setting up distributed tracing across services
25- Implementing service mesh metrics and dashboards
26- Debugging latency and error issues
27- Defining SLOs for service communication
28- Visualizing service dependencies
29- Troubleshooting mesh connectivity
30 
31## Core Concepts
32 
33### 1. Three Pillars of Observability
34 
35```
36┌─────────────────────────────────────────────────────┐
37│                  Observability                       │
38├─────────────────┬─────────────────┬─────────────────┤
39│     Metrics     │     Traces      │      Logs       │
40│                 │                 │                 │
41│ • Request rate  │ • Span context  │ • Access logs   │
42│ • Error rate    │ • Latency       │ • Error details │
43│ • Latency P50   │ • Dependencies  │ • Debug info    │
44│ • Saturation    │ • Bottlenecks   │ • Audit trail   │
45└─────────────────┴─────────────────┴─────────────────┘
46```
47 
48### 2. Golden Signals for Mesh
49 
50| Signal | Description | Alert Threshold |
51|--------|-------------|-----------------|
52| **Latency** | Request duration P50, P99 | P99 > 500ms |
53| **Traffic** | Requests per second | Anomaly detection |
54| **Errors** | 5xx error rate | > 1% |
55| **Saturation** | Resource utilization | > 80% |
56 
57## Templates
58 
59### Template 1: Istio with Prometheus & Grafana
60 
61```yaml
62# Install Prometheus
63apiVersion: v1
64kind: ConfigMap
65metadata:
66  name: prometheus
67  namespace: istio-system
68data:
69  prometheus.yml: |
70    global:
71      scrape_interval: 15s
72    scrape_configs:
73      - job_name: 'istio-mesh'
74        kubernetes_sd_configs:
75          - role: endpoints
76            namespaces:
77              names:
78                - istio-system
79        relabel_configs:
80          - source_labels: [__meta_kubernetes_service_name]
81            action: keep
82            regex: istio-telemetry
83---
84# ServiceMonitor for Prometheus Operator
85apiVersion: monitoring.coreos.com/v1
86kind: ServiceMonitor
87metadata:
88  name: istio-mesh
89  namespace: istio-system
90spec:
91  selector:
92    matchLabels:
93      app: istiod
94  endpoints:
95    - port: http-monitoring
96      interval: 15s
97```
98 
99### Template 2: Key Istio Metrics Queries
100 
101```promql
102# Request rate by service
103sum(rate(istio_requests_total{reporter="destination"}[5m])) by (destination_service_name)
104 
105# Error rate (5xx)
106sum(rate(istio_requests_total{reporter="destination", response_code=~"5.."}[5m]))
107  / sum(rate(istio_requests_total{reporter="destination"}[5m])) * 100
108 
109# P99 latency
110histogram_quantile(0.99,
111  sum(rate(istio_request_duration_milliseconds_bucket{reporter="destination"}[5m]))
112  by (le, destination_service_name))
113 
114# TCP connections
115sum(istio_tcp_connections_opened_total{reporter="destination"}) by (destination_service_name)
116 
117# Request size
118histogram_quantile(0.99,
119  sum(rate(istio_request_bytes_bucket{reporter="destination"}[5m]))
120  by (le, destination_service_name))
121```
122 
123### Template 3: Jaeger Distributed Tracing
124 
125```yaml
126# Jaeger installation for Istio
127apiVersion: install.istio.io/v1alpha1
128kind: IstioOperator
129spec:
130  meshConfig:
131    enableTracing: true
132    defaultConfig:
133      tracing:
134        sampling: 100.0  # 100% in dev, lower in prod
135        zipkin:
136          address: jaeger-collector.istio-system:9411
137---
138# Jaeger deployment
139apiVersion: apps/v1
140kind: Deployment
141metadata:
142  name: jaeger
143  namespace: istio-system
144spec:
145  selector:
146    matchLabels:
147      app: jaeger
148  template:
149    metadata:
150      labels:
151        app: jaeger
152    spec:
153      containers:
154        - name: jaeger
155          image: jaegertracing/all-in-one:1.50
156          ports:
157            - containerPort: 5775   # UDP
158            - containerPort: 6831   # Thrift
159            - containerPort: 6832   # Thrift
160            - containerPort: 5778   # Config
161            - containerPort: 16686  # UI
162            - containerPort: 14268  # HTTP
163            - containerPort: 14250  # gRPC
164            - containerPort: 9411   # Zipkin
165          env:
166            - name: COLLECTOR_ZIPKIN_HOST_PORT
167              value: ":9411"
168```
169 
170### Template 4: Linkerd Viz Dashboard
171 
172```bash
173# Install Linkerd viz extension
174linkerd viz install | kubectl apply -f -
175 
176# Access dashboard
177linkerd viz dashboard
178 
179# CLI commands for observability
180# Top requests
181linkerd viz top deploy/my-app
182 
183# Per-route metrics
184linkerd viz routes deploy/my-app --to deploy/backend
185 
186# Live traffic inspection
187linkerd viz tap deploy/my-app --to deploy/backend
188 
189# Service edges (dependencies)
190linkerd viz edges deployment -n my-namespace
191```
192 
193### Template 5: Grafana Dashboard JSON
194 
195```json
196{
197  "dashboard": {
198    "title": "Service Mesh Overview",
199    "panels": [
200      {
201        "title": "Request Rate",
202        "type": "graph",
203        "targets": [
204          {
205            "expr": "sum(rate(istio_requests_total{reporter=\"destination\"}[5m])) by (destination_service_name)",
206            "legendFormat": "{{destination_service_name}}"
207          }
208        ]
209      },
210      {
211        "title": "Error Rate",
212        "type": "gauge",
213        "targets": [
214          {
215            "expr": "sum(rate(istio_requests_total{response_code=~\"5..\"}[5m])) / sum(rate(istio_requests_total[5m])) * 100"
216          }
217        ],
218        "fieldConfig": {
219          "defaults": {
220            "thresholds": {
221              "steps": [
222                {"value": 0, "color": "green"},
223                {"value": 1, "color": "yellow"},
224                {"value": 5, "color": "red"}
225              ]
226            }
227          }
228        }
229      },
230      {
231        "title": "P99 Latency",
232        "type": "graph",
233        "targets": [
234          {
235            "expr": "histogram_quantile(0.99, sum(rate(istio_request_duration_milliseconds_bucket{reporter=\"destination\"}[5m])) by (le, destination_service_name))",
236            "legendFormat": "{{destination_service_name}}"
237          }
238        ]
239      },
240      {
241        "title": "Service Topology",
242        "type": "nodeGraph",
243        "targets": [
244          {
245            "expr": "sum(rate(istio_requests_total{reporter=\"destination\"}[5m])) by (source_workload, destination_service_name)"
246          }
247        ]
248      }
249    ]
250  }
251}
252```
253 
254### Template 6: Kiali Service Mesh Visualization
255 
256```yaml
257# Kiali installation
258apiVersion: kiali.io/v1alpha1
259kind: Kiali
260metadata:
261  name: kiali
262  namespace: istio-system
263spec:
264  auth:
265    strategy: anonymous  # or openid, token
266  deployment:
267    accessible_namespaces:
268      - "**"
269  external_services:
270    prometheus:
271      url: http://prometheus.istio-system:9090
272    tracing:
273      url: http://jaeger-query.istio-system:16686
274    grafana:
275      url: http://grafana.istio-system:3000
276```
277 
278### Template 7: OpenTelemetry Integration
279 
280```yaml
281# OpenTelemetry Collector for mesh
282apiVersion: v1
283kind: ConfigMap
284metadata:
285  name: otel-collector-config
286data:
287  config.yaml: |
288    receivers:
289      otlp:
290        protocols:
291          grpc:
292            endpoint: 0.0.0.0:4317
293          http:
294            endpoint: 0.0.0.0:4318
295      zipkin:
296        endpoint: 0.0.0.0:9411
297 
298    processors:
299      batch:
300        timeout: 10s
301 
302    exporters:
303      jaeger:
304        endpoint: jaeger-collector:14250
305        tls:
306          insecure: true
307      prometheus:
308        endpoint: 0.0.0.0:8889
309 
310    service:
311      pipelines:
312        traces:
313          receivers: [otlp, zipkin]
314          processors: [batch]
315          exporters: [jaeger]
316        metrics:
317          receivers: [otlp]
318          processors: [batch]
319          exporters: [prometheus]
320---
321# Istio Telemetry v2 with OTel
322apiVersion: telemetry.istio.io/v1alpha1
323kind: Telemetry
324metadata:
325  name: mesh-default
326  namespace: istio-system
327spec:
328  tracing:
329    - providers:
330        - name: otel
331      randomSamplingPercentage: 10
332```
333 
334## Alerting Rules
335 
336```yaml
337apiVersion: monitoring.coreos.com/v1
338kind: PrometheusRule
339metadata:
340  name: mesh-alerts
341  namespace: istio-system
342spec:
343  groups:
344    - name: mesh.rules
345      rules:
346        - alert: HighErrorRate
347          expr: |
348            sum(rate(istio_requests_total{response_code=~"5.."}[5m])) by (destination_service_name)
349            / sum(rate(istio_requests_total[5m])) by (destination_service_name) > 0.05
350          for: 5m
351          labels:
352            severity: critical
353          annotations:
354            summary: "High error rate for {{ $labels.destination_service_name }}"
355 
356        - alert: HighLatency
357          expr: |
358            histogram_quantile(0.99, sum(rate(istio_request_duration_milliseconds_bucket[5m]))
359            by (le, destination_service_name)) > 1000
360          for: 5m
361          labels:
362            severity: warning
363          annotations:
364            summary: "High P99 latency for {{ $labels.destination_service_name }}"
365 
366        - alert: MeshCertExpiring
367          expr: |
368            (certmanager_certificate_expiration_timestamp_seconds - time()) / 86400 < 7
369          labels:
370            severity: warning
371          annotations:
372            summary: "Mesh certificate expiring in less than 7 days"
373```
374 
375## Best Practices
376 
377### Do's
378- **Sample appropriately** - 100% in dev, 1-10% in prod
379- **Use trace context** - Propagate headers consistently
380- **Set up alerts** - For golden signals
381- **Correlate metrics/traces** - Use exemplars
382- **Retain strategically** - Hot/cold storage tiers
383 
384### Don'ts
385- **Don't over-sample** - Storage costs add up
386- **Don't ignore cardinality** - Limit label values
387- **Don't skip dashboards** - Visualize dependencies
388- **Don't forget costs** - Monitor observability costs
389 
390## Resources
391 
392- [Istio Observability](https://istio.io/latest/docs/tasks/observability/)
393- [Linkerd Observability](https://linkerd.io/2.14/features/dashboard/)
394- [OpenTelemetry](https://opentelemetry.io/)
395- [Kiali](https://kiali.io/)
396