Grafana Dashboards

1---
2name: grafana-dashboards
3description: Create and manage production Grafana dashboards for real-time visualization of system and application metrics. Use when building monitoring dashboards, visualizing metrics, or creating operational observability interfaces.
4---
5 
6# Grafana Dashboards
7 
8Create and manage production-ready Grafana dashboards for comprehensive system observability.
9 
10## Do not use this skill when
11 
12- The task is unrelated to grafana dashboards
13- You need a different domain or tool outside this scope
14 
15## Instructions
16 
17- Clarify goals, constraints, and required inputs.
18- Apply relevant best practices and validate outcomes.
19- Provide actionable steps and verification.
20- If detailed examples are required, open `resources/implementation-playbook.md`.
21 
22## Purpose
23 
24Design effective Grafana dashboards for monitoring applications, infrastructure, and business metrics.
25 
26## Use this skill when
27 
28- Visualize Prometheus metrics
29- Create custom dashboards
30- Implement SLO dashboards
31- Monitor infrastructure
32- Track business KPIs
33 
34## Dashboard Design Principles
35 
36### 1. Hierarchy of Information
37```
38┌─────────────────────────────────────┐
39│  Critical Metrics (Big Numbers)     │
40├─────────────────────────────────────┤
41│  Key Trends (Time Series)           │
42├─────────────────────────────────────┤
43│  Detailed Metrics (Tables/Heatmaps) │
44└─────────────────────────────────────┘
45```
46 
47### 2. RED Method (Services)
48- **Rate** - Requests per second
49- **Errors** - Error rate
50- **Duration** - Latency/response time
51 
52### 3. USE Method (Resources)
53- **Utilization** - % time resource is busy
54- **Saturation** - Queue length/wait time
55- **Errors** - Error count
56 
57## Dashboard Structure
58 
59### API Monitoring Dashboard
60 
61```json
62{
63  "dashboard": {
64    "title": "API Monitoring",
65    "tags": ["api", "production"],
66    "timezone": "browser",
67    "refresh": "30s",
68    "panels": [
69      {
70        "title": "Request Rate",
71        "type": "graph",
72        "targets": [
73          {
74            "expr": "sum(rate(http_requests_total[5m])) by (service)",
75            "legendFormat": "{{service}}"
76          }
77        ],
78        "gridPos": {"x": 0, "y": 0, "w": 12, "h": 8}
79      },
80      {
81        "title": "Error Rate %",
82        "type": "graph",
83        "targets": [
84          {
85            "expr": "(sum(rate(http_requests_total{status=~\"5..\"}[5m])) / sum(rate(http_requests_total[5m]))) * 100",
86            "legendFormat": "Error Rate"
87          }
88        ],
89        "alert": {
90          "conditions": [
91            {
92              "evaluator": {"params": [5], "type": "gt"},
93              "operator": {"type": "and"},
94              "query": {"params": ["A", "5m", "now"]},
95              "type": "query"
96            }
97          ]
98        },
99        "gridPos": {"x": 12, "y": 0, "w": 12, "h": 8}
100      },
101      {
102        "title": "P95 Latency",
103        "type": "graph",
104        "targets": [
105          {
106            "expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))",
107            "legendFormat": "{{service}}"
108          }
109        ],
110        "gridPos": {"x": 0, "y": 8, "w": 24, "h": 8}
111      }
112    ]
113  }
114}
115```
116 
117**Reference:** See `assets/api-dashboard.json`
118 
119## Panel Types
120 
121### 1. Stat Panel (Single Value)
122```json
123{
124  "type": "stat",
125  "title": "Total Requests",
126  "targets": [{
127    "expr": "sum(http_requests_total)"
128  }],
129  "options": {
130    "reduceOptions": {
131      "values": false,
132      "calcs": ["lastNotNull"]
133    },
134    "orientation": "auto",
135    "textMode": "auto",
136    "colorMode": "value"
137  },
138  "fieldConfig": {
139    "defaults": {
140      "thresholds": {
141        "mode": "absolute",
142        "steps": [
143          {"value": 0, "color": "green"},
144          {"value": 80, "color": "yellow"},
145          {"value": 90, "color": "red"}
146        ]
147      }
148    }
149  }
150}
151```
152 
153### 2. Time Series Graph
154```json
155{
156  "type": "graph",
157  "title": "CPU Usage",
158  "targets": [{
159    "expr": "100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[5m])) * 100)"
160  }],
161  "yaxes": [
162    {"format": "percent", "max": 100, "min": 0},
163    {"format": "short"}
164  ]
165}
166```
167 
168### 3. Table Panel
169```json
170{
171  "type": "table",
172  "title": "Service Status",
173  "targets": [{
174    "expr": "up",
175    "format": "table",
176    "instant": true
177  }],
178  "transformations": [
179    {
180      "id": "organize",
181      "options": {
182        "excludeByName": {"Time": true},
183        "indexByName": {},
184        "renameByName": {
185          "instance": "Instance",
186          "job": "Service",
187          "Value": "Status"
188        }
189      }
190    }
191  ]
192}
193```
194 
195### 4. Heatmap
196```json
197{
198  "type": "heatmap",
199  "title": "Latency Heatmap",
200  "targets": [{
201    "expr": "sum(rate(http_request_duration_seconds_bucket[5m])) by (le)",
202    "format": "heatmap"
203  }],
204  "dataFormat": "tsbuckets",
205  "yAxis": {
206    "format": "s"
207  }
208}
209```
210 
211## Variables
212 
213### Query Variables
214```json
215{
216  "templating": {
217    "list": [
218      {
219        "name": "namespace",
220        "type": "query",
221        "datasource": "Prometheus",
222        "query": "label_values(kube_pod_info, namespace)",
223        "refresh": 1,
224        "multi": false
225      },
226      {
227        "name": "service",
228        "type": "query",
229        "datasource": "Prometheus",
230        "query": "label_values(kube_service_info{namespace=\"$namespace\"}, service)",
231        "refresh": 1,
232        "multi": true
233      }
234    ]
235  }
236}
237```
238 
239### Use Variables in Queries
240```
241sum(rate(http_requests_total{namespace="$namespace", service=~"$service"}[5m]))
242```
243 
244## Alerts in Dashboards
245 
246```json
247{
248  "alert": {
249    "name": "High Error Rate",
250    "conditions": [
251      {
252        "evaluator": {
253          "params": [5],
254          "type": "gt"
255        },
256        "operator": {"type": "and"},
257        "query": {
258          "params": ["A", "5m", "now"]
259        },
260        "reducer": {"type": "avg"},
261        "type": "query"
262      }
263    ],
264    "executionErrorState": "alerting",
265    "for": "5m",
266    "frequency": "1m",
267    "message": "Error rate is above 5%",
268    "noDataState": "no_data",
269    "notifications": [
270      {"uid": "slack-channel"}
271    ]
272  }
273}
274```
275 
276## Dashboard Provisioning
277 
278**dashboards.yml:**
279```yaml
280apiVersion: 1
281 
282providers:
283  - name: 'default'
284    orgId: 1
285    folder: 'General'
286    type: file
287    disableDeletion: false
288    updateIntervalSeconds: 10
289    allowUiUpdates: true
290    options:
291      path: /etc/grafana/dashboards
292```
293 
294## Common Dashboard Patterns
295 
296### Infrastructure Dashboard
297 
298**Key Panels:**
299- CPU utilization per node
300- Memory usage per node
301- Disk I/O
302- Network traffic
303- Pod count by namespace
304- Node status
305 
306**Reference:** See `assets/infrastructure-dashboard.json`
307 
308### Database Dashboard
309 
310**Key Panels:**
311- Queries per second
312- Connection pool usage
313- Query latency (P50, P95, P99)
314- Active connections
315- Database size
316- Replication lag
317- Slow queries
318 
319**Reference:** See `assets/database-dashboard.json`
320 
321### Application Dashboard
322 
323**Key Panels:**
324- Request rate
325- Error rate
326- Response time (percentiles)
327- Active users/sessions
328- Cache hit rate
329- Queue length
330 
331## Best Practices
332 
3331. **Start with templates** (Grafana community dashboards)
3342. **Use consistent naming** for panels and variables
3353. **Group related metrics** in rows
3364. **Set appropriate time ranges** (default: Last 6 hours)
3375. **Use variables** for flexibility
3386. **Add panel descriptions** for context
3397. **Configure units** correctly
3408. **Set meaningful thresholds** for colors
3419. **Use consistent colors** across dashboards
34210. **Test with different time ranges**
343 
344## Dashboard as Code
345 
346### Terraform Provisioning
347 
348```hcl
349resource "grafana_dashboard" "api_monitoring" {
350  config_json = file("${path.module}/dashboards/api-monitoring.json")
351  folder      = grafana_folder.monitoring.id
352}
353 
354resource "grafana_folder" "monitoring" {
355  title = "Production Monitoring"
356}
357```
358 
359### Ansible Provisioning
360 
361```yaml
362- name: Deploy Grafana dashboards
363  copy:
364    src: "{{ item }}"
365    dest: /etc/grafana/dashboards/
366  with_fileglob:
367    - "dashboards/*.json"
368  notify: restart grafana
369```
370 
371## Reference Files
372 
373- `assets/api-dashboard.json` - API monitoring dashboard
374- `assets/infrastructure-dashboard.json` - Infrastructure dashboard
375- `assets/database-dashboard.json` - Database monitoring dashboard
376- `references/dashboard-design.md` - Dashboard design guide
377 
378## Related Skills
379 
380- `prometheus-configuration` - For metric collection
381- `slo-implementation` - For SLO dashboards
382