Incident Response Incident Response

1---
2name: incident-response-incident-response
3description: "Use when working with incident response incident response"
4---
5 
6## Use this skill when
7 
8- Working on incident response incident response tasks or workflows
9- Needing guidance, best practices, or checklists for incident response incident response
10 
11## Do not use this skill when
12 
13- The task is unrelated to incident response incident response
14- You need a different domain or tool outside this scope
15 
16## Instructions
17 
18- Clarify goals, constraints, and required inputs.
19- Apply relevant best practices and validate outcomes.
20- Provide actionable steps and verification.
21- If detailed examples are required, open `resources/implementation-playbook.md`.
22 
23Orchestrate multi-agent incident response with modern SRE practices for rapid resolution and learning:
24 
25[Extended thinking: This workflow implements a comprehensive incident command system (ICS) following modern SRE principles. Multiple specialized agents collaborate through defined phases: detection/triage, investigation/mitigation, communication/coordination, and resolution/postmortem. The workflow emphasizes speed without sacrificing accuracy, maintains clear communication channels, and ensures every incident becomes a learning opportunity through blameless postmortems and systematic improvements.]
26 
27## Configuration
28 
29### Severity Levels
30- **P0/SEV-1**: Complete outage, security breach, data loss - immediate all-hands response
31- **P1/SEV-2**: Major degradation, significant user impact - rapid response required
32- **P2/SEV-3**: Minor degradation, limited impact - standard response
33- **P3/SEV-4**: Cosmetic issues, no user impact - scheduled resolution
34 
35### Incident Types
36- Performance degradation
37- Service outage
38- Security incident
39- Data integrity issue
40- Infrastructure failure
41- Third-party service disruption
42 
43## Phase 1: Detection & Triage
44 
45### 1. Incident Detection and Classification
46- Use Task tool with subagent_type="incident-responder"
47- Prompt: "URGENT: Detect and classify incident: $ARGUMENTS. Analyze alerts from PagerDuty/Opsgenie/monitoring. Determine: 1) Incident severity (P0-P3), 2) Affected services and dependencies, 3) User impact and business risk, 4) Initial incident command structure needed. Check error budgets and SLO violations."
48- Output: Severity classification, impact assessment, incident command assignments, SLO status
49- Context: Initial alerts, monitoring dashboards, recent changes
50 
51### 2. Observability Analysis
52- Use Task tool with subagent_type="observability-monitoring::observability-engineer"
53- Prompt: "Perform rapid observability sweep for incident: $ARGUMENTS. Query: 1) Distributed tracing (OpenTelemetry/Jaeger), 2) Metrics correlation (Prometheus/Grafana/DataDog), 3) Log aggregation (ELK/Splunk), 4) APM data, 5) Real User Monitoring. Identify anomalies, error patterns, and service degradation points."
54- Output: Observability findings, anomaly detection, service health matrix, trace analysis
55- Context: Severity level from step 1, affected services
56 
57### 3. Initial Mitigation
58- Use Task tool with subagent_type="incident-responder"
59- Prompt: "Implement immediate mitigation for P$SEVERITY incident: $ARGUMENTS. Actions: 1) Traffic throttling/rerouting if needed, 2) Feature flag disabling for affected features, 3) Circuit breaker activation, 4) Rollback assessment for recent deployments, 5) Scale resources if capacity-related. Prioritize user experience restoration."
60- Output: Mitigation actions taken, temporary fixes applied, rollback decisions
61- Context: Observability findings, severity classification
62 
63## Phase 2: Investigation & Root Cause Analysis
64 
65### 4. Deep System Debugging
66- Use Task tool with subagent_type="error-debugging::debugger"
67- Prompt: "Conduct deep debugging for incident: $ARGUMENTS using observability data. Investigate: 1) Stack traces and error logs, 2) Database query performance and locks, 3) Network latency and timeouts, 4) Memory leaks and CPU spikes, 5) Dependency failures and cascading errors. Apply Five Whys analysis."
68- Output: Root cause identification, contributing factors, dependency impact map
69- Context: Observability analysis, mitigation status
70 
71### 5. Security Assessment
72- Use Task tool with subagent_type="security-scanning::security-auditor"
73- Prompt: "Assess security implications of incident: $ARGUMENTS. Check: 1) DDoS attack indicators, 2) Authentication/authorization failures, 3) Data exposure risks, 4) Certificate issues, 5) Suspicious access patterns. Review WAF logs, security groups, and audit trails."
74- Output: Security assessment, breach analysis, vulnerability identification
75- Context: Root cause findings, system logs
76 
77### 6. Performance Engineering Analysis
78- Use Task tool with subagent_type="application-performance::performance-engineer"
79- Prompt: "Analyze performance aspects of incident: $ARGUMENTS. Examine: 1) Resource utilization patterns, 2) Query optimization opportunities, 3) Caching effectiveness, 4) Load balancer health, 5) CDN performance, 6) Autoscaling triggers. Identify bottlenecks and capacity issues."
80- Output: Performance bottlenecks, resource recommendations, optimization opportunities
81- Context: Debug findings, current mitigation state
82 
83## Phase 3: Resolution & Recovery
84 
85### 7. Fix Implementation
86- Use Task tool with subagent_type="backend-development::backend-architect"
87- Prompt: "Design and implement production fix for incident: $ARGUMENTS based on root cause. Requirements: 1) Minimal viable fix for rapid deployment, 2) Risk assessment and rollback capability, 3) Staged rollout plan with monitoring, 4) Validation criteria and health checks. Consider both immediate fix and long-term solution."
88- Output: Fix implementation, deployment strategy, validation plan, rollback procedures
89- Context: Root cause analysis, performance findings, security assessment
90 
91### 8. Deployment and Validation
92- Use Task tool with subagent_type="deployment-strategies::deployment-engineer"
93- Prompt: "Execute emergency deployment for incident fix: $ARGUMENTS. Process: 1) Blue-green or canary deployment, 2) Progressive rollout with monitoring, 3) Health check validation at each stage, 4) Rollback triggers configured, 5) Real-time monitoring during deployment. Coordinate with incident command."
94- Output: Deployment status, validation results, monitoring dashboard, rollback readiness
95- Context: Fix implementation, current system state
96 
97## Phase 4: Communication & Coordination
98 
99### 9. Stakeholder Communication
100- Use Task tool with subagent_type="content-marketing::content-marketer"
101- Prompt: "Manage incident communication for: $ARGUMENTS. Create: 1) Status page updates (public-facing), 2) Internal engineering updates (technical details), 3) Executive summary (business impact/ETA), 4) Customer support briefing (talking points), 5) Timeline documentation with key decisions. Update every 15-30 minutes based on severity."
102- Output: Communication artifacts, status updates, stakeholder briefings, timeline log
103- Context: All previous phases, current resolution status
104 
105### 10. Customer Impact Assessment
106- Use Task tool with subagent_type="incident-responder"
107- Prompt: "Assess and document customer impact for incident: $ARGUMENTS. Analyze: 1) Affected user segments and geography, 2) Failed transactions or data loss, 3) SLA violations and contractual implications, 4) Customer support ticket volume, 5) Revenue impact estimation. Prepare proactive customer outreach list."
108- Output: Customer impact report, SLA analysis, outreach recommendations
109- Context: Resolution progress, communication status
110 
111## Phase 5: Postmortem & Prevention
112 
113### 11. Blameless Postmortem
114- Use Task tool with subagent_type="documentation-generation::docs-architect"
115- Prompt: "Conduct blameless postmortem for incident: $ARGUMENTS. Document: 1) Complete incident timeline with decisions, 2) Root cause and contributing factors (systems focus), 3) What went well in response, 4) What could improve, 5) Action items with owners and deadlines, 6) Lessons learned for team education. Follow SRE postmortem best practices."
116- Output: Postmortem document, action items list, process improvements, training needs
117- Context: Complete incident history, all agent outputs
118 
119### 12. Monitoring and Alert Enhancement
120- Use Task tool with subagent_type="observability-monitoring::observability-engineer"
121- Prompt: "Enhance monitoring to prevent recurrence of: $ARGUMENTS. Implement: 1) New alerts for early detection, 2) SLI/SLO adjustments if needed, 3) Dashboard improvements for visibility, 4) Runbook automation opportunities, 5) Chaos engineering scenarios for testing. Ensure alerts are actionable and reduce noise."
122- Output: New monitoring configuration, alert rules, dashboard updates, runbook automation
123- Context: Postmortem findings, root cause analysis
124 
125### 13. System Hardening
126- Use Task tool with subagent_type="backend-development::backend-architect"
127- Prompt: "Design system improvements to prevent incident: $ARGUMENTS. Propose: 1) Architecture changes for resilience (circuit breakers, bulkheads), 2) Graceful degradation strategies, 3) Capacity planning adjustments, 4) Technical debt prioritization, 5) Dependency reduction opportunities. Create implementation roadmap."
128- Output: Architecture improvements, resilience patterns, technical debt items, roadmap
129- Context: Postmortem action items, performance analysis
130 
131## Success Criteria
132 
133### Immediate Success (During Incident)
134- Service restoration within SLA targets
135- Accurate severity classification within 5 minutes
136- Stakeholder communication every 15-30 minutes
137- No cascading failures or incident escalation
138- Clear incident command structure maintained
139 
140### Long-term Success (Post-Incident)
141- Comprehensive postmortem within 48 hours
142- All action items assigned with deadlines
143- Monitoring improvements deployed within 1 week
144- Runbook updates completed
145- Team training conducted on lessons learned
146- Error budget impact assessed and communicated
147 
148## Coordination Protocols
149 
150### Incident Command Structure
151- **Incident Commander**: Decision authority, coordination
152- **Technical Lead**: Technical investigation and resolution
153- **Communications Lead**: Stakeholder updates
154- **Subject Matter Experts**: Specific system expertise
155 
156### Communication Channels
157- War room (Slack/Teams channel or Zoom)
158- Status page updates (StatusPage, Statusly)
159- PagerDuty/Opsgenie for alerting
160- Confluence/Notion for documentation
161 
162### Handoff Requirements
163- Each phase provides clear context to the next
164- All findings documented in shared incident doc
165- Decision rationale recorded for postmortem
166- Timestamp all significant events
167 
168Production incident requiring immediate response: $ARGUMENTS
169