Systematic Debugging

1---
2name: systematic-debugging
3description: Use when encountering any bug, test failure, or unexpected behavior, before proposing fixes
4---
5 
6# Systematic Debugging
7 
8## Overview
9 
10Random fixes waste time and create new bugs. Quick patches mask underlying issues.
11 
12**Core principle:** ALWAYS find root cause before attempting fixes. Symptom fixes are failure.
13 
14**Violating the letter of this process is violating the spirit of debugging.**
15 
16## The Iron Law
17 
18```
19NO FIXES WITHOUT ROOT CAUSE INVESTIGATION FIRST
20```
21 
22If you haven't completed Phase 1, you cannot propose fixes.
23 
24## When to Use
25 
26Use for ANY technical issue:
27- Test failures
28- Bugs in production
29- Unexpected behavior
30- Performance problems
31- Build failures
32- Integration issues
33 
34**Use this ESPECIALLY when:**
35- Under time pressure (emergencies make guessing tempting)
36- "Just one quick fix" seems obvious
37- You've already tried multiple fixes
38- Previous fix didn't work
39- You don't fully understand the issue
40 
41**Don't skip when:**
42- Issue seems simple (simple bugs have root causes too)
43- You're in a hurry (rushing guarantees rework)
44- Manager wants it fixed NOW (systematic is faster than thrashing)
45 
46## The Four Phases
47 
48You MUST complete each phase before proceeding to the next.
49 
50### Phase 1: Root Cause Investigation
51 
52**BEFORE attempting ANY fix:**
53 
541. **Read Error Messages Carefully**
55   - Don't skip past errors or warnings
56   - They often contain the exact solution
57   - Read stack traces completely
58   - Note line numbers, file paths, error codes
59 
602. **Reproduce Consistently**
61   - Can you trigger it reliably?
62   - What are the exact steps?
63   - Does it happen every time?
64   - If not reproducible → gather more data, don't guess
65 
663. **Check Recent Changes**
67   - What changed that could cause this?
68   - Git diff, recent commits
69   - New dependencies, config changes
70   - Environmental differences
71 
724. **Gather Evidence in Multi-Component Systems**
73 
74   **WHEN system has multiple components (CI → build → signing, API → service → database):**
75 
76   **BEFORE proposing fixes, add diagnostic instrumentation:**
77   ```
78   For EACH component boundary:
79     - Log what data enters component
80     - Log what data exits component
81     - Verify environment/config propagation
82     - Check state at each layer
83 
84   Run once to gather evidence showing WHERE it breaks
85   THEN analyze evidence to identify failing component
86   THEN investigate that specific component
87   ```
88 
89   **Example (multi-layer system):**
90   ```bash
91   # Layer 1: Workflow
92   echo "=== Secrets available in workflow: ==="
93   echo "IDENTITY: ${IDENTITY:+SET}${IDENTITY:-UNSET}"
94 
95   # Layer 2: Build script
96   echo "=== Env vars in build script: ==="
97   env | grep IDENTITY || echo "IDENTITY not in environment"
98 
99   # Layer 3: Signing script
100   echo "=== Keychain state: ==="
101   security list-keychains
102   security find-identity -v
103 
104   # Layer 4: Actual signing
105   codesign --sign "$IDENTITY" --verbose=4 "$APP"
106   ```
107 
108   **This reveals:** Which layer fails (secrets → workflow ✓, workflow → build ✗)
109 
1105. **Trace Data Flow**
111 
112   **WHEN error is deep in call stack:**
113 
114   See `root-cause-tracing.md` in this directory for the complete backward tracing technique.
115 
116   **Quick version:**
117   - Where does bad value originate?
118   - What called this with bad value?
119   - Keep tracing up until you find the source
120   - Fix at source, not at symptom
121 
122### Phase 2: Pattern Analysis
123 
124**Find the pattern before fixing:**
125 
1261. **Find Working Examples**
127   - Locate similar working code in same codebase
128   - What works that's similar to what's broken?
129 
1302. **Compare Against References**
131   - If implementing pattern, read reference implementation COMPLETELY
132   - Don't skim - read every line
133   - Understand the pattern fully before applying
134 
1353. **Identify Differences**
136   - What's different between working and broken?
137   - List every difference, however small
138   - Don't assume "that can't matter"
139 
1404. **Understand Dependencies**
141   - What other components does this need?
142   - What settings, config, environment?
143   - What assumptions does it make?
144 
145### Phase 3: Hypothesis and Testing
146 
147**Scientific method:**
148 
1491. **Form Single Hypothesis**
150   - State clearly: "I think X is the root cause because Y"
151   - Write it down
152   - Be specific, not vague
153 
1542. **Test Minimally**
155   - Make the SMALLEST possible change to test hypothesis
156   - One variable at a time
157   - Don't fix multiple things at once
158 
1593. **Verify Before Continuing**
160   - Did it work? Yes → Phase 4
161   - Didn't work? Form NEW hypothesis
162   - DON'T add more fixes on top
163 
1644. **When You Don't Know**
165   - Say "I don't understand X"
166   - Don't pretend to know
167   - Ask for help
168   - Research more
169 
170### Phase 4: Implementation
171 
172**Fix the root cause, not the symptom:**
173 
1741. **Create Failing Test Case**
175   - Simplest possible reproduction
176   - Automated test if possible
177   - One-off test script if no framework
178   - MUST have before fixing
179   - Use the `superpowers:test-driven-development` skill for writing proper failing tests
180 
1812. **Implement Single Fix**
182   - Address the root cause identified
183   - ONE change at a time
184   - No "while I'm here" improvements
185   - No bundled refactoring
186 
1873. **Verify Fix**
188   - Test passes now?
189   - No other tests broken?
190   - Issue actually resolved?
191 
1924. **If Fix Doesn't Work**
193   - STOP
194   - Count: How many fixes have you tried?
195   - If < 3: Return to Phase 1, re-analyze with new information
196   - **If ≥ 3: STOP and question the architecture (step 5 below)**
197   - DON'T attempt Fix #4 without architectural discussion
198 
1995. **If 3+ Fixes Failed: Question Architecture**
200 
201   **Pattern indicating architectural problem:**
202   - Each fix reveals new shared state/coupling/problem in different place
203   - Fixes require "massive refactoring" to implement
204   - Each fix creates new symptoms elsewhere
205 
206   **STOP and question fundamentals:**
207   - Is this pattern fundamentally sound?
208   - Are we "sticking with it through sheer inertia"?
209   - Should we refactor architecture vs. continue fixing symptoms?
210 
211   **Discuss with your human partner before attempting more fixes**
212 
213   This is NOT a failed hypothesis - this is a wrong architecture.
214 
215## Red Flags - STOP and Follow Process
216 
217If you catch yourself thinking:
218- "Quick fix for now, investigate later"
219- "Just try changing X and see if it works"
220- "Add multiple changes, run tests"
221- "Skip the test, I'll manually verify"
222- "It's probably X, let me fix that"
223- "I don't fully understand but this might work"
224- "Pattern says X but I'll adapt it differently"
225- "Here are the main problems: [lists fixes without investigation]"
226- Proposing solutions before tracing data flow
227- **"One more fix attempt" (when already tried 2+)**
228- **Each fix reveals new problem in different place**
229 
230**ALL of these mean: STOP. Return to Phase 1.**
231 
232**If 3+ fixes failed:** Question the architecture (see Phase 4.5)
233 
234## your human partner's Signals You're Doing It Wrong
235 
236**Watch for these redirections:**
237- "Is that not happening?" - You assumed without verifying
238- "Will it show us...?" - You should have added evidence gathering
239- "Stop guessing" - You're proposing fixes without understanding
240- "Ultrathink this" - Question fundamentals, not just symptoms
241- "We're stuck?" (frustrated) - Your approach isn't working
242 
243**When you see these:** STOP. Return to Phase 1.
244 
245## Common Rationalizations
246 
247| Excuse | Reality |
248|--------|---------|
249| "Issue is simple, don't need process" | Simple issues have root causes too. Process is fast for simple bugs. |
250| "Emergency, no time for process" | Systematic debugging is FASTER than guess-and-check thrashing. |
251| "Just try this first, then investigate" | First fix sets the pattern. Do it right from the start. |
252| "I'll write test after confirming fix works" | Untested fixes don't stick. Test first proves it. |
253| "Multiple fixes at once saves time" | Can't isolate what worked. Causes new bugs. |
254| "Reference too long, I'll adapt the pattern" | Partial understanding guarantees bugs. Read it completely. |
255| "I see the problem, let me fix it" | Seeing symptoms ≠ understanding root cause. |
256| "One more fix attempt" (after 2+ failures) | 3+ failures = architectural problem. Question pattern, don't fix again. |
257 
258## Quick Reference
259 
260| Phase | Key Activities | Success Criteria |
261|-------|---------------|------------------|
262| **1. Root Cause** | Read errors, reproduce, check changes, gather evidence | Understand WHAT and WHY |
263| **2. Pattern** | Find working examples, compare | Identify differences |
264| **3. Hypothesis** | Form theory, test minimally | Confirmed or new hypothesis |
265| **4. Implementation** | Create test, fix, verify | Bug resolved, tests pass |
266 
267## When Process Reveals "No Root Cause"
268 
269If systematic investigation reveals issue is truly environmental, timing-dependent, or external:
270 
2711. You've completed the process
2722. Document what you investigated
2733. Implement appropriate handling (retry, timeout, error message)
2744. Add monitoring/logging for future investigation
275 
276**But:** 95% of "no root cause" cases are incomplete investigation.
277 
278## Supporting Techniques
279 
280These techniques are part of systematic debugging and available in this directory:
281 
282- **`root-cause-tracing.md`** - Trace bugs backward through call stack to find original trigger
283- **`defense-in-depth.md`** - Add validation at multiple layers after finding root cause
284- **`condition-based-waiting.md`** - Replace arbitrary timeouts with condition polling
285 
286**Related skills:**
287- **superpowers:test-driven-development** - For creating failing test case (Phase 4, Step 1)
288- **superpowers:verification-before-completion** - Verify fix worked before claiming success
289 
290## Real-World Impact
291 
292From debugging sessions:
293- Systematic approach: 15-30 minutes to fix
294- Random fixes approach: 2-3 hours of thrashing
295- First-time fix rate: 95% vs 40%
296- New bugs introduced: Near zero vs common
297