Improve Agent

1# Agent Performance Optimization Workflow
2 
3Systematic improvement of existing agents through performance analysis, prompt engineering, and continuous iteration.
4 
5[Extended thinking: Agent optimization requires a data-driven approach combining performance metrics, user feedback analysis, and advanced prompt engineering techniques. Success depends on systematic evaluation, targeted improvements, and rigorous testing with rollback capabilities for production safety.]
6 
7## Phase 1: Performance Analysis and Baseline Metrics
8 
9Comprehensive analysis of agent performance using context-manager for historical data collection.
10 
11### 1.1 Gather Performance Data
12 
13```
14Use: context-manager
15Command: analyze-agent-performance $ARGUMENTS --days 30
16```
17 
18Collect metrics including:
19 
20- Task completion rate (successful vs failed tasks)
21- Response accuracy and factual correctness
22- Tool usage efficiency (correct tools, call frequency)
23- Average response time and token consumption
24- User satisfaction indicators (corrections, retries)
25- Hallucination incidents and error patterns
26 
27### 1.2 User Feedback Pattern Analysis
28 
29Identify recurring patterns in user interactions:
30 
31- **Correction patterns**: Where users consistently modify outputs
32- **Clarification requests**: Common areas of ambiguity
33- **Task abandonment**: Points where users give up
34- **Follow-up questions**: Indicators of incomplete responses
35- **Positive feedback**: Successful patterns to preserve
36 
37### 1.3 Failure Mode Classification
38 
39Categorize failures by root cause:
40 
41- **Instruction misunderstanding**: Role or task confusion
42- **Output format errors**: Structure or formatting issues
43- **Context loss**: Long conversation degradation
44- **Tool misuse**: Incorrect or inefficient tool selection
45- **Constraint violations**: Safety or business rule breaches
46- **Edge case handling**: Unusual input scenarios
47 
48### 1.4 Baseline Performance Report
49 
50Generate quantitative baseline metrics:
51 
52```
53Performance Baseline:
54- Task Success Rate: [X%]
55- Average Corrections per Task: [Y]
56- Tool Call Efficiency: [Z%]
57- User Satisfaction Score: [1-10]
58- Average Response Latency: [Xms]
59- Token Efficiency Ratio: [X:Y]
60```
61 
62## Phase 2: Prompt Engineering Improvements
63 
64Apply advanced prompt optimization techniques using prompt-engineer agent.
65 
66### 2.1 Chain-of-Thought Enhancement
67 
68Implement structured reasoning patterns:
69 
70```
71Use: prompt-engineer
72Technique: chain-of-thought-optimization
73```
74 
75- Add explicit reasoning steps: "Let's approach this step-by-step..."
76- Include self-verification checkpoints: "Before proceeding, verify that..."
77- Implement recursive decomposition for complex tasks
78- Add reasoning trace visibility for debugging
79 
80### 2.2 Few-Shot Example Optimization
81 
82Curate high-quality examples from successful interactions:
83 
84- **Select diverse examples** covering common use cases
85- **Include edge cases** that previously failed
86- **Show both positive and negative examples** with explanations
87- **Order examples** from simple to complex
88- **Annotate examples** with key decision points
89 
90Example structure:
91 
92```
93Good Example:
94Input: [User request]
95Reasoning: [Step-by-step thought process]
96Output: [Successful response]
97Why this works: [Key success factors]
98 
99Bad Example:
100Input: [Similar request]
101Output: [Failed response]
102Why this fails: [Specific issues]
103Correct approach: [Fixed version]
104```
105 
106### 2.3 Role Definition Refinement
107 
108Strengthen agent identity and capabilities:
109 
110- **Core purpose**: Clear, single-sentence mission
111- **Expertise domains**: Specific knowledge areas
112- **Behavioral traits**: Personality and interaction style
113- **Tool proficiency**: Available tools and when to use them
114- **Constraints**: What the agent should NOT do
115- **Success criteria**: How to measure task completion
116 
117### 2.4 Constitutional AI Integration
118 
119Implement self-correction mechanisms:
120 
121```
122Constitutional Principles:
1231. Verify factual accuracy before responding
1242. Self-check for potential biases or harmful content
1253. Validate output format matches requirements
1264. Ensure response completeness
1275. Maintain consistency with previous responses
128```
129 
130Add critique-and-revise loops:
131 
132- Initial response generation
133- Self-critique against principles
134- Automatic revision if issues detected
135- Final validation before output
136 
137### 2.5 Output Format Tuning
138 
139Optimize response structure:
140 
141- **Structured templates** for common tasks
142- **Dynamic formatting** based on complexity
143- **Progressive disclosure** for detailed information
144- **Markdown optimization** for readability
145- **Code block formatting** with syntax highlighting
146- **Table and list generation** for data presentation
147 
148## Phase 3: Testing and Validation
149 
150Comprehensive testing framework with A/B comparison.
151 
152### 3.1 Test Suite Development
153 
154Create representative test scenarios:
155 
156```
157Test Categories:
1581. Golden path scenarios (common successful cases)
1592. Previously failed tasks (regression testing)
1603. Edge cases and corner scenarios
1614. Stress tests (complex, multi-step tasks)
1625. Adversarial inputs (potential breaking points)
1636. Cross-domain tasks (combining capabilities)
164```
165 
166### 3.2 A/B Testing Framework
167 
168Compare original vs improved agent:
169 
170```
171Use: parallel-test-runner
172Config:
173  - Agent A: Original version
174  - Agent B: Improved version
175  - Test set: 100 representative tasks
176  - Metrics: Success rate, speed, token usage
177  - Evaluation: Blind human review + automated scoring
178```
179 
180Statistical significance testing:
181 
182- Minimum sample size: 100 tasks per variant
183- Confidence level: 95% (p < 0.05)
184- Effect size calculation (Cohen's d)
185- Power analysis for future tests
186 
187### 3.3 Evaluation Metrics
188 
189Comprehensive scoring framework:
190 
191**Task-Level Metrics:**
192 
193- Completion rate (binary success/failure)
194- Correctness score (0-100% accuracy)
195- Efficiency score (steps taken vs optimal)
196- Tool usage appropriateness
197- Response relevance and completeness
198 
199**Quality Metrics:**
200 
201- Hallucination rate (factual errors per response)
202- Consistency score (alignment with previous responses)
203- Format compliance (matches specified structure)
204- Safety score (constraint adherence)
205- User satisfaction prediction
206 
207**Performance Metrics:**
208 
209- Response latency (time to first token)
210- Total generation time
211- Token consumption (input + output)
212- Cost per task (API usage fees)
213- Memory/context efficiency
214 
215### 3.4 Human Evaluation Protocol
216 
217Structured human review process:
218 
219- Blind evaluation (evaluators don't know version)
220- Standardized rubric with clear criteria
221- Multiple evaluators per sample (inter-rater reliability)
222- Qualitative feedback collection
223- Preference ranking (A vs B comparison)
224 
225## Phase 4: Version Control and Deployment
226 
227Safe rollout with monitoring and rollback capabilities.
228 
229### 4.1 Version Management
230 
231Systematic versioning strategy:
232 
233```
234Version Format: agent-name-v[MAJOR].[MINOR].[PATCH]
235Example: customer-support-v2.3.1
236 
237MAJOR: Significant capability changes
238MINOR: Prompt improvements, new examples
239PATCH: Bug fixes, minor adjustments
240```
241 
242Maintain version history:
243 
244- Git-based prompt storage
245- Changelog with improvement details
246- Performance metrics per version
247- Rollback procedures documented
248 
249### 4.2 Staged Rollout
250 
251Progressive deployment strategy:
252 
2531. **Alpha testing**: Internal team validation (5% traffic)
2542. **Beta testing**: Selected users (20% traffic)
2553. **Canary release**: Gradual increase (20% → 50% → 100%)
2564. **Full deployment**: After success criteria met
2575. **Monitoring period**: 7-day observation window
258 
259### 4.3 Rollback Procedures
260 
261Quick recovery mechanism:
262 
263```
264Rollback Triggers:
265- Success rate drops >10% from baseline
266- Critical errors increase >5%
267- User complaints spike
268- Cost per task increases >20%
269- Safety violations detected
270 
271Rollback Process:
2721. Detect issue via monitoring
2732. Alert team immediately
2743. Switch to previous stable version
2754. Analyze root cause
2765. Fix and re-test before retry
277```
278 
279### 4.4 Continuous Monitoring
280 
281Real-time performance tracking:
282 
283- Dashboard with key metrics
284- Anomaly detection alerts
285- User feedback collection
286- Automated regression testing
287- Weekly performance reports
288 
289## Success Criteria
290 
291Agent improvement is successful when:
292 
293- Task success rate improves by ≥15%
294- User corrections decrease by ≥25%
295- No increase in safety violations
296- Response time remains within 10% of baseline
297- Cost per task doesn't increase >5%
298- Positive user feedback increases
299 
300## Post-Deployment Review
301 
302After 30 days of production use:
303 
3041. Analyze accumulated performance data
3052. Compare against baseline and targets
3063. Identify new improvement opportunities
3074. Document lessons learned
3085. Plan next optimization cycle
309 
310## Continuous Improvement Cycle
311 
312Establish regular improvement cadence:
313 
314- **Weekly**: Monitor metrics and collect feedback
315- **Monthly**: Analyze patterns and plan improvements
316- **Quarterly**: Major version updates with new capabilities
317- **Annually**: Strategic review and architecture updates
318 
319Remember: Agent optimization is an iterative process. Each cycle builds upon previous learnings, gradually improving performance while maintaining stability and safety.
320