Agent Orchestration Improve Agent

1---
2name: agent-orchestration-improve-agent
3description: "Systematic improvement of existing agents through performance analysis, prompt engineering, and continuous iteration."
4---
5 
6# Agent Performance Optimization Workflow
7 
8Systematic improvement of existing agents through performance analysis, prompt engineering, and continuous iteration.
9 
10[Extended thinking: Agent optimization requires a data-driven approach combining performance metrics, user feedback analysis, and advanced prompt engineering techniques. Success depends on systematic evaluation, targeted improvements, and rigorous testing with rollback capabilities for production safety.]
11 
12## Use this skill when
13 
14- Improving an existing agent's performance or reliability
15- Analyzing failure modes, prompt quality, or tool usage
16- Running structured A/B tests or evaluation suites
17- Designing iterative optimization workflows for agents
18 
19## Do not use this skill when
20 
21- You are building a brand-new agent from scratch
22- There are no metrics, feedback, or test cases available
23- The task is unrelated to agent performance or prompt quality
24 
25## Instructions
26 
271. Establish baseline metrics and collect representative examples.
282. Identify failure modes and prioritize high-impact fixes.
293. Apply prompt and workflow improvements with measurable goals.
304. Validate with tests and roll out changes in controlled stages.
31 
32## Safety
33 
34- Avoid deploying prompt changes without regression testing.
35- Roll back quickly if quality or safety metrics regress.
36 
37## Phase 1: Performance Analysis and Baseline Metrics
38 
39Comprehensive analysis of agent performance using context-manager for historical data collection.
40 
41### 1.1 Gather Performance Data
42 
43```
44Use: context-manager
45Command: analyze-agent-performance $ARGUMENTS --days 30
46```
47 
48Collect metrics including:
49 
50- Task completion rate (successful vs failed tasks)
51- Response accuracy and factual correctness
52- Tool usage efficiency (correct tools, call frequency)
53- Average response time and token consumption
54- User satisfaction indicators (corrections, retries)
55- Hallucination incidents and error patterns
56 
57### 1.2 User Feedback Pattern Analysis
58 
59Identify recurring patterns in user interactions:
60 
61- **Correction patterns**: Where users consistently modify outputs
62- **Clarification requests**: Common areas of ambiguity
63- **Task abandonment**: Points where users give up
64- **Follow-up questions**: Indicators of incomplete responses
65- **Positive feedback**: Successful patterns to preserve
66 
67### 1.3 Failure Mode Classification
68 
69Categorize failures by root cause:
70 
71- **Instruction misunderstanding**: Role or task confusion
72- **Output format errors**: Structure or formatting issues
73- **Context loss**: Long conversation degradation
74- **Tool misuse**: Incorrect or inefficient tool selection
75- **Constraint violations**: Safety or business rule breaches
76- **Edge case handling**: Unusual input scenarios
77 
78### 1.4 Baseline Performance Report
79 
80Generate quantitative baseline metrics:
81 
82```
83Performance Baseline:
84- Task Success Rate: [X%]
85- Average Corrections per Task: [Y]
86- Tool Call Efficiency: [Z%]
87- User Satisfaction Score: [1-10]
88- Average Response Latency: [Xms]
89- Token Efficiency Ratio: [X:Y]
90```
91 
92## Phase 2: Prompt Engineering Improvements
93 
94Apply advanced prompt optimization techniques using prompt-engineer agent.
95 
96### 2.1 Chain-of-Thought Enhancement
97 
98Implement structured reasoning patterns:
99 
100```
101Use: prompt-engineer
102Technique: chain-of-thought-optimization
103```
104 
105- Add explicit reasoning steps: "Let's approach this step-by-step..."
106- Include self-verification checkpoints: "Before proceeding, verify that..."
107- Implement recursive decomposition for complex tasks
108- Add reasoning trace visibility for debugging
109 
110### 2.2 Few-Shot Example Optimization
111 
112Curate high-quality examples from successful interactions:
113 
114- **Select diverse examples** covering common use cases
115- **Include edge cases** that previously failed
116- **Show both positive and negative examples** with explanations
117- **Order examples** from simple to complex
118- **Annotate examples** with key decision points
119 
120Example structure:
121 
122```
123Good Example:
124Input: [User request]
125Reasoning: [Step-by-step thought process]
126Output: [Successful response]
127Why this works: [Key success factors]
128 
129Bad Example:
130Input: [Similar request]
131Output: [Failed response]
132Why this fails: [Specific issues]
133Correct approach: [Fixed version]
134```
135 
136### 2.3 Role Definition Refinement
137 
138Strengthen agent identity and capabilities:
139 
140- **Core purpose**: Clear, single-sentence mission
141- **Expertise domains**: Specific knowledge areas
142- **Behavioral traits**: Personality and interaction style
143- **Tool proficiency**: Available tools and when to use them
144- **Constraints**: What the agent should NOT do
145- **Success criteria**: How to measure task completion
146 
147### 2.4 Constitutional AI Integration
148 
149Implement self-correction mechanisms:
150 
151```
152Constitutional Principles:
1531. Verify factual accuracy before responding
1542. Self-check for potential biases or harmful content
1553. Validate output format matches requirements
1564. Ensure response completeness
1575. Maintain consistency with previous responses
158```
159 
160Add critique-and-revise loops:
161 
162- Initial response generation
163- Self-critique against principles
164- Automatic revision if issues detected
165- Final validation before output
166 
167### 2.5 Output Format Tuning
168 
169Optimize response structure:
170 
171- **Structured templates** for common tasks
172- **Dynamic formatting** based on complexity
173- **Progressive disclosure** for detailed information
174- **Markdown optimization** for readability
175- **Code block formatting** with syntax highlighting
176- **Table and list generation** for data presentation
177 
178## Phase 3: Testing and Validation
179 
180Comprehensive testing framework with A/B comparison.
181 
182### 3.1 Test Suite Development
183 
184Create representative test scenarios:
185 
186```
187Test Categories:
1881. Golden path scenarios (common successful cases)
1892. Previously failed tasks (regression testing)
1903. Edge cases and corner scenarios
1914. Stress tests (complex, multi-step tasks)
1925. Adversarial inputs (potential breaking points)
1936. Cross-domain tasks (combining capabilities)
194```
195 
196### 3.2 A/B Testing Framework
197 
198Compare original vs improved agent:
199 
200```
201Use: parallel-test-runner
202Config:
203  - Agent A: Original version
204  - Agent B: Improved version
205  - Test set: 100 representative tasks
206  - Metrics: Success rate, speed, token usage
207  - Evaluation: Blind human review + automated scoring
208```
209 
210Statistical significance testing:
211 
212- Minimum sample size: 100 tasks per variant
213- Confidence level: 95% (p < 0.05)
214- Effect size calculation (Cohen's d)
215- Power analysis for future tests
216 
217### 3.3 Evaluation Metrics
218 
219Comprehensive scoring framework:
220 
221**Task-Level Metrics:**
222 
223- Completion rate (binary success/failure)
224- Correctness score (0-100% accuracy)
225- Efficiency score (steps taken vs optimal)
226- Tool usage appropriateness
227- Response relevance and completeness
228 
229**Quality Metrics:**
230 
231- Hallucination rate (factual errors per response)
232- Consistency score (alignment with previous responses)
233- Format compliance (matches specified structure)
234- Safety score (constraint adherence)
235- User satisfaction prediction
236 
237**Performance Metrics:**
238 
239- Response latency (time to first token)
240- Total generation time
241- Token consumption (input + output)
242- Cost per task (API usage fees)
243- Memory/context efficiency
244 
245### 3.4 Human Evaluation Protocol
246 
247Structured human review process:
248 
249- Blind evaluation (evaluators don't know version)
250- Standardized rubric with clear criteria
251- Multiple evaluators per sample (inter-rater reliability)
252- Qualitative feedback collection
253- Preference ranking (A vs B comparison)
254 
255## Phase 4: Version Control and Deployment
256 
257Safe rollout with monitoring and rollback capabilities.
258 
259### 4.1 Version Management
260 
261Systematic versioning strategy:
262 
263```
264Version Format: agent-name-v[MAJOR].[MINOR].[PATCH]
265Example: customer-support-v2.3.1
266 
267MAJOR: Significant capability changes
268MINOR: Prompt improvements, new examples
269PATCH: Bug fixes, minor adjustments
270```
271 
272Maintain version history:
273 
274- Git-based prompt storage
275- Changelog with improvement details
276- Performance metrics per version
277- Rollback procedures documented
278 
279### 4.2 Staged Rollout
280 
281Progressive deployment strategy:
282 
2831. **Alpha testing**: Internal team validation (5% traffic)
2842. **Beta testing**: Selected users (20% traffic)
2853. **Canary release**: Gradual increase (20% → 50% → 100%)
2864. **Full deployment**: After success criteria met
2875. **Monitoring period**: 7-day observation window
288 
289### 4.3 Rollback Procedures
290 
291Quick recovery mechanism:
292 
293```
294Rollback Triggers:
295- Success rate drops >10% from baseline
296- Critical errors increase >5%
297- User complaints spike
298- Cost per task increases >20%
299- Safety violations detected
300 
301Rollback Process:
3021. Detect issue via monitoring
3032. Alert team immediately
3043. Switch to previous stable version
3054. Analyze root cause
3065. Fix and re-test before retry
307```
308 
309### 4.4 Continuous Monitoring
310 
311Real-time performance tracking:
312 
313- Dashboard with key metrics
314- Anomaly detection alerts
315- User feedback collection
316- Automated regression testing
317- Weekly performance reports
318 
319## Success Criteria
320 
321Agent improvement is successful when:
322 
323- Task success rate improves by ≥15%
324- User corrections decrease by ≥25%
325- No increase in safety violations
326- Response time remains within 10% of baseline
327- Cost per task doesn't increase >5%
328- Positive user feedback increases
329 
330## Post-Deployment Review
331 
332After 30 days of production use:
333 
3341. Analyze accumulated performance data
3352. Compare against baseline and targets
3363. Identify new improvement opportunities
3374. Document lessons learned
3385. Plan next optimization cycle
339 
340## Continuous Improvement Cycle
341 
342Establish regular improvement cadence:
343 
344- **Weekly**: Monitor metrics and collect feedback
345- **Monthly**: Analyze patterns and plan improvements
346- **Quarterly**: Major version updates with new capabilities
347- **Annually**: Strategic review and architecture updates
348 
349Remember: Agent optimization is an iterative process. Each cycle builds upon previous learnings, gradually improving performance while maintaining stability and safety.
350