Machine Learning Ops ML Pipeline

1---
2name: machine-learning-ops-ml-pipeline
3description: "Design and implement a complete ML pipeline for: $ARGUMENTS"
4---
5 
6# Machine Learning Pipeline - Multi-Agent MLOps Orchestration
7 
8Design and implement a complete ML pipeline for: $ARGUMENTS
9 
10## Use this skill when
11 
12- Working on machine learning pipeline - multi-agent mlops orchestration tasks or workflows
13- Needing guidance, best practices, or checklists for machine learning pipeline - multi-agent mlops orchestration
14 
15## Do not use this skill when
16 
17- The task is unrelated to machine learning pipeline - multi-agent mlops orchestration
18- You need a different domain or tool outside this scope
19 
20## Instructions
21 
22- Clarify goals, constraints, and required inputs.
23- Apply relevant best practices and validate outcomes.
24- Provide actionable steps and verification.
25- If detailed examples are required, open `resources/implementation-playbook.md`.
26 
27## Thinking
28 
29This workflow orchestrates multiple specialized agents to build a production-ready ML pipeline following modern MLOps best practices. The approach emphasizes:
30 
31- **Phase-based coordination**: Each phase builds upon previous outputs, with clear handoffs between agents
32- **Modern tooling integration**: MLflow/W&B for experiments, Feast/Tecton for features, KServe/Seldon for serving
33- **Production-first mindset**: Every component designed for scale, monitoring, and reliability
34- **Reproducibility**: Version control for data, models, and infrastructure
35- **Continuous improvement**: Automated retraining, A/B testing, and drift detection
36 
37The multi-agent approach ensures each aspect is handled by domain experts:
38- Data engineers handle ingestion and quality
39- Data scientists design features and experiments
40- ML engineers implement training pipelines
41- MLOps engineers handle production deployment
42- Observability engineers ensure monitoring
43 
44## Phase 1: Data & Requirements Analysis
45 
46<Task>
47subagent_type: data-engineer
48prompt: |
49  Analyze and design data pipeline for ML system with requirements: $ARGUMENTS
50 
51  Deliverables:
52  1. Data source audit and ingestion strategy:
53     - Source systems and connection patterns
54     - Schema validation using Pydantic/Great Expectations
55     - Data versioning with DVC or lakeFS
56     - Incremental loading and CDC strategies
57 
58  2. Data quality framework:
59     - Profiling and statistics generation
60     - Anomaly detection rules
61     - Data lineage tracking
62     - Quality gates and SLAs
63 
64  3. Storage architecture:
65     - Raw/processed/feature layers
66     - Partitioning strategy
67     - Retention policies
68     - Cost optimization
69 
70  Provide implementation code for critical components and integration patterns.
71</Task>
72 
73<Task>
74subagent_type: data-scientist
75prompt: |
76  Design feature engineering and model requirements for: $ARGUMENTS
77  Using data architecture from: {phase1.data-engineer.output}
78 
79  Deliverables:
80  1. Feature engineering pipeline:
81     - Transformation specifications
82     - Feature store schema (Feast/Tecton)
83     - Statistical validation rules
84     - Handling strategies for missing data/outliers
85 
86  2. Model requirements:
87     - Algorithm selection rationale
88     - Performance metrics and baselines
89     - Training data requirements
90     - Evaluation criteria and thresholds
91 
92  3. Experiment design:
93     - Hypothesis and success metrics
94     - A/B testing methodology
95     - Sample size calculations
96     - Bias detection approach
97 
98  Include feature transformation code and statistical validation logic.
99</Task>
100 
101## Phase 2: Model Development & Training
102 
103<Task>
104subagent_type: ml-engineer
105prompt: |
106  Implement training pipeline based on requirements: {phase1.data-scientist.output}
107  Using data pipeline: {phase1.data-engineer.output}
108 
109  Build comprehensive training system:
110  1. Training pipeline implementation:
111     - Modular training code with clear interfaces
112     - Hyperparameter optimization (Optuna/Ray Tune)
113     - Distributed training support (Horovod/PyTorch DDP)
114     - Cross-validation and ensemble strategies
115 
116  2. Experiment tracking setup:
117     - MLflow/Weights & Biases integration
118     - Metric logging and visualization
119     - Artifact management (models, plots, data samples)
120     - Experiment comparison and analysis tools
121 
122  3. Model registry integration:
123     - Version control and tagging strategy
124     - Model metadata and lineage
125     - Promotion workflows (dev -> staging -> prod)
126     - Rollback procedures
127 
128  Provide complete training code with configuration management.
129</Task>
130 
131<Task>
132subagent_type: python-pro
133prompt: |
134  Optimize and productionize ML code from: {phase2.ml-engineer.output}
135 
136  Focus areas:
137  1. Code quality and structure:
138     - Refactor for production standards
139     - Add comprehensive error handling
140     - Implement proper logging with structured formats
141     - Create reusable components and utilities
142 
143  2. Performance optimization:
144     - Profile and optimize bottlenecks
145     - Implement caching strategies
146     - Optimize data loading and preprocessing
147     - Memory management for large-scale training
148 
149  3. Testing framework:
150     - Unit tests for data transformations
151     - Integration tests for pipeline components
152     - Model quality tests (invariance, directional)
153     - Performance regression tests
154 
155  Deliver production-ready, maintainable code with full test coverage.
156</Task>
157 
158## Phase 3: Production Deployment & Serving
159 
160<Task>
161subagent_type: mlops-engineer
162prompt: |
163  Design production deployment for models from: {phase2.ml-engineer.output}
164  With optimized code from: {phase2.python-pro.output}
165 
166  Implementation requirements:
167  1. Model serving infrastructure:
168     - REST/gRPC APIs with FastAPI/TorchServe
169     - Batch prediction pipelines (Airflow/Kubeflow)
170     - Stream processing (Kafka/Kinesis integration)
171     - Model serving platforms (KServe/Seldon Core)
172 
173  2. Deployment strategies:
174     - Blue-green deployments for zero downtime
175     - Canary releases with traffic splitting
176     - Shadow deployments for validation
177     - A/B testing infrastructure
178 
179  3. CI/CD pipeline:
180     - GitHub Actions/GitLab CI workflows
181     - Automated testing gates
182     - Model validation before deployment
183     - ArgoCD for GitOps deployment
184 
185  4. Infrastructure as Code:
186     - Terraform modules for cloud resources
187     - Helm charts for Kubernetes deployments
188     - Docker multi-stage builds for optimization
189     - Secret management with Vault/Secrets Manager
190 
191  Provide complete deployment configuration and automation scripts.
192</Task>
193 
194<Task>
195subagent_type: kubernetes-architect
196prompt: |
197  Design Kubernetes infrastructure for ML workloads from: {phase3.mlops-engineer.output}
198 
199  Kubernetes-specific requirements:
200  1. Workload orchestration:
201     - Training job scheduling with Kubeflow
202     - GPU resource allocation and sharing
203     - Spot/preemptible instance integration
204     - Priority classes and resource quotas
205 
206  2. Serving infrastructure:
207     - HPA/VPA for autoscaling
208     - KEDA for event-driven scaling
209     - Istio service mesh for traffic management
210     - Model caching and warm-up strategies
211 
212  3. Storage and data access:
213     - PVC strategies for training data
214     - Model artifact storage with CSI drivers
215     - Distributed storage for feature stores
216     - Cache layers for inference optimization
217 
218  Provide Kubernetes manifests and Helm charts for entire ML platform.
219</Task>
220 
221## Phase 4: Monitoring & Continuous Improvement
222 
223<Task>
224subagent_type: observability-engineer
225prompt: |
226  Implement comprehensive monitoring for ML system deployed in: {phase3.mlops-engineer.output}
227  Using Kubernetes infrastructure: {phase3.kubernetes-architect.output}
228 
229  Monitoring framework:
230  1. Model performance monitoring:
231     - Prediction accuracy tracking
232     - Latency and throughput metrics
233     - Feature importance shifts
234     - Business KPI correlation
235 
236  2. Data and model drift detection:
237     - Statistical drift detection (KS test, PSI)
238     - Concept drift monitoring
239     - Feature distribution tracking
240     - Automated drift alerts and reports
241 
242  3. System observability:
243     - Prometheus metrics for all components
244     - Grafana dashboards for visualization
245     - Distributed tracing with Jaeger/Zipkin
246     - Log aggregation with ELK/Loki
247 
248  4. Alerting and automation:
249     - PagerDuty/Opsgenie integration
250     - Automated retraining triggers
251     - Performance degradation workflows
252     - Incident response runbooks
253 
254  5. Cost tracking:
255     - Resource utilization metrics
256     - Cost allocation by model/experiment
257     - Optimization recommendations
258     - Budget alerts and controls
259 
260  Deliver monitoring configuration, dashboards, and alert rules.
261</Task>
262 
263## Configuration Options
264 
265- **experiment_tracking**: mlflow | wandb | neptune | clearml
266- **feature_store**: feast | tecton | databricks | custom
267- **serving_platform**: kserve | seldon | torchserve | triton
268- **orchestration**: kubeflow | airflow | prefect | dagster
269- **cloud_provider**: aws | azure | gcp | multi-cloud
270- **deployment_mode**: realtime | batch | streaming | hybrid
271- **monitoring_stack**: prometheus | datadog | newrelic | custom
272 
273## Success Criteria
274 
2751. **Data Pipeline Success**:
276   - < 0.1% data quality issues in production
277   - Automated data validation passing 99.9% of time
278   - Complete data lineage tracking
279   - Sub-second feature serving latency
280 
2812. **Model Performance**:
282   - Meeting or exceeding baseline metrics
283   - < 5% performance degradation before retraining
284   - Successful A/B tests with statistical significance
285   - No undetected model drift > 24 hours
286 
2873. **Operational Excellence**:
288   - 99.9% uptime for model serving
289   - < 200ms p99 inference latency
290   - Automated rollback within 5 minutes
291   - Complete observability with < 1 minute alert time
292 
2934. **Development Velocity**:
294   - < 1 hour from commit to production
295   - Parallel experiment execution
296   - Reproducible training runs
297   - Self-service model deployment
298 
2995. **Cost Efficiency**:
300   - < 20% infrastructure waste
301   - Optimized resource allocation
302   - Automatic scaling based on load
303   - Spot instance utilization > 60%
304 
305## Final Deliverables
306 
307Upon completion, the orchestrated pipeline will provide:
308- End-to-end ML pipeline with full automation
309- Comprehensive documentation and runbooks
310- Production-ready infrastructure as code
311- Complete monitoring and alerting system
312- CI/CD pipelines for continuous improvement
313- Cost optimization and scaling strategies
314- Disaster recovery and rollback procedures
315