Airflow Dag Patterns

1---
2name: airflow-dag-patterns
3description: Build production Apache Airflow DAGs with best practices for operators, sensors, testing, and deployment. Use when creating data pipelines, orchestrating workflows, or scheduling batch jobs.
4---
5 
6# Apache Airflow DAG Patterns
7 
8Production-ready patterns for Apache Airflow including DAG design, operators, sensors, testing, and deployment strategies.
9 
10## Use this skill when
11 
12- Creating data pipeline orchestration with Airflow
13- Designing DAG structures and dependencies
14- Implementing custom operators and sensors
15- Testing Airflow DAGs locally
16- Setting up Airflow in production
17- Debugging failed DAG runs
18 
19## Do not use this skill when
20 
21- You only need a simple cron job or shell script
22- Airflow is not part of the tooling stack
23- The task is unrelated to workflow orchestration
24 
25## Instructions
26 
271. Identify data sources, schedules, and dependencies.
282. Design idempotent tasks with clear ownership and retries.
293. Implement DAGs with observability and alerting hooks.
304. Validate in staging and document operational runbooks.
31 
32Refer to `resources/implementation-playbook.md` for detailed patterns, checklists, and templates.
33 
34## Safety
35 
36- Avoid changing production DAG schedules without approval.
37- Test backfills and retries carefully to prevent data duplication.
38 
39## Resources
40 
41- `resources/implementation-playbook.md` for detailed patterns, checklists, and templates.
42