Workflow Orchestration Patterns

1---
2name: workflow-orchestration-patterns
3description: Design durable workflows with Temporal for distributed systems. Covers workflow vs activity separation, saga patterns, state management, and determinism constraints. Use when building long-running processes, distributed transactions, or microservice orchestration.
4---
5 
6# Workflow Orchestration Patterns
7 
8Master workflow orchestration architecture with Temporal, covering fundamental design decisions, resilience patterns, and best practices for building reliable distributed systems.
9 
10## Use this skill when
11 
12- Working on workflow orchestration patterns tasks or workflows
13- Needing guidance, best practices, or checklists for workflow orchestration patterns
14 
15## Do not use this skill when
16 
17- The task is unrelated to workflow orchestration patterns
18- You need a different domain or tool outside this scope
19 
20## Instructions
21 
22- Clarify goals, constraints, and required inputs.
23- Apply relevant best practices and validate outcomes.
24- Provide actionable steps and verification.
25- If detailed examples are required, open `resources/implementation-playbook.md`.
26 
27## When to Use Workflow Orchestration
28 
29### Ideal Use Cases (Source: docs.temporal.io)
30 
31- **Multi-step processes** spanning machines/services/databases
32- **Distributed transactions** requiring all-or-nothing semantics
33- **Long-running workflows** (hours to years) with automatic state persistence
34- **Failure recovery** that must resume from last successful step
35- **Business processes**: bookings, orders, campaigns, approvals
36- **Entity lifecycle management**: inventory tracking, account management, cart workflows
37- **Infrastructure automation**: CI/CD pipelines, provisioning, deployments
38- **Human-in-the-loop** systems requiring timeouts and escalations
39 
40### When NOT to Use
41 
42- Simple CRUD operations (use direct API calls)
43- Pure data processing pipelines (use Airflow, batch processing)
44- Stateless request/response (use standard APIs)
45- Real-time streaming (use Kafka, event processors)
46 
47## Critical Design Decision: Workflows vs Activities
48 
49**The Fundamental Rule** (Source: temporal.io/blog/workflow-engine-principles):
50 
51- **Workflows** = Orchestration logic and decision-making
52- **Activities** = External interactions (APIs, databases, network calls)
53 
54### Workflows (Orchestration)
55 
56**Characteristics:**
57 
58- Contain business logic and coordination
59- **MUST be deterministic** (same inputs → same outputs)
60- **Cannot** perform direct external calls
61- State automatically preserved across failures
62- Can run for years despite infrastructure failures
63 
64**Example workflow tasks:**
65 
66- Decide which steps to execute
67- Handle compensation logic
68- Manage timeouts and retries
69- Coordinate child workflows
70 
71### Activities (External Interactions)
72 
73**Characteristics:**
74 
75- Handle all external system interactions
76- Can be non-deterministic (API calls, DB writes)
77- Include built-in timeouts and retry logic
78- **Must be idempotent** (calling N times = calling once)
79- Short-lived (seconds to minutes typically)
80 
81**Example activity tasks:**
82 
83- Call payment gateway API
84- Write to database
85- Send emails or notifications
86- Query external services
87 
88### Design Decision Framework
89 
90```
91Does it touch external systems? → Activity
92Is it orchestration/decision logic? → Workflow
93```
94 
95## Core Workflow Patterns
96 
97### 1. Saga Pattern with Compensation
98 
99**Purpose**: Implement distributed transactions with rollback capability
100 
101**Pattern** (Source: temporal.io/blog/compensating-actions-part-of-a-complete-breakfast-with-sagas):
102 
103```
104For each step:
105  1. Register compensation BEFORE executing
106  2. Execute the step (via activity)
107  3. On failure, run all compensations in reverse order (LIFO)
108```
109 
110**Example: Payment Workflow**
111 
1121. Reserve inventory (compensation: release inventory)
1132. Charge payment (compensation: refund payment)
1143. Fulfill order (compensation: cancel fulfillment)
115 
116**Critical Requirements:**
117 
118- Compensations must be idempotent
119- Register compensation BEFORE executing step
120- Run compensations in reverse order
121- Handle partial failures gracefully
122 
123### 2. Entity Workflows (Actor Model)
124 
125**Purpose**: Long-lived workflow representing single entity instance
126 
127**Pattern** (Source: docs.temporal.io/evaluate/use-cases-design-patterns):
128 
129- One workflow execution = one entity (cart, account, inventory item)
130- Workflow persists for entity lifetime
131- Receives signals for state changes
132- Supports queries for current state
133 
134**Example Use Cases:**
135 
136- Shopping cart (add items, checkout, expiration)
137- Bank account (deposits, withdrawals, balance checks)
138- Product inventory (stock updates, reservations)
139 
140**Benefits:**
141 
142- Encapsulates entity behavior
143- Guarantees consistency per entity
144- Natural event sourcing
145 
146### 3. Fan-Out/Fan-In (Parallel Execution)
147 
148**Purpose**: Execute multiple tasks in parallel, aggregate results
149 
150**Pattern:**
151 
152- Spawn child workflows or parallel activities
153- Wait for all to complete
154- Aggregate results
155- Handle partial failures
156 
157**Scaling Rule** (Source: temporal.io/blog/workflow-engine-principles):
158 
159- Don't scale individual workflows
160- For 1M tasks: spawn 1K child workflows × 1K tasks each
161- Keep each workflow bounded
162 
163### 4. Async Callback Pattern
164 
165**Purpose**: Wait for external event or human approval
166 
167**Pattern:**
168 
169- Workflow sends request and waits for signal
170- External system processes asynchronously
171- Sends signal to resume workflow
172- Workflow continues with response
173 
174**Use Cases:**
175 
176- Human approval workflows
177- Webhook callbacks
178- Long-running external processes
179 
180## State Management and Determinism
181 
182### Automatic State Preservation
183 
184**How Temporal Works** (Source: docs.temporal.io/workflows):
185 
186- Complete program state preserved automatically
187- Event History records every command and event
188- Seamless recovery from crashes
189- Applications restore pre-failure state
190 
191### Determinism Constraints
192 
193**Workflows Execute as State Machines**:
194 
195- Replay behavior must be consistent
196- Same inputs → identical outputs every time
197 
198**Prohibited in Workflows** (Source: docs.temporal.io/workflows):
199 
200- ❌ Threading, locks, synchronization primitives
201- ❌ Random number generation (`random()`)
202- ❌ Global state or static variables
203- ❌ System time (`datetime.now()`)
204- ❌ Direct file I/O or network calls
205- ❌ Non-deterministic libraries
206 
207**Allowed in Workflows**:
208 
209- ✅ `workflow.now()` (deterministic time)
210- ✅ `workflow.random()` (deterministic random)
211- ✅ Pure functions and calculations
212- ✅ Calling activities (non-deterministic operations)
213 
214### Versioning Strategies
215 
216**Challenge**: Changing workflow code while old executions still running
217 
218**Solutions**:
219 
2201. **Versioning API**: Use `workflow.get_version()` for safe changes
2212. **New Workflow Type**: Create new workflow, route new executions to it
2223. **Backward Compatibility**: Ensure old events replay correctly
223 
224## Resilience and Error Handling
225 
226### Retry Policies
227 
228**Default Behavior**: Temporal retries activities forever
229 
230**Configure Retry**:
231 
232- Initial retry interval
233- Backoff coefficient (exponential backoff)
234- Maximum interval (cap retry delay)
235- Maximum attempts (eventually fail)
236 
237**Non-Retryable Errors**:
238 
239- Invalid input (validation failures)
240- Business rule violations
241- Permanent failures (resource not found)
242 
243### Idempotency Requirements
244 
245**Why Critical** (Source: docs.temporal.io/activities):
246 
247- Activities may execute multiple times
248- Network failures trigger retries
249- Duplicate execution must be safe
250 
251**Implementation Strategies**:
252 
253- Idempotency keys (deduplication)
254- Check-then-act with unique constraints
255- Upsert operations instead of insert
256- Track processed request IDs
257 
258### Activity Heartbeats
259 
260**Purpose**: Detect stalled long-running activities
261 
262**Pattern**:
263 
264- Activity sends periodic heartbeat
265- Includes progress information
266- Timeout if no heartbeat received
267- Enables progress-based retry
268 
269## Best Practices
270 
271### Workflow Design
272 
2731. **Keep workflows focused** - Single responsibility per workflow
2742. **Small workflows** - Use child workflows for scalability
2753. **Clear boundaries** - Workflow orchestrates, activities execute
2764. **Test locally** - Use time-skipping test environment
277 
278### Activity Design
279 
2801. **Idempotent operations** - Safe to retry
2812. **Short-lived** - Seconds to minutes, not hours
2823. **Timeout configuration** - Always set timeouts
2834. **Heartbeat for long tasks** - Report progress
2845. **Error handling** - Distinguish retryable vs non-retryable
285 
286### Common Pitfalls
287 
288**Workflow Violations**:
289 
290- Using `datetime.now()` instead of `workflow.now()`
291- Threading or async operations in workflow code
292- Calling external APIs directly from workflow
293- Non-deterministic logic in workflows
294 
295**Activity Mistakes**:
296 
297- Non-idempotent operations (can't handle retries)
298- Missing timeouts (activities run forever)
299- No error classification (retry validation errors)
300- Ignoring payload limits (2MB per argument)
301 
302### Operational Considerations
303 
304**Monitoring**:
305 
306- Workflow execution duration
307- Activity failure rates
308- Retry attempts and backoff
309- Pending workflow counts
310 
311**Scalability**:
312 
313- Horizontal scaling with workers
314- Task queue partitioning
315- Child workflow decomposition
316- Activity batching when appropriate
317 
318## Additional Resources
319 
320**Official Documentation**:
321 
322- Temporal Core Concepts: docs.temporal.io/workflows
323- Workflow Patterns: docs.temporal.io/evaluate/use-cases-design-patterns
324- Best Practices: docs.temporal.io/develop/best-practices
325- Saga Pattern: temporal.io/blog/saga-pattern-made-easy
326 
327**Key Principles**:
328 
3291. Workflows = orchestration, Activities = external calls
3302. Determinism is non-negotiable for workflows
3313. Idempotency is critical for activities
3324. State preservation is automatic
3335. Design for failure and recovery
334