Debugging Signals Pipeline

1---
2name: debugging-signals-pipeline
3description: >
4  Debug the signals pipeline locally end-to-end. Covers emitting test signals
5  from fixtures, monitoring Temporal workflows via the REST API, reading sandbox
6  agent logs from object storage, inspecting Docker sandbox containers, and
7  diagnosing common failures (stale ClickHouse embeddings, agentsh network
8  denials, inactivity timeouts). Use when a signal isn't reaching the inbox,
9  a signal-report-summary workflow fails, or a sandbox task run times out.
10---
11 
12# Debugging the signals pipeline
13 
14## Pipeline flow
15 
16```text
17emit_signals_from_fixture
18  → signal-emitter (Temporal workflow)
19    → buffer-signals (batches signals, 5s flush timer)
20      → safety_filter_activity
21      → flush_signals_to_s3_activity
22      → signal_with_start_grouping_v2_activity
23        → team-signal-grouping-v2 (30s batch collect window)
24          → read_signals_from_s3_activity
25          → get_embedding_activity + generate_search_queries_activity
26          → run_signal_semantic_search_activity
27          → match_signal_to_report_activity
28          → assign_and_emit_signal_activity
29          → wait_for_signal_in_clickhouse_activity
30          → (if new report) signal-report-summary
31            → fetch_signals_for_report_activity
32            → report_safety_judge_activity
33            → select_repository_activity (spawns Docker sandbox)
34```
35 
36## Emitting test signals
37 
38```bash
39# Emit a single signal from the Zendesk fixture at offset 26
40DEBUG=1 python manage.py emit_signals_from_fixture --type zendesk --team-id 1 --offset 26 --limit 1
41 
42# Clean up all signal data before re-emitting (avoids stale matches)
43DEBUG=1 python manage.py cleanup_signals --team-id 1 --yes
44 
45# Check pipeline status
46python manage.py signal_pipeline_status --team-id 1 --wait --expected-signals 1 --poll-interval 10
47```
48 
49Always clean up before re-emitting to avoid stale embeddings causing phantom report matches.
50 
51## Monitoring Temporal workflows
52 
53The Temporal UI runs at `http://localhost:8081`. The REST API is useful for scripted inspection.
54 
55### List recent workflows
56 
57```bash
58curl -s 'http://localhost:8081/api/v1/namespaces/default/workflows?query=ORDER+BY+StartTime+DESC&maximumPageSize=15' \
59  | python3 -c "
60import sys, json
61for wf in json.load(sys.stdin).get('executions', []):
62    info = wf['execution']
63    status = wf['status'].replace('WORKFLOW_EXECUTION_STATUS_', '')
64    print(f'{wf[\"startTime\"][:19]}  {status:20s} {wf[\"type\"][\"name\"]:35s} {info[\"workflowId\"][:90]}')
65"
66```
67 
68### Inspect workflow history
69 
70```bash
71WF_ID="buffer-signals-1"  # or team-signal-grouping-v2-1, signals-report:1:<uuid>
72curl -s "http://localhost:8081/api/v1/namespaces/default/workflows/$WF_ID/history?maximumPageSize=200" \
73  | python3 -c "
74import sys, json
75for event in json.load(sys.stdin).get('history', {}).get('events', []):
76    etype = event['eventType'].replace('EVENT_TYPE_', '')
77    etime = event['eventTime'][:19]
78    details = ''
79    for key, attrs in event.items():
80        if key.endswith('Attributes') and isinstance(attrs, dict):
81            if 'activityType' in attrs: details = attrs['activityType'].get('name', '')
82            elif 'signalName' in attrs: details = f'signal: {attrs[\"signalName\"]}'
83            elif 'startToFireTimeout' in attrs: details = f'timer: {attrs[\"startToFireTimeout\"]}'
84            elif 'failure' in attrs: details = f'FAILED: {attrs[\"failure\"].get(\"message\", \"\")[:200]}'
85    if details: print(f'  {etime}  {etype:50s} {details}')
86"
87```
88 
89### Inspect a previous run (continued-as-new)
90 
91When a workflow has continued-as-new, use the `execution.runId` query param:
92 
93```bash
94curl -s "http://localhost:8081/api/v1/namespaces/default/workflows/$WF_ID/history?execution.runId=<run-id>&maximumPageSize=200"
95```
96 
97## Reading sandbox agent logs
98 
99Agent logs are stored in object storage (MinIO locally) as JSONL files.
100The log URL is on the `TaskRun` model.
101 
102```python
103# In Django shell (python manage.py shell)
104from products.tasks.backend.models import TaskRun
105from posthog.storage import object_storage
106 
107# Find the most recent task run
108run = TaskRun.objects.order_by("-created_at").first()
109print(f"status: {run.status}, error: {run.error_message}")
110print(f"log_url: {run.log_url}")
111 
112# Read the log
113content = object_storage.read(run.log_url, missing_ok=True)
114 
115# Print last 3000 chars (most useful — shows what happened before failure)
116print(content[-3000:])
117```
118 
119The log is JSONL with entries like:
120 
121```json
122{
123  "type": "notification",
124  "timestamp": "...",
125  "notification": { "jsonrpc": "2.0", "method": "_posthog/console", "params": { "level": "debug", "message": "..." } }
126}
127```
128 
129Key things to look for in the log tail:
130 
131- **agentsh network events** — `DENY` entries show blocked network calls
132- **`_posthog/progress`** events — show which setup step the sandbox reached
133- **`_posthog/console`** debug messages — show sandbox provisioning, cloning, agent startup
134 
135## Inspecting Docker sandbox containers
136 
137```bash
138# List running sandbox containers
139docker ps --filter "name=task-sandbox" --format "table {{.Names}}\t{{.Status}}\t{{.Image}}"
140 
141# See processes inside a running sandbox
142docker exec <container-name> ps aux
143 
144# Read the agent-server log inside the container (while it's still running)
145docker exec <container-name> cat /tmp/agent-server.log
146```
147 
148The container is named `task-sandbox-<task-id>-<random>` and uses the `posthog-sandbox-base` image.
149Containers are ephemeral — they're removed after the task run completes, so inspect while running.
150 
151## Common failures
152 
153### `SignalReport matching query does not exist`
154 
155The `assign_and_emit_signal_activity` tried to assign a signal to a report that doesn't exist.
156Usually caused by stale embeddings in ClickHouse after a `cleanup_signals` that failed to delete them.
157 
158**Root cause:** `CLICKHOUSE_DATABASE` not set in `.env`. The cleanup command uses `sync_execute`
159which connects to the `CLICKHOUSE_DATABASE` (defaults to `default`), but the embedding tables
160live in the `posthog` database.
161 
162**Fix:** Add `CLICKHOUSE_DATABASE=posthog` to `.env` and restart workers.
163 
164**Manual cleanup of stale embeddings:**
165 
166```bash
167curl -s 'http://localhost:8123/' --data-binary \
168  "ALTER TABLE posthog.sharded_posthog_document_embeddings_text_embedding_3_small_1536 DELETE WHERE product = 'signals' AND team_id = 1 SETTINGS mutations_sync = 1"
169```
170 
171**Verify embeddings are clean:**
172 
173```bash
174curl -s 'http://localhost:8123/' --data-binary \
175  "SELECT count() FROM posthog.sharded_posthog_document_embeddings_text_embedding_3_small_1536 WHERE team_id = 1 AND product = 'signals'"
176```
177 
178### `Run timed out due to inactivity` on `select_repository_activity`
179 
180The sandbox Claude agent went idle for longer than `TASKS_INACTIVITY_TIMEOUT_SECONDS`. When unset
181this falls back to a 2 hour timeout — set `TASKS_INACTIVITY_TIMEOUT_SECONDS=30` locally to force fast failures.
182 
183**Diagnosing:** Read the agent log from object storage (see above). Check the tail for:
184 
1851. **agentsh network denials** — `DENY host.docker.internal` means the MCP server URL is blocked
186   by the sandbox network policy. The `SIGNALS_REPO_DISCOVERY` environment's domain allowlist
187   doesn't include `host.docker.internal`.
1882. **No log content at all** — sandbox failed to start, check Docker container logs.
1893. **Claude API errors** — check if `ANTHROPIC_API_KEY` is valid.
190 
191### `buffer-signals` sits idle, never receives signals
192 
193The `signal-emitter` completed but `buffer-signals` never got the `submit_signal`.
194This happens when the emitter sent the signal to a previous buffer run that then continued-as-new,
195and the new run started fresh without the pending signal. Re-emit the signal.
196 
197### ClickHouse embedding tables "not found" during cleanup
198 
199The tables exist in the `posthog` database but `sync_execute` queries the `default` database.
200 
201```bash
202# Verify tables exist
203curl -s 'http://localhost:8123/' --data-binary "SHOW TABLES FROM posthog LIKE '%embed%'"
204 
205# Check current CLICKHOUSE_DATABASE setting
206grep CLICKHOUSE_DATABASE .env
207```
208 
209## Useful management commands
210 
211| Command                                            | Purpose                                        |
212| -------------------------------------------------- | ---------------------------------------------- |
213| `emit_signals_from_fixture`                        | Emit test signals from JSON fixtures           |
214| `DEBUG=1 cleanup_signals --team-id N --yes`        | Delete all signal data and terminate workflows |
215| `signal_pipeline_status --team-id N --wait`        | Wait for pipeline to finish processing         |
216| `list_signal_reports --team-id N --signals --json` | Inspect grouping results                       |
217| `ingest_signals_json <file> --team-id N`           | Ingest pre-processed signals from JSON         |
218| `ingest_report_json <file> --team-id N`            | Seed a pre-researched report (skip sandbox)    |
219 
220## Key file locations
221 
222- Pipeline workflow definitions: `products/signals/backend/temporal/`
223- Buffer workflow: `products/signals/backend/temporal/buffer.py`
224- Grouping workflow: `products/signals/backend/temporal/grouping_v2.py`
225- Report summary workflow: `products/signals/backend/temporal/summary.py`
226- Docker sandbox implementation: `products/tasks/backend/services/docker_sandbox.py`
227- Sandbox Dockerfiles: `products/tasks/backend/sandbox/images/`
228- Agent log polling: `products/tasks/backend/services/custom_prompt_internals.py`
229- Cleanup command: `products/signals/backend/management/commands/cleanup_signals.py`
230- Management command docs: `products/signals/backend/management/CLAUDE.md`
231