# Monitoring

Inspect instances, retry failed nodes, drain the dead-letter queue, and stop everything at once.

Every workflow run is an instance you can inspect, pause, retry, or cancel. The
console renders all of this; the API is the same surface for scripting it.

```bash
# Instances for one workflow, newest first
curl -G https://api.console.buildbase.app/api/workflows/$WORKFLOW_ID/instances \
  -H "Authorization: $BUILDBASE_TOKEN" \
  --data-urlencode 'filter={"status":"failed"}' \
  --data-urlencode 'sort={"createdAt":-1}'
```

> **Note:**
  These endpoints are governed by `workflows_instances`, not `workflows` — runs
  and definitions are separate grants. Listing and inspecting needs `read`;
  pausing, resuming, retrying and cancelling are all `POST`, so they need
  `create`.

Reading an instance's logs additionally needs `read` on `workflows_logs`, and
its per-node records `read` on `workflows_actions`. Admin-created tokens bypass
every check. See the [admin API](/reference/admin-api) for the full resource
list.



## Instance status

| Status      | Meaning                                         | Terminal |
| ----------- | ----------------------------------------------- | -------- |
| `running`   | Executing, or waiting on a delay or event       | No       |
| `paused`    | Halted, resumable                               | No       |
| `completed` | Finished successfully                           | Yes      |
| `failed`    | A node exhausted its retries with no error path | Yes      |
| `canceled`  | Stopped manually                                | Yes      |

Individual action nodes carry their own finer-grained status: `pending`,
`queued`, `running`, `waiting`, `completed`, `failed`, `skipped`, `canceled`.

A node showing `skipped` inside a `completed` instance is normal — it means a
condition routed around it.

## Inspecting

| Method | Path                                           | Returns                          |
| ------ | ---------------------------------------------- | -------------------------------- |
| `GET`  | `/api/workflows/instances`                     | All instances, paginated         |
| `GET`  | `/api/workflows/instances/counts`              | Counts by status, for dashboards |
| `GET`  | `/api/workflows/:id/instances`                 | Instances of one workflow        |
| `GET`  | `/api/workflows/instances/:instanceId`         | One instance                     |
| `GET`  | `/api/workflows/instances/:instanceId/actions` | Per-node state for the run       |
| `GET`  | `/api/workflows/instances/:instanceId/logs`    | Execution log                    |
| `GET`  | `/api/workflows/:id/test-instances`            | Runs produced by `/test`         |

Instances are also addressable by subject, which is how you answer "what is
running for this customer":

- `/api/workflows/audience/:audienceId/instances`
- `/api/workflows/workspace/:workspaceId/instances`
- `/api/workflows/user/:userId/instances`

## Pausing and resuming

Pausing works at four scopes, and each records **why** it was paused —
`manual`, `workflow`, `org`, `audience`, `workspace`, or `version`.

| Scope               | Endpoint                                           |
| ------------------- | -------------------------------------------------- |
| One instance        | `POST /api/workflows/instances/:instanceId/pause`  |
| One audience member | `POST /api/workflows/audience/:audienceId/pause`   |
| One workspace       | `POST /api/workflows/workspace/:workspaceId/pause` |
| Everything          | `POST /api/workflows/emergency-stop`               |

Each has a matching resume, and `emergency-resume` reverses the org-wide stop.

That `pausedBy` provenance matters: each resume filters on the scope that
paused the instance, so resuming a workspace does not restart instances paused
for a different reason. An instance paused by the org-wide stop stays paused
until `emergency-resume`, even if you resume its workspace.

The response reports what it acted on:

```json
{ "success": true, "data": { "audienceId": "…", "resumedInstances": 42 } }
```

Resume processes instances in batches, re-queueing each one's pending actions
as it goes, so a single call handles the whole scope however many are paused.
`resumedInstances` is the total it acted on.

Resuming a workflow offers two modes:

| Mode       | Behavior                                                        |
| ---------- | --------------------------------------------------------------- |
| `all`      | Resume every paused instance                                    |
| `new_only` | Leave paused instances paused; only new events create instances |

`GET /:id/resume/preview` reports what each mode would affect before you commit.

## Retrying

A failed node can be retried in place without re-running the whole instance:

```bash
curl -X POST \
  https://api.console.buildbase.app/api/workflows/instances/$INSTANCE_ID/retry/$NODE_ID \
  -H "Authorization: $BUILDBASE_TOKEN"
```

Execution resumes from that node with the instance's existing context, so
earlier nodes do not run twice.

To stop a run entirely, `POST /api/workflows/instances/:instanceId/cancel`.

## The dead-letter queue

A trigger event that cannot be processed lands in the DLQ rather than being
dropped. This is the layer **before** an instance exists — an event that never
started a run.

| Method | Path                                              | Purpose                         |
| ------ | ------------------------------------------------- | ------------------------------- |
| `GET`  | `/api/workflows/dead-letter-queue`                | List undelivered trigger events |
| `POST` | `/api/workflows/dead-letter-queue/:eventId/retry` | Re-process one event            |

Trigger events carry their own status — `pending`, `processing`, `processed`,
`failed`, `skipped`. **Only `failed` events are retryable.** The retry flips the
status atomically, so two concurrent retries cannot both start the same event.

A retry on any other status returns `400`:

```json
{ "success": false, "message": "Cannot retry event with status: processed" }
```

An unknown event ID returns `404`. Because the guard is atomic rather than a
read-then-write, that `400` is also what a losing race returns — retrying an
event twice is safe.

Check the DLQ when a workflow "did not fire at all". A missing instance points
at the trigger layer, whereas a `failed` instance points at a node.

## Emergency stop

`POST /api/workflows/emergency-stop` halts every running instance across the
organization. Use it when a workflow is sending wrong mail — it is faster than
pausing workflows one at a time, and `emergency-resume` puts everything back.

Pausing does not cancel. In-flight instances hold their position and continue
from where they stopped, which means a delay that expired during the pause fires
immediately on resume.

## Next Steps

- [Webhook actions](/workflows/webhook-actions) — the most common failure source.
- [Workflows overview](/workflows/overview) — lifecycle, versions, and re-entry.