Operate
Incidents and on-call
Learn how CLI notices an outage, proposes a fix, tells the story of what happened and writes the postmortem.
CLI checks every live app once a minute. When one stops answering, it opens an incident, tells the right people, and suggests one action to fix it. Nothing changes until a person (or an agent allowed to) applies it. When the app recovers, CLI writes up what happened.
How to use it: when you get an alert, open the incident: it tells the story in order (what changed just before, what it cost) and offers one fix, such as rolling back or adding memory. Press it to apply.
Each app's What happened tab shows its outages alongside everything else (see /docs/what-happened).
CLI checks every live app once a minute. When one stops answering, it opens an incident, tells the right people, and its on-call agent proposes one action to fix it. Nothing runs until a person, or an agent allowed to, applies it. When the service recovers, CLI writes a postmortem.
Uptime checks
The platform's leader process sends a request to each live app every minute. It hits GET /, or the service's healthcheckPath if you set one. The check expects a 2xx answer.
You can see whether a service is up, why it's down if it is, its uptime over the last 24 hours and 7 days, and its recent incidents.
From your terminal:
cli healthAsk your agent:
"Is the web service up? Any incidents this week?"
Agents use the cli_service_health tool.
With the API:
GET /v1/services/:id/healthSilencing alerts for a service
Some services are expected to go quiet, such as a worker you stop on purpose. Turn their alerts off with healthAlerts:
With the API:
PATCH /v1/services/:id
{"healthAlerts": false}Agents can do the same with cli_update_service and healthAlerts: false.
What happens when a service goes down
- A check fails. One failure doesn't count as an outage.
- After 3 failed checks in a row, the service is down. An incident opens.
- A
service_downevent fires. It goes to every notification channel that listens for it (by default, all of them), and owners and admins get an email. - The on-call agent looks at what happened and proposes one action (see below). A
proposal_createdevent fires. - When checks pass again,
service_upfires, the incident closes, and CLI writes the postmortem.
To see your incidents:
In the dashboard: each incident has its own page at /incidents/:id.
With the API:
GET /v1/projects/:id/incidentsThe on-call agent's proposals
The on-call agent watches failed deploys and outages. For each, it works out the most likely fix and proposes one of four actions:
| Action | When it's proposed |
|---|---|
| Roll back | The service went down within 2 hours of a deploy, and an earlier healthy version exists |
| Restart | The service went down with no recent deploy to blame |
| More memory | The app was killed for running out of memory (for example RUNTIME_OOM, or "out of memory" in the logs) |
| Fix pull request | A GitHub service's deploy failed with a code problem; CLI can open a PR with the smallest fix |
Each proposal has a one-line title (what applying it does), a reason, and the evidence it's based on. There's at most one open proposal per service and action, and an open proposal expires after 24 hours.
Proposals appear in the inbox, in browser push notifications and in your alert channels.
Apply or dismiss a proposal
In the dashboard: open the proposal from the bell and choose to apply or dismiss it.
Ask your agent:
"Are there any open on-call proposals? Tell me what they'd do before applying anything."
With the API:
GET /v1/projects/:id/proposals?status=open
POST /v1/proposals/:id/apply
POST /v1/proposals/:id/dismissApplying needs the developer role. POST /v1/proposals/:id/apply?dryRun=true shows what it would do without running it.
The incident story
Each incident can be told as a story on one screen. It contains:
- What happened, in order: deploys, changes with their reasons, alerts, failovers and recoveries, without repeats.
- The likely cause, in plain words: the deploy just before, running out of memory, a failover, or "nothing changed just before".
- What it cost: what the project spent while the incident lasted.
- One suggested fix:
- out of memory: give the service the next size up and redeploy;
- a deploy just before: roll back to the version before it;
- otherwise: restart, by redeploying what's live.
- The postmortem, once the incident is over.
In the dashboard: open the incident at /incidents/:id.
With the API:
GET /v1/incidents/:id/storyOne-click fix
The Fix button on the incident page applies the suggested fix. It runs as you, through every normal check, so your role, the project's approval rules and spending limits all apply. A resize is followed by a deploy, since a new plan only takes effect on the next deploy. The fix is recorded in the flight recorder with the reason "Fix the … outage: …", so you can undo it (see /docs/flight-recorder).
In the dashboard: click Fix on the incident page.
With the API:
POST /v1/incidents/:id/fixThis needs the developer role. If CLI has no automatic fix to suggest, it answers CONFLICT.
Postmortems
When an incident ends, CLI writes a postmortem and stores it with the incident. It covers:
- what happened, step by step;
- the likely cause;
- what brought the service back, and who did it (or that it recovered on its own);
- how long it was down and what was spent meanwhile;
- suggestions, such as adding a smoke test or canary when a deploy caused the outage.
A line for the postmortem also goes into the flight recorder as a system entry, and the home screen mentions the outage under "since you last looked".
With the API: the postmortem is the postmortem field of GET /v1/incidents/:id/story.
An example: an outage at 3 a.m.
- Health checks fail three times.
service_downfires, the incident opens, and you get an email and a Slack message. - The on-call agent proposes "Give web 1024 MB of memory and deploy again: the new version ran out of memory at 512 MB."
- In the morning you open the incident. The story shows the deploy, the out-of-memory kills and the outage, with the cause and the suggested fix. One click on Fix resizes and redeploys.
- Recovery closes the incident, the postmortem is written, and Home shows it.
Related
- Monitoring and logs for events, channels, logs and status pages.
- Previews and safe releases for smoke tests and canaries that catch bad deploys before they take traffic.
- Hosted agents to run your own agent on a schedule or when a service goes down.