Notifications / Send alerts to PagerDuty
Send alerts to PagerDuty
Open a PagerDuty incident when a monitor goes down and resolve it when the monitor recovers.
The PagerDuty type sends alerts to a PagerDuty service through the Events API v2. PagerDuty then pages your on-call team with its own escalation policies and schedules.
Unlike a chat message, a PagerDuty alert has a lifecycle: Uptime Kotlin opens it when a monitor goes down and resolves it when the monitor comes back, so the incident closes on its own in PagerDuty.
Get an integration key
- In PagerDuty, open Services, then Service Directory, and pick the service that should receive the alerts, or create one.
- In the Integrations tab of the service, add an integration of type Events API V2.
- Copy its Integration Key, a string of 32 characters.
Anyone holding this key can open incidents on the service, so keep it private.
Fill in the form
| Field | Meaning |
|---|---|
| Integration key | The Events API v2 key of the service. When editing, leave it empty to keep the current key. |
| Severity | critical, error, warning or info: the severity of the alert opened when a monitor goes down. critical when left empty. The same for every monitor. |
Use one channel per severity if your monitors do not all deserve the same: a critical channel for production, a warning channel for staging.
What PagerDuty receives
Every monitor has its own alert, identified by a deduplication key such as uptime-kotlin:probe: followed by the ID of the monitor.
| Event in Uptime Kotlin | Event sent to PagerDuty |
|---|---|
| The monitor goes down | trigger: opens an alert, with the severity of the channel. |
| A reminder | trigger with the same key: PagerDuty adds it to the open alert, no new incident. |
| The monitor recovers | resolve: resolves the alert, and the incident if it has no other alert. |
The alert shows:
- Summary:
Service Payments API is FAILURE: HTTP check failed: Request: unexpected HTTP status 503, 1,024 characters at most. - Source: the URL of the monitor, or its name when it has no URL.
- Component: the name of the monitor.
- Custom details:
probe_id,status,message,response_time_msand, for HTTP monitors,status_code.
PagerDuty already escalates and re-notifies while an incident is open, so you can usually leave Resend alert every (s) at 0 on monitors that use this channel.
Acknowledging an incident in Uptime Kotlin does not acknowledge it in PagerDuty, and the other way round: acknowledge it on both sides, or only in PagerDuty.
Check the channel
Choose Test notification: Uptime Kotlin opens an info alert named Uptime Kotlin test notification and resolves it right away. It shows up in the alerts of the service, already resolved. Whether it pages someone depends on the urgency rules of the service.
If the test fails, the error from PagerDuty is shown in the form. Event object is invalid usually means the key was not copied in full, or belongs to another kind of integration than Events API V2.
EU service region
Accounts hosted in the EU service region send their events to another address. Set NOTIFICATIONS_PAGERDUTY_EVENTS_URL to https://events.eu.pagerduty.com/v2/enqueue on the application and on the workers. See Configuration.
