Prometheus metrics

Uptime Kotlin watches your services, but something has to watch Uptime Kotlin. The application and the workers publish Prometheus metrics: checks run, queues, worker heartbeats, notifications, memory. The repository ships a Grafana dashboard and alerting rules built on them.

Where the metrics are served

The metrics are served at /q/metrics on a separate management port, never on the port of the web interface:

The endpoint has no authentication. Do not publish these ports on the internet: let Prometheus reach them through the Docker network, as below, rather than with a ports entry.

Scrape the metrics with Prometheus

Run Prometheus in the same Compose project as Uptime Kotlin, so it reaches the containers by name:

prometheus:
  image: prom/prometheus:v3.13.4
  container_name: uptime_kotlin_prometheus
  restart: unless-stopped
  volumes:
    - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
    - ./alerts.yml:/etc/prometheus/alerts.yml:ro
    - prometheus-data:/prometheus

Add prometheus-data: under the top-level volumes key, then create prometheus.yml next to compose.yaml:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - /etc/prometheus/alerts.yml

scrape_configs:
  - job_name: uptime-kotlin-api
    metrics_path: /q/metrics
    static_configs:
      - targets: ['uptime-kotlin:9000']
        labels:
          role: api

  - job_name: uptime-kotlin-worker
    metrics_path: /q/metrics
    static_configs:
      - targets: ['uptime-kotlin-worker:9002']
        labels:
          role: worker

Keep the job names starting with uptime-kotlin and the role label: the dashboard and the alerts select the targets with job=~"uptime-kotlin.*". Without workers, drop the second job. In cluster mode, list every worker under targets, and every replica of the application under the first job.

Import the Grafana dashboard

The dashboard lives in the repository, in deploy/monitoring/grafana/dashboards/uptime-kotlin.json. In Grafana, open Dashboards, then New and Import, upload the file and pick your Prometheus data source.

It is organised in rows:

• Overview: the firing alerts and a status tile for each concern, whether the checks run on schedule, the check queue lag, the workers, the dead-lettered tasks and the failed notifications.
• Probe Checks: checks per minute by result and by protocol, check duration, probes by state.
• Workers and queues: heartbeat age of every worker, worker saturation, queue lag and content.
• Instances · CPU and memory: one line per process, with its memory against its container limit.
• Runtime details, Notifications and API: heap, threads, delivery attempts by channel, request rate and latency of the API.

The Region, Instance and Protocol filters at the top narrow the panels. Two settings of the dashboard are constants: Prometheus URL and Uptime Kotlin URL, used by the links to the alerts and to the application. Set them to your own addresses in the dashboard settings, under Variables.

Alerting rules

deploy/monitoring/alerts.yml holds ready-made rules. Mount it in Prometheus as above: the alerts appear in the Alerts tab of Prometheus and on the dashboard. To receive them by e-mail, Slack or PagerDuty, add an Alertmanager.

When checks lag, the workers are often saturated: compare uptime_probe_worker_active with uptime_probe_worker_concurrency, then raise SCHEDULER_WORKER_CONCURRENCY (see Configuration) or add a worker.

A worker stopped on purpose keeps firing UptimeKotlinWorkerSilent until its heartbeat is purged, an hour later.

Metrics reference

Every instance reads the queue, probe and worker metrics from PostgreSQL, so they have the same value on every instance: aggregate them with max(), never sum(). The other metrics count what each process does itself and add up.

The region label is the SCHEDULER_WORKER_NAME of the process, and worker adds its host name. The queue and probe gauges are refreshed every 30 seconds, the heartbeats every 15 seconds.

The processes also publish the standard metrics of the runtime and of the HTTP server, such as http_server_requests_seconds for the API, which the dashboard uses.

Try it on your machine

A clone of the repository starts Prometheus and Grafana already wired together, for development:

docker compose --profile monitoring up -d

Prometheus answers on http://localhost:9090 and Grafana on http://localhost:3000 (admin / admin), with the Uptime Kotlin dashboard as home page. This setup scrapes an application started on the host, on ports 9000 and 9002: for a deployment, use the configuration above.