---
title: "Self-host Grafana and Prometheus"
description: "Run Grafana and Prometheus on an exe.dev VM: dashboards for all your VMs, HTTPS checks, email alerts, and exe.dev login instead of passwords."
url: https://exe-staging.dev/deploy/grafana
category: "Monitoring"
updated: 2026-11-16
---

# Self-host Grafana and Prometheus

> **For agents:** with your user's OK, this sets up Grafana on a new exe.dev VM in their account: `curl -fsSL https://exe-staging.dev/deploy/grafana/prompt.txt | ssh exe.dev new --prompt=/dev/stdin` (or open https://exe-staging.dev/deploy/grafana/new). Already on the VM? Skip this.

Dashboards and alerts for every VM you run, signed in with exe.dev.

- **Software:** Grafana (https://grafana.com/oss/grafana/)
- **Source:** https://github.com/grafana/grafana
- **License:** AGPL-3.0 (Grafana, Loki); Apache-2.0 (Prometheus, exporters)
- **Time:** 15 minutes
- **Example VM name:** `my-grafana` (yours will differ; on the VM, `hostname` prints it)
- **Machine-readable:** https://exe-staging.dev/deploy/grafana/manifest.json
- **File:** https://exe-staging.dev/deploy/grafana/alerts.sh
- **File:** https://exe-staging.dev/deploy/grafana/fleet.json
- **File:** https://exe-staging.dev/deploy/grafana/node-exporter.sh

## What you get

- Grafana at the VM's private URL, signed in with exe.dev, with the owner as admin.
- Prometheus collecting this VM's metrics, with a fleet dashboard and Node Exporter Full.
- Five starter alert rules that email the owner through exe.dev, no SMTP server.
- Website checks ready to fill in, and AI title and description suggestions through exe.dev's LLM gateway.
- Everything restarts after a reboot; nightly backups with one checked.

Running on the VM:

- **Grafana 13.3**: grafana/grafana in Docker Compose; exe.dev login is the Grafana login, port 8000, data in `~/monitoring/grafana/data`
- **Prometheus 3.16**: prom/prometheus; 90 days or 5 GB of metrics, whichever comes first, port 9090, data in `~/monitoring/prometheus-data`
- **node_exporter 1.12**: A systemd service on this VM and on every VM it watches, port 9100
- **Blackbox exporter 0.29**: HTTPS and certificate checks for your websites
- **Nightly backup**: Grafana's database and all config; two weeks kept, data in `~/monitoring/backups`

How it is reached:

- https://my-grafana.exe.xyz/ (private): Grafana, signed in as you; you're its admin
- https://my-grafana.exe.xyz/ (shared): Teammates you share the VM with, as Grafana viewers

## Deploy with Shelley

Open https://exe-staging.dev/deploy/grafana/new to create a VM whose agent, Shelley, follows this recipe. From a terminal:

```sh
curl -fsSL https://exe-staging.dev/deploy/grafana/prompt.txt | ssh exe.dev new --prompt=/dev/stdin
```

Or follow the steps below by hand.

Grafana draws the dashboards, Prometheus collects the numbers, and a small
exporter on each VM reports CPU, memory, disk, network and systemd units.
This recipe runs all of it on one exe.dev VM that watches itself and your
other VMs, checks your websites, and emails you when something breaks.

An exe.dev VM suits it well. It's a persistent machine with a real disk, so
months of metrics stay put. Grafana is private by default, and your exe.dev
login *is* the Grafana login: no admin password to set or leak. Other VMs
are scraped through exe.dev's [VM-to-VM integration](/docs/integrations-vm-to-vm),
with no exporter exposed to the internet and no keys on either VM. Alerts
go out through exe.dev's [email gateway](/docs/send-email), Slack and
Discord integrations, and AI features use the
[LLM gateway](/docs/integrations-llm), so no SMTP password, webhook URL or
API key sits on the disk. And Shelley is on board to write the next
dashboard or alert for you.

## Open Grafana

Open the URL Shelley gives you, like `https://my-grafana.exe.xyz/`. Sign
in to exe.dev if asked, and you're in Grafana as its admin; there's no
Grafana password.

Open **Dashboards → VMs**:

- **VM fleet** shows every VM Prometheus watches: up or down, CPU, memory,
  disk, failed systemd units, and the websites you check.
- **Node Exporter Full** goes deep on one VM. Pick it in the **Nodename**
  menu at the top.

Right now both show one VM, this one. Next, add another.

## Watch another VM

Each VM you want to watch runs node_exporter on port 9100. exe.dev's proxy
keeps that port private, and a VM-to-VM integration lets this VM, and only
this VM, read it. Say the other VM is called `web1`:

1. **On web1**, install node_exporter:

   ```bash
   curl -fsSL https://exe-staging.dev/deploy/grafana/node-exporter.sh | sh
   ```

   It [downloads node_exporter](https://exe-staging.dev/deploy/grafana/node-exporter.sh)
   from its GitHub release, checks the checksum, and runs it as a systemd
   service. It ends by printing `node_exporter 1.12.1 is running`.

2. **On your computer**, let this VM reach web1's port 9100:

   ```bash
   ssh exe.dev integrations add http-proxy --name node-web1 \
     --target https://web1.exe.xyz:9100/ --peer --attach vm:my-grafana
   ```

3. **On my-grafana**, tell Prometheus about it:

   ```bash
   cd ~/monitoring
   vm=web1
   printf -- '- targets: [node-%s.int.exe.xyz]\n  labels: {vm: %s, __scheme__: https}\n' \
     "$vm" "$vm" > "prometheus/vms/$vm.yml"
   sleep 40; curl -s localhost:9090/api/v1/query --data-urlencode 'query=up{job="node"}' | jq -c '.data.result[] | [.metric.vm, .value[1]]'
   ```

   Each VM should print with `"1"`. Prometheus picks up new files within
   seconds, and web1 appears on **VM fleet**. A `"0"` means the integration
   or node_exporter isn't answering; see [Troubleshooting](#troubleshooting).

The integration injects a key that only works for web1, and it never
lands on either VM. Requests reach web1 marked `X-Exedev-Source-Vm:
my-grafana`. To stop watching web1, delete `prometheus/vms/web1.yml` and run
`ssh exe.dev integrations remove node-web1`.

<details id="tailscale" class="dp-details">
<summary>Already on Tailscale? Scrape over your tailnet instead</summary>

If your VMs are on your [Tailscale](https://tailscale.com/) network,
Prometheus can scrape them directly, without an integration per VM. Join
this VM and each watched VM (Tailscale is preinstalled; exe.dev's
[Tailscale integration](/docs/integrations-tailscale-wif) joins without a
login link):

```bash
sudo systemctl enable --now tailscaled
sudo tailscale up --hostname=$(hostname)
```

Then list the VM by its Tailscale name, over plain HTTP:

```bash
cd ~/monitoring
vm=web1
printf -- '- targets: ["%s:9100"]\n  labels: {vm: %s}\n' "$vm" "$vm" > "prometheus/vms/$vm.yml"
```

The Prometheus container resolves names through the VM, so MagicDNS names
work. Every device on your tailnet can now read port 9100 unless your
[Tailscale access rules](https://tailscale.com/kb/1018/acls) say otherwise.

</details>

## Check your websites

The Blackbox exporter fetches URLs like a visitor would and records whether
they answered, how fast, and when their TLS certificate expires. List your
sites, one URL per line, on my-grafana:

```bash
cd ~/monitoring
cat > prometheus/sites/sites.yml <<'EOF'
- targets:
    - https://example.com/
    - https://app.example.com/healthz
EOF
```

They appear under **Websites** on **VM fleet** within a minute. Two alert
rules watch them: **Site down** after five minutes of failures, and
**Certificate expiring** two weeks ahead.

A private exe.dev app answers outsiders with exe.dev's login page and a
`401`, so it shows as down. To check it, make a VM-to-VM integration for
it as above (`--target https://my-app.exe.xyz/`) and list
`https://<integration>.int.exe.xyz/` instead.
For a public status page and a friendlier editor, run
[Uptime Kuma](/deploy/uptime-kuma) instead or as well.

## Get alerts

Five starter rules live in **Alerting → Alert rules → VM alerts**: VM down,
disk above 85%, a failed systemd unit, site down, and certificate expiring.
They go to the **Email me** contact point, which mails you through exe.dev's
email gateway.

Try it: open **Alerting → Contact points**, edit **Email me**, and press
**Test**. The email arrives from exe.dev within a minute. Edit the rules'
thresholds, or add your own, right in Grafana.

<details id="chat-alerts" class="dp-details">
<summary>Slack, Discord or Telegram, with the secret kept off the VM</summary>

exe.dev holds the webhook URL or bot token and injects it, so Grafana only
ever sees an `int.exe.xyz` address. Set up the integration once, attach it
to my-grafana, then add a contact point in **Alerting → Contact points →
Add contact point**, and pick it in **Alerting → Notification policies**
(edit the default policy, or add a child policy).

**Slack.** On your computer, run `ssh exe.dev integrations setup slack`,
pick a channel, then `ssh exe.dev integrations attach <name> vm:my-grafana`.
In Grafana choose **Slack**, leave **Token** empty, and set **Webhook URL**
to `https://<name>.int.exe.xyz/`. See [Slack integration](/docs/integrations-slack).

**Discord.** Run `ssh exe.dev integrations setup discord`, then attach it
the same way. In Grafana choose **Discord**, set **Webhook URL** to
`https://<name>.int.exe.xyz/`, and turn on **Use Discord's Webhook
Username**: exe.dev rejects messages that set their own name or avatar.
See [Discord integration](/docs/integrations-discord).

**Telegram.** Add the **Telegram Bot API** integration from the
[integration catalog](/docs/integrations-catalog) with your bot's token, and
attach it. Grafana's Telegram contact point wants the token itself, so use
a **Webhook** contact point instead: URL
`https://<name>.int.exe.xyz/sendMessage`, method POST, and under
**Optional Webhook settings → Custom Payload**, this template with your
chat ID in place of `123456789`:

```
{{ coll.Dict "chat_id" "123456789" "text" (tmpl.Exec "default.message" .) | data.ToJSON }}
```

All three are rate limited by exe.dev (Slack and Discord to five messages a
minute), which suits alerts. Keep **Email me** as a fallback.

</details>

## Share it with your team

On your computer:

```bash
ssh exe.dev share add my-grafana teammate@example.com
```

Once they've signed in to exe.dev, they open the same URL and Grafana makes
them an account as a **Viewer**. Promote them in **Administration → Users
and access → Users**. `ssh exe.dev share add my-grafana team` shares with
your whole exe.dev team, and `ssh exe.dev share remove my-grafana
teammate@example.com` locks someone out again. Grafana has no sign-out
button here; signing out of exe.dev signs you out.

The proxy shares every port 3000-9999 with the same people. Prometheus
listens only inside the VM, but node_exporter on 9100 is readable by anyone
the VM is shared with.

## Let AI help

The [LLM plugin](https://grafana.com/grafana/plugins/grafana-llm-app/) is
set up to use exe.dev's LLM gateway, with no API key. While editing a
panel or dashboard, press **✨ Auto generate** beside **Title** or
**Description**, or let it summarize your changes when you save.

For bigger jobs, ask Shelley on the VM. It can read the config in
`~/monitoring`, query Prometheus at `localhost:9090`, write a dashboard
JSON into `grafana/dashboards/`, or add an alert rule with Grafana's API.
For example: *"Add a dashboard for the Postgres on db1 and alert me if it
has more than 80 connections."*

<details id="logs" class="dp-details">
<summary>Add logs with Loki (optional)</summary>

Loki stores logs the way Prometheus stores metrics, and Grafana Alloy ships
them. This adds both to the stack and collects this VM's systemd journal
and Docker container logs; Loki keeps 30 days. On my-grafana:

```bash
cd ~/monitoring
mkdir -p loki-data
cat > loki.yaml <<'EOF'
auth_enabled: false
server:
  http_listen_port: 3100
common:
  path_prefix: /loki
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
schema_config:
  configs:
    - from: 2026-01-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h
limits_config:
  retention_period: 30d
compactor:
  working_directory: /loki/compactor
  retention_enabled: true
  delete_request_store: filesystem
EOF

cat > alloy.alloy <<'EOF'
loki.write "local" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}

loki.relabel "journal" {
  forward_to = []
  rule {
    source_labels = ["__journal__systemd_unit"]
    target_label  = "unit"
  }
}

loki.source.journal "journal" {
  path          = "/var/log/journal"
  relabel_rules = loki.relabel.journal.rules
  labels        = {job = "journal", vm = sys.env("VM")}
  forward_to    = [loki.write.local.receiver]
}

discovery.docker "containers" {
  host = "unix:///var/run/docker.sock"
}

discovery.relabel "containers" {
  targets = discovery.docker.containers.targets
  rule {
    source_labels = ["__meta_docker_container_name"]
    regex         = "/(.*)"
    target_label  = "container"
  }
}

loki.source.docker "containers" {
  host       = "unix:///var/run/docker.sock"
  targets    = discovery.relabel.containers.output
  labels     = {job = "docker", vm = sys.env("VM")}
  forward_to = [loki.write.local.receiver]
}
EOF

cat > compose.loki.yaml <<EOF
services:
  loki:
    image: grafana/loki:3.7.10
    restart: unless-stopped
    user: "1000:1000"
    command: ["-config.file=/etc/loki/loki.yaml"]
    volumes:
      - ./loki.yaml:/etc/loki/loki.yaml:ro
      - ./loki-data:/loki
  alloy:
    image: grafana/alloy:v1.22.0
    restart: unless-stopped
    command: ["run", "--storage.path=/var/lib/alloy/data", "/etc/alloy/config.alloy"]
    environment:
      VM: $(hostname)
    volumes:
      - ./alloy.alloy:/etc/alloy/config.alloy:ro
      - /var/log/journal:/var/log/journal:ro
      - /etc/machine-id:/etc/machine-id:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
    depends_on: [loki]
EOF

cat > grafana/provisioning/datasources/loki.yaml <<'EOF'
apiVersion: 1
datasources:
  - name: Loki
    uid: loki
    type: loki
    access: proxy
    url: http://loki:3100
EOF

echo 'COMPOSE_FILE=compose.yaml:compose.loki.yaml' >> .env
docker compose up -d --wait
docker compose restart grafana
```

In Grafana, open **Drilldown → Logs**, or **Explore** with the Loki data
source and the query `{vm="my-grafana"}`. Alloy reads the Docker socket and
the journal, so treat it as root-equivalent; Loki is reachable only inside
the Compose network. Loki's data in `loki-data` isn't in the nightly backup.

</details>

## Keep your data

Prometheus keeps 90 days of metrics or 5 GB, whichever fills first, in
`~/monitoring/prometheus-data`. Grafana's dashboards, users and alert
settings live in `~/monitoring/grafana/data/grafana.db`. Each night a backup
of Grafana's database and all the config goes to `~/monitoring/backups`.
Metrics history isn't in it: it's big, and it rebuilds itself.

Keep a copy **off the VM**. On your computer:

```bash
scp -r my-grafana.exe.xyz:monitoring/backups .
```

Check disk use on the VM with `du -sh ~/monitoring/*-data` and `df -h /`.
The **Disk almost full** alert fires at 85%.

## The nuts and bolts

Happy clicking Deploy? You can stop here. These are Shelley's setup steps
and your maintenance reference.

<details id="create-the-vm" data-manual-setup class="dp-details">
<summary>Install by hand</summary>

### Create the VM

On your computer (pick another name if `my-grafana` is taken):

```bash
ssh exe.dev new --name=my-grafana
ssh my-grafana.exe.xyz
```

All remaining installation commands run **on the VM**, as `exedev`.

### Install node_exporter and start Docker at boot

This VM watches itself too, so it gets node_exporter like any other.
[The script](https://exe-staging.dev/deploy/grafana/node-exporter.sh) installs the
official release binary as a systemd service on port 9100:

```bash
sudo systemctl enable --now docker.service
curl -fsSL https://exe-staging.dev/deploy/grafana/node-exporter.sh | sh
```

It ends with `node_exporter 1.12.1 is running on <vm>:9100`.

### Write the configuration

Grafana makes the VM's owner its admin, so it needs your exe.dev email.
The reflection integration knows it:

```bash
mkdir -p ~/monitoring && cd ~/monitoring
mkdir -p grafana/data grafana/dashboards grafana/provisioning/datasources \
  grafana/provisioning/dashboards grafana/provisioning/plugins \
  prometheus/vms prometheus/sites prometheus-data backups
owner=$(curl -fsS https://reflection.int.exe.xyz/email | jq -r '.email // empty')
echo "Grafana admin: ${owner:?not found; set owner=you@example.com and run this block again}"
test -e grafana.env || (umask 077 && cat > grafana.env <<EOF
GF_SERVER_ROOT_URL=https://$(hostname).exe.xyz/
GF_SECURITY_ADMIN_USER=$owner
GF_SECURITY_ADMIN_PASSWORD=$(openssl rand -hex 24)
EOF
)
```

`grafana.env` (mode 600) holds a random password for Grafana's built-in
admin. You never type it, because the login form is off, but keep it with
your backups.

The Compose file. Grafana listens on port 8000, the one exe.dev's proxy
serves, and trusts the proxy's `X-ExeDev-Email` header as the user name.
Prometheus listens only on the VM's loopback:

```bash
cat > compose.yaml <<'EOF'
name: monitoring
services:
  grafana:
    image: grafana/grafana:13.3.1
    restart: unless-stopped
    user: "1000:1000"
    ports: ["8000:3000"]
    env_file: grafana.env
    environment:
      GF_AUTH_PROXY_ENABLED: "true"
      GF_AUTH_PROXY_HEADER_NAME: X-ExeDev-Email
      GF_AUTH_PROXY_HEADER_PROPERTY: username
      GF_AUTH_PROXY_HEADERS: Email:X-ExeDev-Email
      GF_AUTH_PROXY_AUTO_SIGN_UP: "true"
      GF_AUTH_DISABLE_LOGIN_FORM: "true"
      GF_AUTH_DISABLE_SIGNOUT_MENU: "true"
      GF_AUTH_BASIC_ENABLED: "false"
      GF_USERS_AUTO_ASSIGN_ORG_ROLE: Viewer
      GF_PLUGINS_PREINSTALL_SYNC: grafana-llm-app@1.2.0
    volumes:
      - ./grafana/data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning:ro
      - ./grafana/dashboards:/etc/grafana/dashboards:ro
    depends_on: [prometheus]
  prometheus:
    image: prom/prometheus:v3.16.0
    restart: unless-stopped
    user: "1000:1000"
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
      - --storage.tsdb.retention.time=90d
      - --storage.tsdb.retention.size=5GB
      - --web.enable-lifecycle
    ports: ["127.0.0.1:9090:9090"]
    extra_hosts: ["host.docker.internal:host-gateway"]
    volumes:
      - ./prometheus:/etc/prometheus:ro
      - ./prometheus-data:/prometheus
  blackbox:
    image: prom/blackbox-exporter:v0.29.1
    restart: unless-stopped
EOF
```

Prometheus scrapes every VM listed in `prometheus/vms/` (this one over the
Docker host gateway, the others through their integrations) and probes every
URL in `prometheus/sites/`. It rereads both directories on its own:

```bash
cat > prometheus/prometheus.yml <<'EOF'
global:
  scrape_interval: 30s
  evaluation_interval: 30s
scrape_configs:
  - job_name: node
    file_sd_configs:
      - files: [/etc/prometheus/vms/*.yml]
  - job_name: sites
    scrape_interval: 60s
    metrics_path: /probe
    params:
      module: [http_2xx]
    file_sd_configs:
      - files: [/etc/prometheus/sites/*.yml]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox:9115
  - job_name: prometheus
    static_configs:
      - targets: [localhost:9090]
EOF
printf -- '- targets: [host.docker.internal:9100]\n  labels: {vm: %s}\n' \
  "$(hostname)" > "prometheus/vms/$(hostname).yml"
```

Grafana's data source, dashboards, and LLM plugin. The dashboards are the
community's [Node Exporter Full](https://github.com/rfmoz/grafana-dashboards)
(Apache-2.0, pinned to a commit) and this recipe's
[VM fleet](https://exe-staging.dev/deploy/grafana/fleet.json). The LLM plugin
talks to exe.dev's gateway as an OpenAI-compatible provider, with no key:

```bash
cat > grafana/provisioning/datasources/prometheus.yaml <<'EOF'
apiVersion: 1
datasources:
  - name: Prometheus
    uid: prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    jsonData:
      timeInterval: 30s
EOF

cat > grafana/provisioning/dashboards/files.yaml <<'EOF'
apiVersion: 1
providers:
  - name: files
    folder: VMs
    type: file
    options:
      path: /etc/grafana/dashboards
EOF

cat > grafana/provisioning/plugins/llm.yaml <<'EOF'
apiVersion: 1
apps:
  - type: grafana-llm-app
    jsonData:
      provider: custom
      openAI:
        url: https://llm.int.exe.xyz
        apiPath: /openai/v1
      models:
        default: base
        mapping:
          base: gpt-5.4-mini
          large: gpt-5.5
EOF

curl -fsSL -o grafana/dashboards/node-exporter-full.json \
  https://raw.githubusercontent.com/rfmoz/grafana-dashboards/b07d8a90d8fadd112e9ac841b777ee863cda850b/prometheus/node-exporter-full.json
curl -fsSL -o grafana/dashboards/fleet.json https://exe-staging.dev/deploy/grafana/fleet.json
jq -r .title grafana/dashboards/*.json
```

`jq` prints `Node Exporter Full` and `VM fleet`.

### Start the stack

```bash
docker compose up -d --wait
docker compose ps
curl -fsS localhost:8000/api/health | jq -r .database
curl -fsS -H "X-ExeDev-Email: $owner" localhost:8000/api/user | jq -r '.login, .isGrafanaAdmin'
sleep 35; curl -s localhost:9090/api/v1/query --data-urlencode 'query=up{job="node"}' | jq -c '.data.result[] | [.metric.vm, .value[1]]'
```

All three services should be **Up**. The checks print `ok`, then your email
and `true` (you're Grafana's admin), then this VM with `"1"`.

### Add the starter alerts

[alerts.sh](https://exe-staging.dev/deploy/grafana/alerts.sh) uses Grafana's API to
create the **Email me** contact point (exe.dev's email gateway, to the
owner), point the notification policy at it, and add the five rules. They
stay editable in Grafana's UI:

```bash
curl -fsSL -o alerts.sh https://exe-staging.dev/deploy/grafana/alerts.sh
chmod +x alerts.sh && ./alerts.sh
```

It prints `Alerts ready: 5 rules in 'VM alerts', sent to <your email> by email.`

### Set up nightly backups

This copies Grafana's SQLite database safely while it runs, checks the copy,
and archives it with the config. Two weeks are kept:

```bash
cat > backup.sh <<'EOF'
#!/bin/sh
set -eu
umask 077
cd "$(dirname "$0")"
mkdir -p backups
work=$(mktemp -d)
trap 'rm -rf "$work"' EXIT
sqlite3 grafana/data/grafana.db ".backup '$work/grafana.db'"
test "$(sqlite3 "$work/grafana.db" 'PRAGMA integrity_check;')" = ok
out="backups/monitoring-$(date +%Y%m%d-%H%M%S).tar.gz"
tar -czf "$out.tmp" compose.yaml grafana.env alerts.sh backup.sh \
  prometheus grafana/provisioning grafana/dashboards -C "$work" grafana.db
mv "$out.tmp" "$out"
find backups -name 'monitoring-*.tar.gz' -mtime +14 -delete
EOF
chmod +x backup.sh

sudo tee /etc/systemd/system/monitoring-backup.service >/dev/null <<'EOF'
[Unit]
Description=Back up Grafana and the monitoring config

[Service]
Type=oneshot
User=exedev
ExecStart=/home/exedev/monitoring/backup.sh
EOF
sudo tee /etc/systemd/system/monitoring-backup.timer >/dev/null <<'EOF'
[Unit]
Description=Back up Grafana nightly

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now monitoring-backup.timer
sudo systemctl start monitoring-backup.service
systemctl list-timers monitoring-backup.timer
tar -tzf backups/monitoring-*.tar.gz | grep -c .
```

The timer lists its next run, and `tar` counts the files in the first
archive. Open `https://<vm>.exe.xyz/` and try the walkthrough above.

</details>

<details id="back-up-and-restore" class="dp-details">
<summary>Back up and restore</summary>

Take an extra backup any time, on the VM:

```bash
cd ~/monitoring && ./backup.sh && ls -lh backups | tail -3
```

### Restore

On a new VM, run **Install by hand** first, then copy your archive into
`~/monitoring/backups`. On the VM, substitute the archive's name; the
current database and config are set aside, not deleted:

```bash
cd ~/monitoring
backup=backups/monitoring-YYYYMMDD-HHMMSS.tar.gz
old="before-restore-$(date +%s)"
tar -tzf "$backup" >/dev/null &&
docker compose stop grafana &&
mkdir "$old" && cp -a grafana.env compose.yaml prometheus grafana/provisioning grafana/dashboards "$old"/ &&
mv grafana/data/grafana.db "$old"/ && rm -f grafana/data/grafana.db-wal grafana/data/grafana.db-shm &&
tar -xzf "$backup" --exclude=grafana.db &&
tar -xzf "$backup" -O grafana.db > grafana/data/grafana.db &&
docker compose up -d --wait
```

On a VM with a different name, fix `GF_SERVER_ROOT_URL` in `grafana.env`
and rename this VM's file in `prometheus/vms/`, then run
`docker compose up -d`. The VM-to-VM integrations must be attached to the
new VM (`ssh exe.dev integrations attach node-web1 vm:<new-vm>`).

</details>

<details id="update" class="dp-details">
<summary>Update</summary>

The images are pinned to the versions this recipe was tested with. To
update, read the release notes for
[Grafana](https://github.com/grafana/grafana/releases),
[Prometheus](https://github.com/prometheus/prometheus/releases) and the
[Blackbox exporter](https://github.com/prometheus/blackbox_exporter/releases),
change the tags in `compose.yaml`, then on the VM:

```bash
cd ~/monitoring
./backup.sh && docker compose pull && docker compose up -d --wait
```

For node_exporter, rerun the script with the version you want, on each VM:
`curl -fsSL https://exe-staging.dev/deploy/grafana/node-exporter.sh | NODE_EXPORTER_VERSION=1.12.1 sh`.

</details>

<details id="troubleshooting" class="dp-details">
<summary>Something not working?</summary>

Start with the logs and Prometheus's view of its targets. On the VM:

```bash
cd ~/monitoring
docker compose ps
docker compose logs --tail=50 grafana prometheus
curl -s localhost:9090/api/v1/targets | jq -r '.data.activeTargets[] | [.labels.vm // .labels.instance, .health, .lastError] | @tsv'
```

- **A VM shows `down` with `401` or `307`.** The integration isn't
  attached to this VM, or its target is wrong. Check
  `curl -s https://reflection.int.exe.xyz/integrations | jq -r '.integrations[].name'`
  and the target's port (`:9100`).
- **A VM shows `down` with `502` or `connection refused`.** node_exporter
  isn't running there: `systemctl status node_exporter` on that VM.
- **Down after a reboot?** Run `sudo systemctl enable --now docker.service`.
- **Grafana shows a login form or "invalid username or password".** You
  reached it without exe.dev's proxy, so there's no `X-ExeDev-Email`.
  Use `https://<vm>.exe.xyz/`, not an SSH tunnel.
- **You're a Viewer, not the admin.** Grafana's admin is whoever
  `GF_SECURITY_ADMIN_USER` in `grafana.env` named at its first start.
  Act as that account from the VM to promote yourself:

  ```bash
  cd ~/monitoring
  me=you@example.com
  admin=$(sed -n 's/^GF_SECURITY_ADMIN_USER=//p' grafana.env)
  id=$(curl -s -H "X-ExeDev-Email: $admin" "localhost:8000/api/users/lookup?loginOrEmail=$me" | jq .id)
  curl -s -X PUT -H "X-ExeDev-Email: $admin" -H 'Content-Type: application/json' \
    -d '{"isGrafanaAdmin":true}' localhost:8000/api/admin/users/$id/permissions
  curl -s -X PATCH -H "X-ExeDev-Email: $admin" -H 'Content-Type: application/json' \
    -d '{"role":"Admin"}' localhost:8000/api/org/users/$id
  ```
- **No alert email?** Check spam, then **Alerting → Contact points** for
  the error next to **Email me**, and `docker compose logs grafana | grep -i webhook`.
- **✨ Auto generate is missing.** `docker compose logs grafana | grep -i llm`;
  the plugin installs when Grafana starts, so it needs the internet then.

</details>

## Frequently asked questions

### How much does it cost to self-host Grafana on exe.dev?

Grafana, Prometheus and the exporters are free and open source. You pay for your [exe.dev plan](https://exe.dev/pricing); one VM can watch the rest of your VMs. AI suggestions use your exe.dev [LLM allocation](/docs/integrations-llm).

### Can it monitor servers that aren't on exe.dev?

Yes. Put them on your Tailscale network with this VM, or give them an HTTPS address Prometheus can scrape, and list them in a file under `prometheus/vms/` the same way. Websites anywhere can be checked with the Blackbox exporter.

### Why not just use Grafana Cloud?

Grafana Cloud's free tier is generous, but your VMs' metrics then leave exe.dev, each VM needs an agent with a Cloud API key, and retention and series counts are capped. Here the data stays on a disk you own, sign-in is your exe.dev login, and VMs are scraped without any key on them.

### How much disk does Prometheus need?

About 2,000 series per VM at a 30-second scrape, which is roughly 10 MB a day per VM after compression. The 5 GB cap holds 90 days for five to six VMs; raise `--storage.tsdb.retention.size` or [resize the disk](/docs/disk-usage) for more.

### Is the auth proxy login safe?

Grafana believes the `X-ExeDev-Email` header on any request that reaches port 8000. Only exe.dev's proxy and programs on the VM itself can reach that port, and the proxy replaces whatever header a visitor sends, so outsiders can't forge it. Anyone who can run commands on the VM can act as any Grafana user, the same as they could read its disk.

### Who watches the watcher?

Grafana can't alert you while its own VM is down. Watch `https://my-grafana.exe.xyz/api/health` from somewhere else, such as an [Uptime Kuma](/deploy/uptime-kuma) VM or an outside service.


## Related

- [Self-host Uptime Kuma](https://exe-staging.dev/deploy/uptime-kuma.md): A little lookout for your websites. Green bars encouraged.
- [Deploy a Docker Compose app from GitHub](https://exe-staging.dev/deploy/docker-compose.md): Your compose.yaml on a VM of its own. git push to deploy.
- [Self-host PostgreSQL](https://exe-staging.dev/deploy/postgres.md): Your own Postgres on a real disk, with backups you've actually restored.
- [Self-host Open WebUI](https://exe-staging.dev/deploy/open-webui.md): A private ChatGPT for you and your team. No API keys.
- [Dev, Test, Prod: Choose One, Two, or Three](https://exe-staging.dev/deploy/dev-test-prod.md): You can build, test, and deploy on exe.dev.
- [Give agents API access without giving them the keys](https://exe-staging.dev/deploy/agent-secrets.md): Your agents call your APIs. The keys stay with exe.dev, outside the VM.
