Self-host Grafana and Prometheus
Dashboards and alerts for every VM you run, signed in with exe.dev.
Shelley handles the setup. About 15 minutes.
Prefer a terminal?
curl -fsSL https://exe-staging.dev/deploy/grafana/prompt.txt | ssh exe.dev new --prompt=/dev/stdinGrafana draws the dashboards, Prometheus collects the numbers, and a small exporter on each VM reports CPU, memory, disk, network and systemd units. This recipe runs all of it on one exe.dev VM that watches itself and your other VMs, checks your websites, and emails you when something breaks.
An exe.dev VM suits it well. It's a persistent machine with a real disk, so months of metrics stay put. Grafana is private by default, and your exe.dev login is the Grafana login: no admin password to set or leak. Other VMs are scraped through exe.dev's VM-to-VM integration, with no exporter exposed to the internet and no keys on either VM. Alerts go out through exe.dev's email gateway, Slack and Discord integrations, and AI features use the LLM gateway, so no SMTP password, webhook URL or API key sits on the disk. And Shelley is on board to write the next dashboard or alert for you.
What you get#
- Grafana at the VM's private URL, signed in with exe.dev, with the owner as admin.
- Prometheus collecting this VM's metrics, with a fleet dashboard and Node Exporter Full.
- Five starter alert rules that email the owner through exe.dev, no SMTP server.
- Website checks ready to fill in, and AI title and description suggestions through exe.dev's LLM gateway.
- Everything restarts after a reboot; nightly backups with one checked.
What's running, and where
https://my-grafana.exe.xyz/
Grafana, signed in as you; you're its admin
https://my-grafana.exe.xyz/
Teammates you share the VM with, as Grafana viewers
Deploy with Shelley#
Click Deploy and Shelley will set up Grafana on a new VM, then check that it works. No config-file wrangling required.
Prefer a terminal?
curl -fsSL \
https://exe-staging.dev/deploy/grafana/prompt.txt \
| ssh exe.dev new --prompt=/dev/stdinWhat Shelley is told
Set up Grafana on this VM by following the exe.dev recipe at https://exe-staging.dev/deploy/grafana.md from top to bottom.
The recipe's example VM name is my-grafana. This VM's name is the output of `hostname`: use it wherever the recipe writes my-grafana. Skip the step that creates the VM; you are on it.
You are done when all of these are true. Check each one yourself:
- Grafana at the VM's private URL, signed in with exe.dev, with the owner as admin.
- Prometheus collecting this VM's metrics, with a fleet dashboard and Node Exporter Full.
- Five starter alert rules that email the owner through exe.dev, no SMTP server.
- Website checks ready to fill in, and AI title and description suggestions through exe.dev's LLM gateway.
- Everything restarts after a reboot; nightly backups with one checked.
Read the whole recipe, including the closed details. For initial setup,
follow "Install by hand" from top to bottom, skipping VM creation and
SSH. The walkthrough at the top is for the owner afterwards: don't add
other VMs or websites, create integrations, or set up Loki or Tailscale.
Do not run Restore or Update.
Work as exedev in ~/monitoring. Never overwrite an existing grafana.env.
The owner's email comes from the reflection integration; if it prints
nothing, ask the owner for their exe.dev email and use that.
Don't send a test alert or email; the owner will. Leave the VM private
and do not reboot. Verify docker.service and node_exporter are enabled,
every container restarts unless-stopped, and the checks in each step pass.
Finish with https://<vm>.exe.xyz/ and the three steps from "Watch
another VM" with this VM's name filled in.
Finish with the URLs I should open.
Also at /deploy/grafana/prompt.txt. The recipe it follows is /deploy/grafana.md.
Open Grafana#
Open the URL Shelley gives you, like https://my-grafana.exe.xyz/. Sign
in to exe.dev if asked, and you're in Grafana as its admin; there's no
Grafana password.
Open Dashboards → VMs:
- VM fleet shows every VM Prometheus watches: up or down, CPU, memory, disk, failed systemd units, and the websites you check.
- Node Exporter Full goes deep on one VM. Pick it in the Nodename menu at the top.
Right now both show one VM, this one. Next, add another.
Watch another VM#
Each VM you want to watch runs node_exporter on port 9100. exe.dev's proxy
keeps that port private, and a VM-to-VM integration lets this VM, and only
this VM, read it. Say the other VM is called web1:
-
On web1, install node_exporter:
curl -fsSL https://exe-staging.dev/deploy/grafana/node-exporter.sh | shIt downloads node_exporter from its GitHub release, checks the checksum, and runs it as a systemd service. It ends by printing
node_exporter 1.12.1 is running. -
On your computer, let this VM reach web1's port 9100:
ssh exe.dev integrations add http-proxy --name node-web1 \ --target https://web1.exe.xyz:9100/ --peer --attach vm:my-grafana -
On my-grafana, tell Prometheus about it:
cd ~/monitoring vm=web1 printf -- '- targets: [node-%s.int.exe.xyz]\n labels: {vm: %s, __scheme__: https}\n' \ "$vm" "$vm" > "prometheus/vms/$vm.yml" sleep 40; curl -s localhost:9090/api/v1/query --data-urlencode 'query=up{job="node"}' | jq -c '.data.result[] | [.metric.vm, .value[1]]'Each VM should print with
"1". Prometheus picks up new files within seconds, and web1 appears on VM fleet. A"0"means the integration or node_exporter isn't answering; see Troubleshooting.
The integration injects a key that only works for web1, and it never
lands on either VM. Requests reach web1 marked X-Exedev-Source-Vm: my-grafana. To stop watching web1, delete prometheus/vms/web1.yml and run
ssh exe.dev integrations remove node-web1.
Already on Tailscale? Scrape over your tailnet instead
If your VMs are on your Tailscale network, Prometheus can scrape them directly, without an integration per VM. Join this VM and each watched VM (Tailscale is preinstalled; exe.dev's Tailscale integration joins without a login link):
sudo systemctl enable --now tailscaled
sudo tailscale up --hostname=$(hostname)
Then list the VM by its Tailscale name, over plain HTTP:
cd ~/monitoring
vm=web1
printf -- '- targets: ["%s:9100"]\n labels: {vm: %s}\n' "$vm" "$vm" > "prometheus/vms/$vm.yml"
The Prometheus container resolves names through the VM, so MagicDNS names work. Every device on your tailnet can now read port 9100 unless your Tailscale access rules say otherwise.
Check your websites#
The Blackbox exporter fetches URLs like a visitor would and records whether they answered, how fast, and when their TLS certificate expires. List your sites, one URL per line, on my-grafana:
cd ~/monitoring
cat > prometheus/sites/sites.yml <<'EOF'
- targets:
- https://example.com/
- https://app.example.com/healthz
EOF
They appear under Websites on VM fleet within a minute. Two alert rules watch them: Site down after five minutes of failures, and Certificate expiring two weeks ahead.
A private exe.dev app answers outsiders with exe.dev's login page and a
401, so it shows as down. To check it, make a VM-to-VM integration for
it as above (--target https://my-app.exe.xyz/) and list
https://<integration>.int.exe.xyz/ instead.
For a public status page and a friendlier editor, run
Uptime Kuma instead or as well.
Get alerts#
Five starter rules live in Alerting → Alert rules → VM alerts: VM down, disk above 85%, a failed systemd unit, site down, and certificate expiring. They go to the Email me contact point, which mails you through exe.dev's email gateway.
Try it: open Alerting → Contact points, edit Email me, and press Test. The email arrives from exe.dev within a minute. Edit the rules' thresholds, or add your own, right in Grafana.
Slack, Discord or Telegram, with the secret kept off the VM
exe.dev holds the webhook URL or bot token and injects it, so Grafana only
ever sees an int.exe.xyz address. Set up the integration once, attach it
to my-grafana, then add a contact point in Alerting → Contact points →
Add contact point, and pick it in Alerting → Notification policies
(edit the default policy, or add a child policy).
Slack. On your computer, run ssh exe.dev integrations setup slack,
pick a channel, then ssh exe.dev integrations attach <name> vm:my-grafana.
In Grafana choose Slack, leave Token empty, and set Webhook URL
to https://<name>.int.exe.xyz/. See Slack integration.
Discord. Run ssh exe.dev integrations setup discord, then attach it
the same way. In Grafana choose Discord, set Webhook URL to
https://<name>.int.exe.xyz/, and turn on Use Discord's Webhook
Username: exe.dev rejects messages that set their own name or avatar.
See Discord integration.
Telegram. Add the Telegram Bot API integration from the
integration catalog with your bot's token, and
attach it. Grafana's Telegram contact point wants the token itself, so use
a Webhook contact point instead: URL
https://<name>.int.exe.xyz/sendMessage, method POST, and under
Optional Webhook settings → Custom Payload, this template with your
chat ID in place of 123456789:
{{ coll.Dict "chat_id" "123456789" "text" (tmpl.Exec "default.message" .) | data.ToJSON }}
All three are rate limited by exe.dev (Slack and Discord to five messages a minute), which suits alerts. Keep Email me as a fallback.
Share it with your team#
On your computer:
ssh exe.dev share add my-grafana teammate@example.com
Once they've signed in to exe.dev, they open the same URL and Grafana makes
them an account as a Viewer. Promote them in Administration → Users
and access → Users. ssh exe.dev share add my-grafana team shares with
your whole exe.dev team, and ssh exe.dev share remove my-grafana teammate@example.com locks someone out again. Grafana has no sign-out
button here; signing out of exe.dev signs you out.
The proxy shares every port 3000-9999 with the same people. Prometheus listens only inside the VM, but node_exporter on 9100 is readable by anyone the VM is shared with.
Let AI help#
The LLM plugin is set up to use exe.dev's LLM gateway, with no API key. While editing a panel or dashboard, press ✨ Auto generate beside Title or Description, or let it summarize your changes when you save.
For bigger jobs, ask Shelley on the VM. It can read the config in
~/monitoring, query Prometheus at localhost:9090, write a dashboard
JSON into grafana/dashboards/, or add an alert rule with Grafana's API.
For example: "Add a dashboard for the Postgres on db1 and alert me if it
has more than 80 connections."
Add logs with Loki (optional)
Loki stores logs the way Prometheus stores metrics, and Grafana Alloy ships them. This adds both to the stack and collects this VM's systemd journal and Docker container logs; Loki keeps 30 days. On my-grafana:
cd ~/monitoring
mkdir -p loki-data
cat > loki.yaml <<'EOF'
auth_enabled: false
server:
http_listen_port: 3100
common:
path_prefix: /loki
replication_factor: 1
ring:
kvstore:
store: inmemory
storage:
filesystem:
chunks_directory: /loki/chunks
rules_directory: /loki/rules
schema_config:
configs:
- from: 2026-01-01
store: tsdb
object_store: filesystem
schema: v13
index:
prefix: index_
period: 24h
limits_config:
retention_period: 30d
compactor:
working_directory: /loki/compactor
retention_enabled: true
delete_request_store: filesystem
EOF
cat > alloy.alloy <<'EOF'
loki.write "local" {
endpoint {
url = "http://loki:3100/loki/api/v1/push"
}
}
loki.relabel "journal" {
forward_to = []
rule {
source_labels = ["__journal__systemd_unit"]
target_label = "unit"
}
}
loki.source.journal "journal" {
path = "/var/log/journal"
relabel_rules = loki.relabel.journal.rules
labels = {job = "journal", vm = sys.env("VM")}
forward_to = [loki.write.local.receiver]
}
discovery.docker "containers" {
host = "unix:///var/run/docker.sock"
}
discovery.relabel "containers" {
targets = discovery.docker.containers.targets
rule {
source_labels = ["__meta_docker_container_name"]
regex = "/(.*)"
target_label = "container"
}
}
loki.source.docker "containers" {
host = "unix:///var/run/docker.sock"
targets = discovery.relabel.containers.output
labels = {job = "docker", vm = sys.env("VM")}
forward_to = [loki.write.local.receiver]
}
EOF
cat > compose.loki.yaml <<EOF
services:
loki:
image: grafana/loki:3.7.10
restart: unless-stopped
user: "1000:1000"
command: ["-config.file=/etc/loki/loki.yaml"]
volumes:
- ./loki.yaml:/etc/loki/loki.yaml:ro
- ./loki-data:/loki
alloy:
image: grafana/alloy:v1.22.0
restart: unless-stopped
command: ["run", "--storage.path=/var/lib/alloy/data", "/etc/alloy/config.alloy"]
environment:
VM: $(hostname)
volumes:
- ./alloy.alloy:/etc/alloy/config.alloy:ro
- /var/log/journal:/var/log/journal:ro
- /etc/machine-id:/etc/machine-id:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
depends_on: [loki]
EOF
cat > grafana/provisioning/datasources/loki.yaml <<'EOF'
apiVersion: 1
datasources:
- name: Loki
uid: loki
type: loki
access: proxy
url: http://loki:3100
EOF
echo 'COMPOSE_FILE=compose.yaml:compose.loki.yaml' >> .env
docker compose up -d --wait
docker compose restart grafana
In Grafana, open Drilldown → Logs, or Explore with the Loki data
source and the query {vm="my-grafana"}. Alloy reads the Docker socket and
the journal, so treat it as root-equivalent; Loki is reachable only inside
the Compose network. Loki's data in loki-data isn't in the nightly backup.
Keep your data#
Prometheus keeps 90 days of metrics or 5 GB, whichever fills first, in
~/monitoring/prometheus-data. Grafana's dashboards, users and alert
settings live in ~/monitoring/grafana/data/grafana.db. Each night a backup
of Grafana's database and all the config goes to ~/monitoring/backups.
Metrics history isn't in it: it's big, and it rebuilds itself.
Keep a copy off the VM. On your computer:
scp -r my-grafana.exe.xyz:monitoring/backups .
Check disk use on the VM with du -sh ~/monitoring/*-data and df -h /.
The Disk almost full alert fires at 85%.
The nuts and bolts#
Happy clicking Deploy? You can stop here. These are Shelley's setup steps and your maintenance reference.
Install by hand
Create the VM#
On your computer (pick another name if my-grafana is taken):
ssh exe.dev new --name=my-grafana
ssh my-grafana.exe.xyz
All remaining installation commands run on the VM, as exedev.
Install node_exporter and start Docker at boot#
This VM watches itself too, so it gets node_exporter like any other. The script installs the official release binary as a systemd service on port 9100:
sudo systemctl enable --now docker.service
curl -fsSL https://exe-staging.dev/deploy/grafana/node-exporter.sh | sh
It ends with node_exporter 1.12.1 is running on <vm>:9100.
Write the configuration#
Grafana makes the VM's owner its admin, so it needs your exe.dev email. The reflection integration knows it:
mkdir -p ~/monitoring && cd ~/monitoring
mkdir -p grafana/data grafana/dashboards grafana/provisioning/datasources \
grafana/provisioning/dashboards grafana/provisioning/plugins \
prometheus/vms prometheus/sites prometheus-data backups
owner=$(curl -fsS https://reflection.int.exe.xyz/email | jq -r '.email // empty')
echo "Grafana admin: ${owner:?not found; set owner=you@example.com and run this block again}"
test -e grafana.env || (umask 077 && cat > grafana.env <<EOF
GF_SERVER_ROOT_URL=https://$(hostname).exe.xyz/
GF_SECURITY_ADMIN_USER=$owner
GF_SECURITY_ADMIN_PASSWORD=$(openssl rand -hex 24)
EOF
)
grafana.env (mode 600) holds a random password for Grafana's built-in
admin. You never type it, because the login form is off, but keep it with
your backups.
The Compose file. Grafana listens on port 8000, the one exe.dev's proxy
serves, and trusts the proxy's X-ExeDev-Email header as the user name.
Prometheus listens only on the VM's loopback:
cat > compose.yaml <<'EOF'
name: monitoring
services:
grafana:
image: grafana/grafana:13.3.1
restart: unless-stopped
user: "1000:1000"
ports: ["8000:3000"]
env_file: grafana.env
environment:
GF_AUTH_PROXY_ENABLED: "true"
GF_AUTH_PROXY_HEADER_NAME: X-ExeDev-Email
GF_AUTH_PROXY_HEADER_PROPERTY: username
GF_AUTH_PROXY_HEADERS: Email:X-ExeDev-Email
GF_AUTH_PROXY_AUTO_SIGN_UP: "true"
GF_AUTH_DISABLE_LOGIN_FORM: "true"
GF_AUTH_DISABLE_SIGNOUT_MENU: "true"
GF_AUTH_BASIC_ENABLED: "false"
GF_USERS_AUTO_ASSIGN_ORG_ROLE: Viewer
GF_PLUGINS_PREINSTALL_SYNC: grafana-llm-app@1.2.0
volumes:
- ./grafana/data:/var/lib/grafana
- ./grafana/provisioning:/etc/grafana/provisioning:ro
- ./grafana/dashboards:/etc/grafana/dashboards:ro
depends_on: [prometheus]
prometheus:
image: prom/prometheus:v3.16.0
restart: unless-stopped
user: "1000:1000"
command:
- --config.file=/etc/prometheus/prometheus.yml
- --storage.tsdb.path=/prometheus
- --storage.tsdb.retention.time=90d
- --storage.tsdb.retention.size=5GB
- --web.enable-lifecycle
ports: ["127.0.0.1:9090:9090"]
extra_hosts: ["host.docker.internal:host-gateway"]
volumes:
- ./prometheus:/etc/prometheus:ro
- ./prometheus-data:/prometheus
blackbox:
image: prom/blackbox-exporter:v0.29.1
restart: unless-stopped
EOF
Prometheus scrapes every VM listed in prometheus/vms/ (this one over the
Docker host gateway, the others through their integrations) and probes every
URL in prometheus/sites/. It rereads both directories on its own:
cat > prometheus/prometheus.yml <<'EOF'
global:
scrape_interval: 30s
evaluation_interval: 30s
scrape_configs:
- job_name: node
file_sd_configs:
- files: [/etc/prometheus/vms/*.yml]
- job_name: sites
scrape_interval: 60s
metrics_path: /probe
params:
module: [http_2xx]
file_sd_configs:
- files: [/etc/prometheus/sites/*.yml]
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: blackbox:9115
- job_name: prometheus
static_configs:
- targets: [localhost:9090]
EOF
printf -- '- targets: [host.docker.internal:9100]\n labels: {vm: %s}\n' \
"$(hostname)" > "prometheus/vms/$(hostname).yml"
Grafana's data source, dashboards, and LLM plugin. The dashboards are the community's Node Exporter Full (Apache-2.0, pinned to a commit) and this recipe's VM fleet. The LLM plugin talks to exe.dev's gateway as an OpenAI-compatible provider, with no key:
cat > grafana/provisioning/datasources/prometheus.yaml <<'EOF'
apiVersion: 1
datasources:
- name: Prometheus
uid: prometheus
type: prometheus
access: proxy
url: http://prometheus:9090
isDefault: true
jsonData:
timeInterval: 30s
EOF
cat > grafana/provisioning/dashboards/files.yaml <<'EOF'
apiVersion: 1
providers:
- name: files
folder: VMs
type: file
options:
path: /etc/grafana/dashboards
EOF
cat > grafana/provisioning/plugins/llm.yaml <<'EOF'
apiVersion: 1
apps:
- type: grafana-llm-app
jsonData:
provider: custom
openAI:
url: https://llm.int.exe.xyz
apiPath: /openai/v1
models:
default: base
mapping:
base: gpt-5.4-mini
large: gpt-5.5
EOF
curl -fsSL -o grafana/dashboards/node-exporter-full.json \
https://raw.githubusercontent.com/rfmoz/grafana-dashboards/b07d8a90d8fadd112e9ac841b777ee863cda850b/prometheus/node-exporter-full.json
curl -fsSL -o grafana/dashboards/fleet.json https://exe-staging.dev/deploy/grafana/fleet.json
jq -r .title grafana/dashboards/*.json
jq prints Node Exporter Full and VM fleet.
Start the stack#
docker compose up -d --wait
docker compose ps
curl -fsS localhost:8000/api/health | jq -r .database
curl -fsS -H "X-ExeDev-Email: $owner" localhost:8000/api/user | jq -r '.login, .isGrafanaAdmin'
sleep 35; curl -s localhost:9090/api/v1/query --data-urlencode 'query=up{job="node"}' | jq -c '.data.result[] | [.metric.vm, .value[1]]'
All three services should be Up. The checks print ok, then your email
and true (you're Grafana's admin), then this VM with "1".
Add the starter alerts#
alerts.sh uses Grafana's API to create the Email me contact point (exe.dev's email gateway, to the owner), point the notification policy at it, and add the five rules. They stay editable in Grafana's UI:
curl -fsSL -o alerts.sh https://exe-staging.dev/deploy/grafana/alerts.sh
chmod +x alerts.sh && ./alerts.sh
It prints Alerts ready: 5 rules in 'VM alerts', sent to <your email> by email.
Set up nightly backups#
This copies Grafana's SQLite database safely while it runs, checks the copy, and archives it with the config. Two weeks are kept:
cat > backup.sh <<'EOF'
#!/bin/sh
set -eu
umask 077
cd "$(dirname "$0")"
mkdir -p backups
work=$(mktemp -d)
trap 'rm -rf "$work"' EXIT
sqlite3 grafana/data/grafana.db ".backup '$work/grafana.db'"
test "$(sqlite3 "$work/grafana.db" 'PRAGMA integrity_check;')" = ok
out="backups/monitoring-$(date +%Y%m%d-%H%M%S).tar.gz"
tar -czf "$out.tmp" compose.yaml grafana.env alerts.sh backup.sh \
prometheus grafana/provisioning grafana/dashboards -C "$work" grafana.db
mv "$out.tmp" "$out"
find backups -name 'monitoring-*.tar.gz' -mtime +14 -delete
EOF
chmod +x backup.sh
sudo tee /etc/systemd/system/monitoring-backup.service >/dev/null <<'EOF'
[Unit]
Description=Back up Grafana and the monitoring config
[Service]
Type=oneshot
User=exedev
ExecStart=/home/exedev/monitoring/backup.sh
EOF
sudo tee /etc/systemd/system/monitoring-backup.timer >/dev/null <<'EOF'
[Unit]
Description=Back up Grafana nightly
[Timer]
OnCalendar=daily
Persistent=true
[Install]
WantedBy=timers.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now monitoring-backup.timer
sudo systemctl start monitoring-backup.service
systemctl list-timers monitoring-backup.timer
tar -tzf backups/monitoring-*.tar.gz | grep -c .
The timer lists its next run, and tar counts the files in the first
archive. Open https://<vm>.exe.xyz/ and try the walkthrough above.
Back up and restore
Take an extra backup any time, on the VM:
cd ~/monitoring && ./backup.sh && ls -lh backups | tail -3
Restore#
On a new VM, run Install by hand first, then copy your archive into
~/monitoring/backups. On the VM, substitute the archive's name; the
current database and config are set aside, not deleted:
cd ~/monitoring
backup=backups/monitoring-YYYYMMDD-HHMMSS.tar.gz
old="before-restore-$(date +%s)"
tar -tzf "$backup" >/dev/null &&
docker compose stop grafana &&
mkdir "$old" && cp -a grafana.env compose.yaml prometheus grafana/provisioning grafana/dashboards "$old"/ &&
mv grafana/data/grafana.db "$old"/ && rm -f grafana/data/grafana.db-wal grafana/data/grafana.db-shm &&
tar -xzf "$backup" --exclude=grafana.db &&
tar -xzf "$backup" -O grafana.db > grafana/data/grafana.db &&
docker compose up -d --wait
On a VM with a different name, fix GF_SERVER_ROOT_URL in grafana.env
and rename this VM's file in prometheus/vms/, then run
docker compose up -d. The VM-to-VM integrations must be attached to the
new VM (ssh exe.dev integrations attach node-web1 vm:<new-vm>).
Update
The images are pinned to the versions this recipe was tested with. To
update, read the release notes for
Grafana,
Prometheus and the
Blackbox exporter,
change the tags in compose.yaml, then on the VM:
cd ~/monitoring
./backup.sh && docker compose pull && docker compose up -d --wait
For node_exporter, rerun the script with the version you want, on each VM:
curl -fsSL https://exe-staging.dev/deploy/grafana/node-exporter.sh | NODE_EXPORTER_VERSION=1.12.1 sh.
Something not working?
Start with the logs and Prometheus's view of its targets. On the VM:
cd ~/monitoring
docker compose ps
docker compose logs --tail=50 grafana prometheus
curl -s localhost:9090/api/v1/targets | jq -r '.data.activeTargets[] | [.labels.vm // .labels.instance, .health, .lastError] | @tsv'
-
A VM shows
downwith401or307. The integration isn't attached to this VM, or its target is wrong. Checkcurl -s https://reflection.int.exe.xyz/integrations | jq -r '.integrations[].name'and the target's port (:9100). -
A VM shows
downwith502orconnection refused. node_exporter isn't running there:systemctl status node_exporteron that VM. -
Down after a reboot? Run
sudo systemctl enable --now docker.service. -
Grafana shows a login form or "invalid username or password". You reached it without exe.dev's proxy, so there's no
X-ExeDev-Email. Usehttps://<vm>.exe.xyz/, not an SSH tunnel. -
You're a Viewer, not the admin. Grafana's admin is whoever
GF_SECURITY_ADMIN_USERingrafana.envnamed at its first start. Act as that account from the VM to promote yourself:cd ~/monitoring me=you@example.com admin=$(sed -n 's/^GF_SECURITY_ADMIN_USER=//p' grafana.env) id=$(curl -s -H "X-ExeDev-Email: $admin" "localhost:8000/api/users/lookup?loginOrEmail=$me" | jq .id) curl -s -X PUT -H "X-ExeDev-Email: $admin" -H 'Content-Type: application/json' \ -d '{"isGrafanaAdmin":true}' localhost:8000/api/admin/users/$id/permissions curl -s -X PATCH -H "X-ExeDev-Email: $admin" -H 'Content-Type: application/json' \ -d '{"role":"Admin"}' localhost:8000/api/org/users/$id -
No alert email? Check spam, then Alerting → Contact points for the error next to Email me, and
docker compose logs grafana | grep -i webhook. -
✨ Auto generate is missing.
docker compose logs grafana | grep -i llm; the plugin installs when Grafana starts, so it needs the internet then.
Frequently asked questions#
How much does it cost to self-host Grafana on exe.dev?
Grafana, Prometheus and the exporters are free and open source. You pay for your exe.dev plan; one VM can watch the rest of your VMs. AI suggestions use your exe.dev LLM allocation.
Can it monitor servers that aren't on exe.dev?
Yes. Put them on your Tailscale network with this VM, or give them an HTTPS address Prometheus can scrape, and list them in a file under prometheus/vms/ the same way. Websites anywhere can be checked with the Blackbox exporter.
Why not just use Grafana Cloud?
Grafana Cloud's free tier is generous, but your VMs' metrics then leave exe.dev, each VM needs an agent with a Cloud API key, and retention and series counts are capped. Here the data stays on a disk you own, sign-in is your exe.dev login, and VMs are scraped without any key on them.
How much disk does Prometheus need?
About 2,000 series per VM at a 30-second scrape, which is roughly 10 MB a day per VM after compression. The 5 GB cap holds 90 days for five to six VMs; raise --storage.tsdb.retention.size or resize the disk for more.
Is the auth proxy login safe?
Grafana believes the X-ExeDev-Email header on any request that reaches port 8000. Only exe.dev's proxy and programs on the VM itself can reach that port, and the proxy replaces whatever header a visitor sends, so outsiders can't forge it. Anyone who can run commands on the VM can act as any Grafana user, the same as they could read its disk.
Who watches the watcher?
Grafana can't alert you while its own VM is down. Watch https://my-grafana.exe.xyz/api/health from somewhere else, such as an Uptime Kuma VM or an outside service.
Updated Nov 16, 2026 · tested end to end Nov 16, 2026 · Markdown · manifest.json · Something wrong? Tell us