Monitoring Proxmox VE

Node64 monitors Proxmox VE systems – a single node or a whole cluster – read-only via the Proxmox API: nodes, VMs and containers, crashes, restarts and thresholds for CPU, RAM and disk. Node64 never starts, stops or changes anything in Proxmox.

Overview

Monitoring is part of Health checks and lives under Health checks → Proxmox VE. It only runs while the Health checks module is switched on.

• State per guest: running, backup, locked (migration, snapshot …), paused, suspended, stopped, crashed (via qmpstatus), unknown.
• CPU, RAM, disk, network and uptime per node and guest, graphs for 1 h, 24 h, 7 and 30 days.
• Rules with inheritance system → node → guest and alerts through your existing notification channels.
• You can keep single guests or nodes as a regular check – with status pages, uptime bars and sync with ipv64.

Setup

1. Create a read-only token in Proxmox

pveum acl modify / --users '<user@realm>' --roles PVEAuditor
pveum acl modify / --tokens '<user@realm!token>' --roles PVEAuditor

Run as root on a node, for example under Datacenter → Node → Shell:

pveum user add node64@pve
pveum aclmod / -user node64@pve -role PVEAuditor
pveum user token add node64@pve monitor --privsep 0

Proxmox shows the secret only once – copy it right away. The token ID is then node64@pve!monitor.

With privilege separation (--privsep 1, the default in the Proxmox UI) the token has no permissions of its own, even if the user does – and only the intersection of user and token permissions applies. In that case the token itself needs the role, in addition to its user's (already set above):

pveum user token add node64@pve monitor --privsep 1
pveum aclmod / -token 'node64@pve!monitor' -role PVEAuditor

Newer versions also accept pveum acl modify / --tokens 'node64@pve!monitor' --roles PVEAuditor. The PVEAuditor role on / covers everything Node64 reads. If the token sees no guests, Test connection warns you; the notice on the system page shows the matching commands for user and token.

2. Add a system

1. Address(es)
   https://<node>:8006 – one is enough. In a cluster Node64 learns the other nodes itself and falls back to them if one fails. Additional addresses (one per line) are fixed fallbacks.
2. Token ID and secret
   The secret is stored encrypted on the node and never shown again. When editing, an empty field keeps the stored secret. The stored secret only applies to the same addresses and the same pin – if you change the target, enter the secret again.
3. Test connection
   Shows version, cluster, nodes, number of guests and the certificate's SHA-256 fingerprint. For self-signed certificates (the Proxmox default) compare it in Proxmox under Node → System → Certificates and confirm Fingerprint matches – pin it. With certificates from a known CA, such as Let's Encrypt via ACME, pinning is optional. There is no “skip verification” option. Node64 never sends the token to an unconfirmed certificate: confirm the fingerprint first, then run Test connection again.
4. Interval
   15–300 seconds, default 30. The cluster status is queried every minute, the qmpstatus of running VMs every 2 minutes.

If the certificate is reissued in Proxmox, the query reports certificate mismatch. Then edit the system, run Test connection and confirm the new fingerprint.

Overview and system

Proxmox VE lists all systems with state (OK, partly impaired, unreachable, error, off), version, cluster and quorum, nodes online, guests running or faulty and the last query. From here you edit, switch and delete systems. Tiles above it show the state across all systems: systems, nodes online, guests running and faulty, CPU and RAM load and the last incident.

A system shows tiles at the top – nodes online with cluster and quorum, guests, running, faulty (clickable as a filter) and CPU, RAM and core load – and below them nodes and guests as a table: name, ID, type (VM or container), node, state including backup, lock and crash, CPU, RAM, disk, network, uptime, CPU history of the last hour and the as a check switch. You can filter by state (including “faulty”), node, type and search by name, ID or tags. The system's rules are listed below.

Templates and guests with the Proxmox tag nomonitor are not monitored. A stopped guest with a hanging lock counts as stopped after 4 hours.

On the dashboard a Proxmox card sums up all systems. A system only counts as faulty there with a problem of its own – unreachable, node offline or quorum lost; faulty guests are counted separately under “guests faulty”.

If a system shows token secret cannot be decrypted or token secret missing, the key for the secret is missing, for example after an import without a password. The system is not queried then – enter the secret again.

Guest and node in detail

The detail page of a guest or node shows current values and graphs for 1 h, 24 h, 7 days and 30 days: CPU with peak, RAM and disk, network in and out, disk read and write. The rules of that level are listed below.

1-minute values are kept for 48 hours, 5-minute values as long as Keep single measurements and hourly values as long as Keep hourly values – the same local settings as for health checks.

Rules and notifications

Rules apply with inheritance system → node → guest; the lower level overrides the upper one. Each row shows what applies without an own value and where it comes from (“default”, “inherited from system”, “inherited from node”).

Channels

At system level you pick specific channels just like for a check; without a selection alerts go to the default channels. Node and guest inherit the selection – shown as “inherited from system” or “inherited from node” – or get their own selection (nothing ticked = default channels). Deleted channels are skipped when sending; if none of the selected ones remain, alerts go to the default channels. For guests kept as a check, outage and recovery use the check's channels; when created, the check takes over the channels from the Proxmox rules.

Mute, quiet hours and maintenance

Mute silences alerts for a system, node or guest. Quiet hours apply to all Proxmox alerts, crashes included: like check alerts they are held back during quiet hours and then sent as one summary – per guest or node with the latest state (still faulty or above the threshold, faulty in between and fine again, restart) and a link to the Proxmox page. During a maintenance window for all checks there are no Proxmox alerts; a window for specific checks applies to guests kept as a check. Proxmox alerts have their own kinds:

Reported incidents survive a node restart: no second alert afterwards, just the recovery once everything is fine again.

Keep as a check

The as a check switch (per guest and node) creates a regular health check of type proxmox – name “System – guest”, group “Proxmox VE”, interval as the system. That brings status pages, uptime bars, sync with ipv64 and the check's settings such as channels, reminders and “critical”. The same works under Health checks → New check with the type Proxmox VE (VM, container, node) – see Check from a monitored system. The check does not query Proxmox itself but reads the latest query result:

• up: running, backup, locked (guest) or online (node)
• down: pve_stopped, pve_paused, pve_suspended, pve_crashed, pve_offline, pve_unknown, pve_missing (guest no longer exists), pve_unreachable (system unreachable, off or data too old)

Outage and recovery alerts then come only from the check; restarts and thresholds still come from Proxmox VE. Like Docker checks, proxmox checks run only locally on the node that queries the system – not online and not movable to other nodes. Switching the system off pauses its checks; deleting it removes them. If the system is switched off when you enable “as a check”, the check is created paused. There are no duplicate alerts.

API

Under /api/healthchecks/proxmox, authenticated like the rest of the API:

Prometheus

Fetched from the node's local Prometheus endpoint at /api/metrics, with an API key like the other metrics (see Prometheus & Grafana):

At most 1000 guests in total, names at most 64 characters, no templates. All series also carry the label system_id.

Export and import

The configuration export contains the systems with settings and rules. The token secret is only included in an export with a password, encrypted. Without a password a system is imported switched off and shows “secret missing” until you enter it. “As a check” checks are switched over to the system on the new node. Selected channels are rewritten to the new node's channels like for checks; a channel missing there is dropped – if none remain, the default channels apply. Systems with the same name are matched by their addresses on import.

Something unclear, missing, or a question about Node64? Ask in the official forum v64.tech.