Prometheus & Grafana

Node64 exposes its metrics at /api/metrics in the Prometheus text format – from the core through health checks, DynDNS, DNS64, PROXY64 and CDN64 to certificates. Plug your node into your existing monitoring, build Grafana dashboards and write your own alerting rules.

Overview

Setup

1. Under the API menu item (/api/docs, section API keys), create a dedicated key, for example named prometheus. More in Local API.
2. Save the key on your Prometheus server in a file such as /etc/prometheus/node64.key. The file contains nothing but the key (node64_…).
3. Add a scrape job to your prometheus.yml and reload Prometheus.

scrape_configs:
  - job_name: node64
    scheme: https            # or http, depending on how you open the web interface
    metrics_path: /api/metrics
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/node64.key   # contains: node64_...
    tls_config:
      insecure_skip_verify: true   # only with the web interface's self-signed certificate
    static_configs:
      - targets: ["node64.lan:8080"]

Node64 speaks HTTP and HTTPS on the same port. If the web interface has a proper certificate (see Certificates), drop insecure_skip_verify. Every node has its own API keys, so for several nodes create one scrape job per node, each with its own key file.

Quick manual test:

curl -s -H "Authorization: Bearer $(cat node64.key)" http://node64.lan:8080/api/metrics | head

Authentication and protection

• /api/metrics is protected like the rest of the API: without a key you get 401.
• Every scrape updates the key's "last used" – so the key list tells you whether Prometheus is still scraping.
• Node64 slows down invalid keys just like for the rest of the API (after 5 failed attempts, only one attempt every 30 seconds).
• Use a key just for Prometheus. If you no longer need it, delete it without breaking any other scripts.

All metrics

Points in time are Unix time in seconds (suffix _seconds), durations are in seconds, data volumes in bytes. Metrics that are "always 1" carry their information in a label – count them with count or sum.

Core

Health checks

Labels per check: token (stays the same even if the name changes), check (name), type and group.

Proxmox VE

Per system (system), node and guest: node64_pve_up, node64_pve_error, node64_pve_poll_failures_total, node64_pve_quorate, node64_pve_guests_running, node64_pve_guests_bad, node64_pve_node_up, node64_pve_guest_state, node64_pve_guest_up plus CPU, RAM and disk as a ratio 0..1 (node64_pve_node_*_ratio, node64_pve_guest_*_ratio), and per guest network and disk I/O in bytes/s (node64_pve_guest_net_*_bytes, node64_pve_guest_disk_*_bytes), RAM in bytes and CPUs (node64_pve_guest_memory_bytes, node64_pve_guest_cpus). The full list is under Proxmox VE.

Docker

For Docker monitoring. Per host: node64_docker_up, node64_docker_enabled, node64_docker_poll_failures_total, node64_docker_last_poll_seconds, node64_docker_containers_running, node64_docker_containers_bad. Per container: node64_docker_container_state{state,detail}, …_container_up, …_container_restarts, …_container_cpu_ratio, …_container_cpus, …_container_memory_ratio, …_container_memory_bytes, …_container_memory_limit_bytes, …_container_uptime_seconds, …_container_net_in_bytes, …_container_net_out_bytes, …_container_disk_read_bytes and …_container_disk_write_bytes.

DynDNS

DNS64

The counters run since the DNS server last started. Changing the settings restarts it – rate() and increase() handle that just fine.

PROXY64 and CDN64

Labels per host: host_id (stays the same), host (first name) and kind (proxy, redirect, dead, passthrough).

CDN64 – these values come from the node's last sync with ipv64 (every minute):

Certificates

The values of the central certificate management belong to the core and are always included. Labels per certificate: cert_id, name (first name) and source (ipv64, alias, http, dns, ca, self, upload).

Honeypot64

Only with the module switched on; details under Honeypot64.

Example alerting rules

You can drop these rules straight into a Prometheus rule file and route them through Alertmanager. Adjust thresholds and wait times (for) to your setup.

groups:
  - name: node64
    rules:
      - alert: Node64Down
        expr: up{job="node64"} == 0
        for: 5m
        annotations:
          summary: "Node64 {{ $labels.instance }} is not responding"
      - alert: Node64ModuleError
        expr: node64_module_error == 1
        for: 10m
        annotations:
          summary: "Module {{ $labels.module }} reports an error"
      - alert: Node64CheckDown
        expr: node64_hc_check_up == 0 and on(token) node64_hc_check_maintenance == 0
        for: 2m
        annotations:
          summary: "{{ $labels.check }} is down"
      - alert: Node64CertExpiresSoon
        expr: node64_hc_check_cert_expiry_seconds - time() < 14 * 86400
        annotations:
          summary: "Certificate of {{ $labels.check }} expires in less than 14 days"
      - alert: Node64DynDNSError
        expr: node64_dyndns_target_ok == 0
        for: 15m
      - alert: Node64DNSUpstreamDown
        expr: node64_dns64_upstream_up == 0
        for: 5m
      - alert: Node64CertExpires
        expr: node64_cert_expiry_seconds - time() < 14 * 86400 and on(cert_id) node64_cert_auto_renew == 1
        annotations:
          summary: "Certificate {{ $labels.name }} expires in less than 14 days (renewal failing?)"
      - alert: Node64CDN64TargetDown
        expr: node64_cdn64_host_target_up == 0 and on(host_id) node64_cdn64_host_enabled == 1
        for: 5m
      - alert: Node64CDN64TunnelDown
        expr: node64_cdn64_online_edge_connected == 0
        for: 3m
      - alert: Node64IPv64Disconnected
        expr: node64_link_linked == 1 and node64_link_connected == 0
        for: 10m

Grafana

Grafana doesn't need a separate interface: it reads the values from Prometheus.

1. In Grafana, go to Connections → Data sources, add a data source of type Prometheus and enter the address of your Prometheus server.
2. Create a new dashboard and add panels with the queries below.
3. Optionally add a variable for several nodes, for example with label_values(node64_info, node_name).

There's no ready-made dashboard template – but with these queries you'll have your own dashboard up in no time:

If your node is linked, you'll also find the results of online checks from Nuremberg and Falkenstein in the ipv64.net portal.

Something unclear, missing, or a question about Node64? Ask in the official forum v64.tech.