Cluster – keep several nodes in sync

If you run more than one Node64 – say the Raspberry Pi in the basement, a VM on the server and a node at your parents' place – you connect them into a cluster. The nodes then sync settings and entries directly with each other: encrypted, even across different networks and even without ipv64.net.

Why a cluster?

• One picture on every node: checks, maintenance windows, notification settings and other areas are the same on every node – you can edit them anywhere.
• Several measuring points: a check can be measured by several nodes; each node shows up as its own measuring point. You still get exactly one notification.
• Share certificates: set up certificates, the ACME account, DNS credentials and your own CAs once and use them on every node.
• Independent of the cloud: a cluster without an ipv64 account works entirely locally – health checks included, with their own evaluation and notifications.
• No extra port: the nodes talk to each other over the normal web port.

Requirements

• The same ipv64 account or none: all nodes of a cluster have the same ipv64 account – or none at all. Mixing is not possible.
• Reachability: the nodes must reach each other at their address and web port, at home or across networks (e.g. via VPN or port forwarding).
• Correct time: the hosts' clocks should be right via NTP – requests between nodes are signed with a timestamp.
• Same version: ideally keep all nodes on the same release; a node that is too old gets suspended.

What it looks like

  # Three nodes of the same account (or all without an account)
 ┌───────────────┐   HTTPS, web port      ┌───────────────┐
 │ Node64 "Pi"   │◄──────────────────────►│ Node64 "NAS"  │
 │ home network  │   signed               │ home network  │
 └──┬────────────┘                        └────────┬──────┘
    │  ▲                                           ▲
    │  │   checks · certificates · measurements    │
    │  │            ┌───────────────┐              │
    │  └───────────►│ Node64 "Work" │◄─────────────┘
    │               │ other network │
    │               └───────────────┘
    │ outbound, HTTPS 443 (optional)
    ▼
 ┌───────────────┐
 │   ipv64.net   │  health checks keep running via the ipv64 servers;
 └───────────────┘  the cluster is an additional path

Add another node

On a node that is already in the cluster (or the first node), click Code for another node under Cluster.

• The node shows an 8-character code (e.g. ABCD-EFGH) and its own address.
• The code is valid for 10 minutes and only once; you can discard it at any time.

Join a cluster

On the new node, open Cluster → Join a cluster and fill in three fields:

1. Address of a node in the cluster
   with https://, e.g. https://node64-pi.lan:8080.
2. Code
   the code from there.
3. Address of this node
   the way the others reach it. The suggestion is https:// plus the address you are currently using to open the page; if NODE64_HTTPS_PORT is set, with that port.

The node then appears on every member. Each node can also leave the cluster on its own – the others are informed automatically.

Always connected via HTTPS

• The nodes always talk encrypted. You don't need a separate HTTPS port: the normal web port (NODE64_HTTP_PORT, default 8080) also speaks TLS.
• The certificate doesn't have to be valid – a self-signed one is enough. Protection against eavesdropping and man-in-the-middle comes from binding the connection to the keys exchanged during pairing.
• A proxy in between that breaks up TLS therefore won't work (the node is suspended with the reason tls_binding). Talk to the nodes directly.

What gets synced?

Under What gets synced there is one switch per module and, below it, one per area:

• An area is only synced with nodes where it is on as well – i.e. only if it is on on both nodes.
• Modules that are not running on this node are greyed out.
• Conflicts: if two nodes change the same thing, the newer change wins. Deleted stays deleted.
• Notification channels remain each node's own setting.
• Certificates: only one node renews – the one that issued last. Another one only takes over once a third of the renewal window has passed (10 days for Let's Encrypt). Deleting on one node deletes everywhere.
• Members without the Health checks module switched on and suspended nodes don't count for measurements, notifications or passing changes on to ipv64.
• Switching happens only here on the Cluster page. The Health checks sync page only shows whether syncing is on or off and links here.

Planned: PROXY64

PROXY64 is meant to be synced via the cluster later on as well – with switches per area like the other modules. This will be built once demand is high enough; the cluster is already prepared for it. Planned, for example:

• PROXY64: proxy hosts and access lists shared across all nodes – the target of a host may differ per node, and a host can be switched off on a particular node (service at one location only).
• Statistics are not copied but fetched live from each node and shown with node names and as a total.

Never synced on purpose: DynDNS (two nodes would overwrite the same domain in turn), login, two-factor, passkeys, SSO and API keys, as well as ports, logging, backup and updates.

Managing members

At the top it says whether this node belongs to an ipv64 account ("This node belongs to ipv64 account #…") or has none. Below is the list of all members:

Remove from cluster takes a node out; all others are informed.

Suspended nodes

If something fundamental changes on a node, it gets suspended: the others stop talking to it but keep working normally among themselves.

Except for version_too_old, a node is only taken back by hand with Resume. It is checked again; if it still doesn't fit, it stays suspended and the interface shows the reason.

When a node is gone

1. Offline, not suspended
   The others keep pinging it every 30 seconds; after 2 minutes without an answer it counts as offline (shown in the list with the last error). It is neither suspended nor removed – being offline is not a reason for that.
2. The others keep working
   The remaining nodes sync with each other as usual.
3. Notifications
   If it measures a check together with others, the next reachable measuring node takes over the alerts – without duplicates. If it is the only one measuring a check, the check becomes "not monitored" after 3 intervals (at least 5 minutes) without a measurement.
4. Passing changes to ipv64
   If it was the node passing changes on to ipv64, the next connected and reachable node takes over.
5. It comes back
   Both sides catch up on everything that changed in the meantime from their last state – including changes made on the node itself – and measurements too. A node is only removed permanently with Remove from cluster.

Alert: if a node is unreachable or suspended for more than 5 minutes, your notification channels receive "🔴 Cluster: node … unreachable" (with the last error or the suspension reason), and when it is back "🟢 … reachable again" – each with a link to the cluster page. Exactly one node sends it: the one with the smallest ID among the reachable ones; whoever sent the alert also sends the all-clear. The dashboard then shows a notice at the top, and the Cluster tile (only with members) shows the state of every node.

Split brain: parts of the cluster separated for a while

There is no leader and no majority rule: if the connection between parts of the cluster is gone, each part keeps working on its own and nothing is locked. After reconnecting, every node catches up from its last state, and all end up with the same result:

Who notifies?

• If only one node measures a check, that node notifies – or ipv64, if it is connected.
• If several nodes of the cluster measure a check, exactly one notifies. If it fails, the next one takes over – without duplicate alerts.
• A check's detail page shows under Notified by who sends the alerts.

Working with ipv64.net

• For connected nodes, health checks keep running via the ipv64 servers; the cluster is an additional path.
• A connected node passes changes from nodes without their own health checks sync on to ipv64.
• If no connected node is reachable, changes and alerts simply stay within the cluster.

More under Connect to ipv64.net.

Security

• Each node has its own Ed25519 key, stored encrypted in its database.
• Every request and every response between nodes is signed – over method, path, time, account, a one-time value, the content and an identifier of the TLS connection. A man in the middle would have different connections to both sides and is rejected before any data flows.
• Replayed requests are detected and rejected.
• Joining is only possible with the one-time code (10 minutes, discarded after 5 failed attempts).
• If a node's account or key changes, it is suspended automatically.
• The endpoints for the nodes live on the normal web port under /peer/; they only accept signed requests from members or the join code.

Technical details: port, endpoints, timings

Port and connection

Endpoints

On the web port, without logging into the interface – they only accept signed requests from members or the join code:

Timings and limits

API

In the API reference the section is called "Cluster"; paths still start with /api/group.

Every change is recorded in the audit log (area cluster). For monitoring there are Prometheus values for the members – see Prometheus.

Monitoring

• Dashboard: the Cluster tile with the state of every node and a notice at the top when a node is gone or suspended; plus the alert after 5 minutes.
• Interface Cluster and GET /api/group – per member: online, state, reason, areas, last reached and last error.
• Prometheus: node64_group_members, node64_group_members_online and node64_group_member_up{node_id,node_name} – see Prometheus.
• Log (area peer): "cluster member suspended/resumed", "peer request rejected", "cluster entries updated"; fetch errors at debug level. The audit log lists everything under sync.

Something unclear, missing, or a question about Node64? Ask in the official forum v64.tech.