Alerts
When NODE64 gets in touch – by email, webhook or bell.
A dashboard only helps if you look at it. Alerts turn that around: NODE64 gets in touch when something happens.
Rules #
A rule consists of a trigger, a threshold and channels.
There are two kinds. Event-driven triggers fire the moment something happens. State-driven ones check every five minutes whether a measurement crosses a limit – and most of them only once it has stayed across it for a minimum duration, so that a single spike sets nothing off.
| Trigger | When it fires | Default |
|---|---|---|
| Anomaly detected (intrusion, encryption) | Changed anchor file, back door, new counterpart for a service, login from an unfamiliar origin — anything pointing to an incident in progress | from "high" |
| New vulnerability (CVE) | As soon as a finding at or above the chosen rating appears | from "high" |
| Critical check failed | When a hardening or best-practice check fires | from "high" |
| Device stopped reporting | When a device stays silent well beyond its interval | – |
| Disk filling up | When a mount point is this full | 85 % |
| Inodes running out | When a mount point is running out of inodes | 85 % |
| Memory running low | Memory in use above the limit | 90 % |
| Swap heavily used | Swap in use above the limit | 50 % |
| Sustained CPU load | CPU usage above the limit | 90 % |
| System load per core too high | Load per core – 200 % means twice oversubscribed | 200 % |
| Temperature too high | Chassis or a drive too warm | 70 °C |
| Unusual data volume | An interface's throughput against its own weekly average | 400 % |
| Drive reporting problems | A drive reports itself as ailing | – |
| Pool no longer healthy | A ZFS, btrfs or RAID pool is degraded | – |
| Certificate expiring | Remaining validity below the limit | 21 days |
| Agent is outdated | A device is running an old agent build | – |
| Container image lags behind | The tag has moved on in the registry | 30 days |
| Container near its memory limit | A container is using nearly as much memory as it was granted. What goes beyond that, the kernel kills | 90 % |
| Container is being throttled | A container gets less CPU time than it asks for — it runs, but slowly, and from the outside that is hard to see | 25 % of the time |
| Container was killed (OOM) | The kernel ended a container because it ran out of memory. No threshold: it happened or it did not | – |
The subject: only this one #
Seven triggers can concern several things on the same device – disk full, inodes full, temperature, traffic spike and the three container triggers. They offer a subject field: leave it empty and the rule applies to all of them; put /var, nvme0n1, eth0 or a container name there and it applies only to that one.
For the temperature trigger, system means the chassis or processor sensor; any other value means a drive.
The quickest route is a device's Pulse tab: next to every usage bar, every temperature reading and every interface there is Set up an alert – that opens this form with the subject already filled in.
Rules apply globally, to a group or to a single device. Where several exist, the most specific wins: device before group before global. That way you can be stricter about the production server than about the test box.
The first time you open the area, NODE64 creates a sensible default rule – new findings at "high" and above, by email.
Channels #
Email to the members of your account and to additional recipients you enter. Additional addresses must be confirmed – otherwise somebody could flood strangers' inboxes.
Webhook: an HTTP POST to your own URL with the finding as JSON. That connects NODE64 to Matrix, Slack, Gotify, your own automation or whatever you use.
{
"event": "finding.new",
"device": "my-server",
"severity": "high",
"title": "SSH allows password authentication",
"url": "https://app.node64.de/security/123"
}The bell in the header shows everything that came in – even without email.
Webhooks only over HTTPS and only to public addresses. Internal ranges (
127.0.0.1,192.168.x.x, cloud metadata) are rejected, and the hostname is re-checked after DNS resolution. Otherwise the webhook would be a convenient way to send our server into your internal network.
Testing #
Next to every rule there is a test alert. It goes through the same path as a real one – bell, email, webhook. No special case, no mock-up: if the test arrives, the real thing arrives. And if a recipient is rejected, you see it in the result.
Keeping it quiet #
Alerts are only as useful as their hit rate. Two levers against them turning into noise:
- Raise the threshold – not every low finding needs to ring at night.
- Postpone or accept findings – whoever decided deliberately is not reminded again. See Understanding findings.