Monitor restic backups
restic is quiet when it works and quieter when it stops. The cron line gets commented out during maintenance. The repo credential expires. A mount goes missing and the snapshot backs up an empty directory without complaint. Exit code 0 means the process finished. It says nothing about whether the backup is any good.
Monitoring restic properly means catching all three failure shapes. The run that never happened, the run that failed, and the run that succeeded looking wrong.
The one line
restic prints a machine-readable summary when you pass --json. The last line of that output holds the numbers worth watching. Send them to your check's ping URL after every run, and hit /fail when the run errors.
restic backup /srv --json | tail -n1 > /tmp/summary.json curl -fsS -m 10 --retry 3 PING_URL -d "$(jq -r \ '"bytes=\(.data_added)&files=\(.files_new + .files_changed)&duration=\(.total_duration|floor)&snapshot=\(.snapshot_id[0:8])"' \ /tmp/summary.json)"
That sends data_added, the changed file count, the duration and the snapshot id. In cron, that is the whole integration. A failed restic run skips the check-in, and your wrapper's error branch reports it.
From a systemd timer, keep the same commands in the script the service runs, and hang the failure report on an OnFailure unit so a crash that never reaches your script's error branch still reports in.
# /etc/systemd/system/restic-backup.service [Unit] Description=Nightly restic backup OnFailure=restic-backup-fail.service [Service] Type=oneshot ExecStart=/usr/local/bin/restic-backup.sh # /etc/systemd/system/restic-backup-fail.service [Service] Type=oneshot ExecStart=/usr/bin/curl -fsS -m 10 PING_URL/fail
Silence, and why it matters more than errors
Most dead backups die silently. The job stops being scheduled, the host gets rebuilt without it, the timer never fires again after a change nobody connected to backups. None of that produces an error, so nothing that waits for errors will ever fire.
Lastcopy expects a check-in every period you set, nightly for most restic jobs. When nothing arrives within the period plus a grace window, one email says the backup went missing and how overdue it is. One recovery notice when it comes back, and nothing in between. The pattern has a name and a longer write-up, a dead man's switch for backups.
What the numbers catch
After 7 runs Lastcopy has learned this job's normal data_added, file count and duration, using the median and MAD of the trailing 30 runs. The nightly job that normally adds 2 GB and suddenly adds 40 KB gets flagged even though restic exited 0, and the email states its evidence, the run's number against the median. There are no thresholds to configure.
Verify the repository weekly
Plain restic check verifies structure only. Reading data back is what detects corruption, and --read-data-subset=n/t reads slice n of t, so rotating the slice weekly covers the whole repository every t weeks. (The x% form picks a random subset, so repeats never guarantee coverage.) restic check takes an exclusive lock, so schedule it away from the backup window.
SLICE=$(( $(date +%V) % 5 + 1 )) # rotate 1..5 by ISO week restic check --read-data-subset=$SLICE/5 \ && curl -fsS PING_URL/verify -d "verify=pass&subset=$SLICE/5" \ || curl -fsS PING_URL/verify -d "verify=fail"
Each result becomes a verification row on the check's evidence report.
Drill a restore quarterly
A verified repository can still fail you at restore time. A drill is a restore into a scratch path with the contents checked, a row count, a checksum, a file you know must exist. Lastcopy schedules the drill on the cadence you set, nags weekly while one is overdue, and records the result on the register auditors ask for.
restic restore latest --target /srv/drill --include /srv/files test -s /srv/drill/srv/files/customers.db ; RC=$? rm -rf /srv/drill [ $RC -eq 0 ] \ && curl -fsS PING_URL/drill -d "result=pass&by=$USER" \ || curl -fsS PING_URL/drill -d "result=fail&by=$USER"
A failed drill alerts immediately and stays on the record with its remediation, which is stronger evidence than a perfect page. The register maps to ISO 27001 A.8.13 and SOC 2, covered on the compliance page.
Or keep it simpler
If all you want to know is that the cron ran, a plain cron monitor does that well, for less. Lastcopy earns its keep on the run that succeeded looking wrong and on the restore evidence the audit eventually asks for.