Backup Jobs (Job Queue)

Every backup, restore, export, and related task is a job stored in the dashboard database. Agents claim work by polling (site-scoped). The Jobs page is the live operator view.

Job lifecycle

Status Meaning
Pending / Queued Created; waiting for an agent to claim it
Running Agent executing; may show file counts, restic progress, elapsed time
Completed Finished successfully; remains until cleared or cleanup policy
Failed Error from agent or system — open details and agent logs

Where jobs come from

  • Dashboard Run / Backup all (primary repository per site)
  • Schedules (minute runner)
  • Restore / Export / archive restore
  • Public API, the built-in MCP server, or a connected chat app

The Restore page also shows a focused Active job card for restore/export types; full history always lives under Jobs.

Default filter is Recent (about 7 days + active work). Switch to Active only, All statuses, or a single status. Search matches type, site, agent, progress text, or job id. Stat tiles jump to common filters.

Actions

  • Refresh — pull now (the page also live-updates).
  • Clear finished — remove completed/failed rows from the list (does not delete backup data).

Admins can enable automatic retention under Settings → Jobs cleanup (delete finished jobs older than N days).

Restic under the hood

Reaver uses restic on the agent for:

  • Incremental, deduplicated snapshots
  • Optional client-side encryption of repository data (off by default)
  • Restore by snapshot id

Local and remote restic passwords are both dashboard-managed; see Security and Agents.

Destination timeouts and retries

Remote (Offsite) backups and restores do not hang forever when an SSH/S3 host is unreachable (for example mid-reboot):

  • SSH uses a short ConnectTimeout (default 30s).
  • Probe/connect restic commands have a wall-clock limit (default 60s).
  • Connect-like failures are retried up to 2 more times (3 attempts total), with short backoff (5s then 15s).
  • After retries fail, the job is marked Failed with a clear message (destination may be rebooting, offline, firewalled, or credentials wrong). Failed-backup email uses that message.
  • Wrong password / SSH auth failures are not retried as connect errors.

Progress text may show “Retrying destination connection (attempt 2/3)…” while retries run.

Tune timeouts on the agent host via environment variables documented under Agents → Destination connect timeouts.

Variable Default Role
REAVER_SSH_CONNECT_TIMEOUT 30 OpenSSH connect timeout (SFTP)
REAVER_DEST_CONNECT_TIMEOUT 60 Probe / short-command process timeout
REAVER_DEST_CONNECT_RETRIES 2 (extra) Extra connect retries after first attempt
REAVER_DEST_BACKUP_TIMEOUT unset Optional hard cap; unset means run as long as restic keeps producing output

Troubleshooting failed or stuck jobs

  1. Confirm the agent for that site is online.
  2. Check destination credentials (Offsite → Test) for remote backups.
  3. Confirm a primary repository is set if Dashboard Run does nothing useful.
  4. If the job sits on “Connecting to remote destination…”, check Offsite credentials and host reachability, then re-queue.
  5. Read the failed job message: connect-timeout text means fix host reachability or Offsite settings, then re-queue.
  6. Enable Debug logging under Settings for data/debug.log.
  7. On the agent host, inspect worker logs / journalctl for the reaver-agent service.