Backup jobs

A backup job is a single execution record: one row per scheduled run and one per manual trigger. Jobs answer "did last night's backup run?", "how big is it?", and "can I still download last week's dump?".

What a job records

Status lifecycle

queued → running → success
                 → failed

• queued — waiting for the dispatcher. Jobs are picked up within seconds (the UI promises "within a minute").
• running — an isolated worker container is doing the dump. Big databases can legitimately run for hours.
• success — archive confirmed in storage; size and location recorded.
• failed — the dump or upload failed; the error summary says why.

(The API also defines partial and canceled statuses; they're reserved and not produced today.)

One more state you'll see in the UI: Expired. That's not a job status but a badge on successful jobs whose archive has since been deleted by retention — the row stays in history (the tooltip shows when it expired and under which rule: age exceeded or keep-last-N), but the file is gone and download is disabled.

Stuck jobs heal themselves

If a worker dies without reporting back (container killed, network blip), the job would sit in running forever. A reaper checks long-running jobs against the actual container state hourly and resolves them:

• The archive actually landed and the worker exited cleanly → the job is healed to success, with size and location backfilled from storage. You don't lose a good backup to a lost status report.
• Objects landed but the worker failed → the partial objects are cleaned up and the job is marked failed.
• Nothing landed → marked failed with the container's exit reason as the error summary.

The check is container-state-driven, not time-driven — a legitimate 12-hour dump stays running because the container is genuinely still running. Each reaper outcome writes its own audit entry (backup_job.reaped_healed, backup_job.reaped_orphan_cleaned, backup_job.reaped_stuck).

Download a backup

In the datasource's Backup jobs table, use the row's download action. The dashboard requests a signed URL valid for 15 minutes and starts the download. Rules:

• Only successful, non-expired jobs with an archive on file can be downloaded — the action is disabled otherwise, with the reason shown (e.g. "Backup expired by retention policy and was removed from storage").
• Downloads work for backups in Norcube-managed storage. For BYO destinations the archive is in your bucket — fetch it directly with your own credentials; Backup deliberately has no read access to hand out.
• Every link generation is audited (backup_job.download_link_generated) — the audit row records that a link was minted, never the link itself.

You can also restore in-product — into a new database, over the existing one, or onto a new server — straight from the backup row. Downloading is the manual alternative for restoring with your own tooling (pg_restore --dbname=... <file> / mongorestore --archive < <file> — see Supported engines).

Delete a backup

Use the row's delete action. Deleting a single backup:

• is refused with a conflict while the job is queued or running,
• removes the archive from managed storage within about a minute (the cleanup runs asynchronously); archives in BYO buckets are yours and are not touched,
• keeps nothing — unlike retention expiry, a manually deleted job is removed from history too,
• writes a backup_job.deleted audit entry before anything is removed.

Behaviour and edge cases

• Backups don't lock your database. pg_dump takes only ACCESS SHARE table locks — reads and writes proceed normally; DDL (ALTER TABLE etc.) waits until the dump finishes.
• Failures are sticky. A failed job is never retried by itself; the next scheduled slot simply runs a fresh job.
• Manual and scheduled runs can overlap. Backup deliberately allows parallel jobs for the same datasource — each worker is independent.
• Job history lives with the datasource. Deleting the datasource deletes its job records; export first if you need the trail (nrc snapdb backup list --datasource <id> --all-pages -o json).

Related

• Browse backup history — the dashboard and CLI views.
• Encryption and security — how credentials and archives are protected end to end.