Restore testing

A successful backup job only tells you the dump ran. Restore testing tells you the backup restores — Norcube takes a real backup, restores it into a disposable database, confirms the data actually came back, then throws that database away. The result shows up as a Passed mark next to the backup and a Restore health score on the datasource.

A restore test is a normal restore under the hood, tagged as a test run — so it rides the same isolated workers and produces the same full, redacted log — but it always restores into a throwaway database, never your live one. That database runs either on an engine Norcube spins up for you (the default — nothing to provide) or on a scratch server you nominate.

Why test restores

Backups fail silently in ways a green "backup succeeded" never shows: a subtly corrupt archive, an engine-version mismatch that only bites on restore, a retention rule that quietly aged out the copy you needed. The only real proof a backup works is restoring it. Restore testing automates that proof on a schedule you choose, so "can we recover?" is answered continuously instead of during an incident.

How a test runs

Every test — scheduled or on-demand — does the same things:

1. Restores the backup into a fresh throwaway database in new-database mode — on the Norcube-managed engine or your scratch server. It never overwrites anything and never reuses the datasource's own credentials.
2. Validates the result to the depth you configured (see Validation depth).
3. Tears down — the throwaway database is dropped (scratch target), or the whole managed engine is destroyed with the task (managed target).
4. Records the outcome — pass or fail, how deep it validated, duration, and the full restore log.

Where you see the results

Once tests are running, results surface in four places on the dashboard:

• Restore test column — the Backup jobs table gains a per-backup mark: Testing… while a test is in flight, Passed once it restores and validates, Passed with warnings if it restored but had to skip an object (see below), or Failed if it didn't. A tooltip shows when it last ran.
• Restore health card — a status-page-style strip of recent tests on the datasource page, with the pass rate and when the last test ran.
• Restore tests tab — the full history of test runs for the datasource: status, result, validation depth, target, duration, and each run's log. This is where you drill into a failed test to read exactly why it didn't restore.
• Datasource list — a compact health bar per datasource, so you can scan recoverability across every database at a glance.

Setting up automated tests

Open the datasource, then Configure on the Restore health card (or the datasource's Restore testing settings). Three choices drive everything:

1. Cadence — how often to test

Scheduled tests always verify the latest successful, still-stored backup at the moment the cadence comes due. If a datasource has no successful backup yet, nothing is tested until the first one lands.

2. Target — where the test restores

Two options:

• Norcube-managed (default). Norcube starts a throwaway database engine inside the test task itself, restores into it over the task's loopback, validates, and destroys the whole engine when the task ends. You provide nothing — no server, no connection string, no network access. The managed Postgres engine bundles the common extensions (including pgvector) and contrib; the managed MongoDB engine tracks the current major. If a database uses an extension the managed engine doesn't carry (for example PostGIS or TimescaleDB), use a scratch target instead — your own server already has it.
• Your scratch database (customer_scratch). A throwaway Postgres/MongoDB server you provide. Norcube creates a uniquely-named database on it per test and drops it afterwards, so it only ever holds data mid-test. Supply a write-capable connection string; it's KMS-encrypted at rest, never returned by any API, and only used to run tests.

A scratch server is reached from the same static egress IP as every backup and restore worker — 18.196.207.101. If it sits behind a firewall, allow that IP on the database port, exactly as for connecting a datasource. The managed target needs none of this — it's only ever reachable on the task's own loopback, with no public IP, security group, or egress.

3. Validation depth — how thoroughly to check

Each test climbs a ladder and records the deepest rung it reached:

• Restored cleanly (L1) — the archive applied without error. The restore tool exiting 0 is itself a strong signal a backup is usable.
• Data present (L2) — additionally, the restored engine is up and the restored database actually contains user tables/collections, not an empty shell.

Both Postgres and MongoDB run the full L1 + L2 ladder. A deeper custom-query (L3) rung — assert against a query you write — is reserved in the settings but not yet running.

Engine versions

The managed engine restores older or same-major backups cleanly — a Postgres 15 backup into the managed Postgres 17 engine, or an older MongoDB into the current major — which is the normal, supported direction. The only case that needs a scratch target is a source newer than the managed engine's major.

MongoDB: whole-instance backups

If a MongoDB datasource's connection string names a database (…/mydb), the backup and each test cover that one database. If it names none, the backup covers the whole instance — every database on the server. A whole-instance test restores all of them into the throwaway engine under their original names, then confirms at least one user database came back with data.

Whole-instance tests deliberately skip the MongoDB system databases — admin, config, and local. Those hold the server's own users, sharding metadata, and oplog rather than your data, and restoring them would overwrite the throwaway engine's own credentials mid-test. Excluding them keeps the test connection alive and still proves your real collections restore.

Test now (on-demand)

You don't have to wait for a cadence. On any successful backup row, the Test now action queues a one-off test of that specific backup — useful right after connecting a database, or to confirm a particular backup before relying on it. It uses the datasource's configured target, and defaults to the managed target when nothing is configured, so it works with zero setup. The button is disabled while a test for that backup is already running, and the row's mark switches to Testing… until it finishes.

Passed with warnings

Sometimes a test restores all your data but can't rebuild one object. The classic case is a Postgres index or primary key on a very large value: Postgres can't build a btree index on a value bigger than about 2700 bytes, so pg_restore loads every row, skips that one index, and reports it. Your data came back; one object didn't.

That's a recoverable backup with a caveat, not a failure, so the test records it as Passed with warnings — an amber mark whose tooltip shows how many objects were skipped and an example reason, with the full detail in the restore log. It still counts as recoverable in the Restore health score.

It's worth acting on, because an object that won't rebuild on a clean server would trip a real recovery too. The fix is usually on the source schema (for the index example, Postgres suggests indexing a hash of the value instead of the raw value). MongoDB tests don't surface warnings this way yet — a Mongo test is Passed or Failed.

Is it safe? What a test touches

Restore testing is built so a test can never write to the database being backed up:

• Always new-database mode. A test only ever restores into a brand-new, uniquely-named database. It never runs an overwrite.
• Never the datasource's credentials. Overwrite restores reuse the datasource's stored credentials (your production database); a test never does — it uses the managed engine's throwaway credentials, or your separate scratch connection.
• Refused at dispatch. As a backstop, the platform refuses to launch any test run that isn't new-database mode, so even a malformed job can't point at production.
• Isolated target. The managed engine only exists on the test task's loopback; a scratch server is checked to be a different host than the source when you save the configuration.
• Cleaned up. The throwaway database is dropped (scratch), or the whole engine is destroyed with the task (managed).

The blast radius of a test is therefore a single short-lived database that is never your production system.

Following tests

Each test run uploads its full restore log — the same redacted pg_restore / mongorestore output as a manual restore — kept for passes and failures alike. Open a run in the Restore tests tab to read it; a failed test's log is usually the fastest way to see why a backup wouldn't restore.

A test that loses contact with the platform is marked failed within about an hour, never silently blessed — the same reaper discipline as manual restores.

Related