# Operations

> Day-2 operation of a released installation: cleanup, backups, restore drills, verified upgrades, schema-aware rollback, monitoring, and secret rotation.

[Canonical HTML page](https://safeonlineexam.com/docs/operate/operations)

Keep the deployment record for every change: release version, immutable image
digest, verification result, active revision or Compose environment, migration
result, cleanup result, and numbered secret versions where used. The public release
process and an institution's deployment are separate: publishing a release does
not change a school service.

## Scheduled cleanup [#scheduled-cleanup]

PostgreSQL does not apply a native TTL to runtime records. The cleanup command
deletes expired sessions, transient states, and operation locks in bounded batches
using `FOR UPDATE SKIP LOCKED`. Run it at least daily and alert when it fails or
stops running.

* **Cloud Run bundle:** the installer creates a non-public cleanup job, a Cloud
  Scheduler trigger, and runs it once during installation. Check its last
  execution after every upgrade.
* **Compose bundle:** run the maintenance profile through a monitored systemd
  timer or cron job:

```bash
docker compose \
  --env-file .env.secrets \
  -f compose.yaml \
  -f compose.secrets.yaml \
  --profile maintenance run --rm cleanup
```

Use absolute paths in unattended jobs, rotate logs, and retain enough metadata to
show when cleanup last succeeded.

## Backups and restore drills [#backups-and-restore-drills]

<Callout type="warn" title="A backup is not verified until it restores">
  Restore into a separate, controlled target—not the live database. Record recovery
  time and the newest restored row, then dispose of the drill target under your data
  handling policy.
</Callout>

For Cloud Run, retain automated backups and PITR according to the selected Cloud
SQL profile. The upgrade helper waits for its on-demand backup operation to finish
and requires a `SUCCESSFUL` backup before it migrates or moves traffic. For
Compose, `./upgrade.sh .env.secrets` creates a custom-format PostgreSQL backup and
validates it with `pg_restore --list`; encrypt and move that backup to approved
off-host storage.

A restore drill should validate the archive, use `pg_restore --exit-on-error`, run
the intended image's migrations, inspect the application tables, and exercise an
isolated LTI/assessment flow. Do not rely on a raw live volume copy as the only
recovery mechanism.

## Upgrades [#upgrades]

Start from the next versioned GitHub Release. Verify its checksum and attestation,
preserve existing protected state, and merge new template keys rather than
overwriting local configuration.

| Target    | Command                     | What the helper protects                                                                                                                                                                                                                                                                                        |
| --------- | --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Cloud Run | `./upgrade.sh cloudrun.env` | Validates the existing target; requires a successful Cloud SQL backup; records prior traffic; migrates; updates cleanup; stages a no-traffic candidate; checks `/ready` and JWKS; then cuts traffic. With a disabled generated URL, it verifies the custom origin and restores that policy after the candidate. |
| Compose   | `./upgrade.sh .env.secrets` | Backs up and validates PostgreSQL; pulls the exact digest; runs migrations through the dependency gate; restarts; and checks readiness.                                                                                                                                                                         |

Neither helper is an unattended image updater. Review each release, its
attestation, migration notes, compatibility, and backup before running it.

## Rollback and recovery [#rollback-and-recovery]

<Callout type="warn" title="Traffic rollback does not roll back the schema">
  An older application can receive traffic only after you confirm it supports the
  current migrated schema. Do not automatically run down-migrations.
</Callout>

The Cloud Run bundle writes a protected rollback record. Its rollback command
requires that record and `--confirm-schema-compatible` before changing traffic.
When `TOOL_URL` is configured, rollback verifies that custom origin after traffic
is restored. For Compose, restore the earlier immutable image only after the same
compatibility review. Schema or data recovery means a forward correction or a
restore into a controlled target—not overwriting the active database during
diagnosis.

## Pool sizing [#pool-sizing]

`DATABASE_POOL_MAX` is per application process. Reserve connections for migration,
cleanup, administrators, monitoring, and restore work:

```text
(maximum app instances × DATABASE_POOL_MAX) + job/admin reserve < max_connections
```

The maintained Cloud Run configuration uses a pool maximum of 5 and up to 10 app
instances, or 50 possible app connections before the reserve. Recalculate both
sides before changing capacity or pool settings.

## Secret and certificate rotation [#secret-and-certificate-rotation]

Create one new immutable secret version at a time, point the runtime at it, deploy
or stage the revision, smoke test, and only then disable the old version. Rotating
`SESSION_SECRET` invalidates sessions; rotating `STATE_ENCRYPTION_KEY` invalidates
outstanding opaque LTI/OAuth state; rotating the LTI signing key requires Canvas
and JWKS coordination.

For a certificate-encryption identity, distribute and validate the replacement
private identity first, then deploy the replacement public certificate, verify the
public-key hash, require fresh configurations, and retire the old client identity
after the planned overlap. Do not introduce plaintext compatibility mode as an
incident response shortcut.

## Monitoring and release smoke checks [#monitoring-and-release-smoke-checks]

Alert on application 5xx and `/ready` failures, migration/cleanup failures,
database connection saturation, storage growth, backup age/failure, stale cleanup,
and certificate expiry when encryption is enabled. On a VPS also monitor host
security updates, disk capacity, container health, and off-host backup transfer.

After every release, check:

```bash
curl -fsS "${TOOL_URL}/health"
curl -fsS "${TOOL_URL}/ready"
curl -fsS "${TOOL_URL}/.well-known/jwks.json"
curl -fsS "${TOOL_URL}/lti/config"
curl -fsS "${TOOL_URL}/js/canvas-seb-detector.js" | head
curl -fsS "${TOOL_URL}/js/canvas-seb-theme-loader.js" | head
```

Then run the relevant [Canvas and SEB acceptance sequence](/docs/operate). Public
route checks establish service health; they do not prove that Canvas registration,
OAuth, a real client, or assessment exit work.