Operations

Day-2 operation of a released installation: cleanup, backups, restore drills, verified upgrades, schema-aware rollback, monitoring, and secret rotation.

Keep the deployment record for every change: release version, immutable image digest, verification result, active revision or Compose environment, migration result, cleanup result, and numbered secret versions where used. The public release process and an institution's deployment are separate: publishing a release does not change a school service.

Scheduled cleanup

PostgreSQL does not apply a native TTL to runtime records. The cleanup command deletes expired sessions, transient states, and operation locks in bounded batches using FOR UPDATE SKIP LOCKED. Run it at least daily and alert when it fails or stops running.

Cloud Run bundle: the installer creates a non-public cleanup job, a Cloud Scheduler trigger, and runs it once during installation. Check its last execution after every upgrade.
Compose bundle: run the maintenance profile through a monitored systemd timer or cron job:

docker compose \
  --env-file .env.secrets \
  -f compose.yaml \
  -f compose.secrets.yaml \
  --profile maintenance run --rm cleanup

Use absolute paths in unattended jobs, rotate logs, and retain enough metadata to show when cleanup last succeeded.

Backups and restore drills

A backup is not verified until it restores

Restore into a separate, controlled target—not the live database. Record recovery time and the newest restored row, then dispose of the drill target under your data handling policy.

For Cloud Run, retain automated backups and PITR according to the selected Cloud SQL profile. The upgrade helper waits for its on-demand backup operation to finish and requires a SUCCESSFUL backup before it migrates or moves traffic. For Compose, ./upgrade.sh .env.secrets creates a custom-format PostgreSQL backup and validates it with pg_restore --list; encrypt and move that backup to approved off-host storage.

A restore drill should validate the archive, use pg_restore --exit-on-error, run the intended image's migrations, inspect the application tables, and exercise an isolated LTI/assessment flow. Do not rely on a raw live volume copy as the only recovery mechanism.

Upgrades

Start from the next versioned GitHub Release. Verify its checksum and attestation, preserve existing protected state, and merge new template keys rather than overwriting local configuration.

Target	Command	What the helper protects
Cloud Run	`./upgrade.sh cloudrun.env`	Validates the existing target; requires a successful Cloud SQL backup; records prior traffic; migrates; updates cleanup; stages a no-traffic candidate; checks `/ready` and JWKS; then cuts traffic. With a disabled generated URL, it verifies the custom origin and restores that policy after the candidate.
Compose	`./upgrade.sh .env.secrets`	Backs up and validates PostgreSQL; pulls the exact digest; runs migrations through the dependency gate; restarts; and checks readiness.

Neither helper is an unattended image updater. Review each release, its attestation, migration notes, compatibility, and backup before running it.

Rollback and recovery

Traffic rollback does not roll back the schema

An older application can receive traffic only after you confirm it supports the current migrated schema. Do not automatically run down-migrations.

The Cloud Run bundle writes a protected rollback record. Its rollback command requires that record and --confirm-schema-compatible before changing traffic. When TOOL_URL is configured, rollback verifies that custom origin after traffic is restored. For Compose, restore the earlier immutable image only after the same compatibility review. Schema or data recovery means a forward correction or a restore into a controlled target—not overwriting the active database during diagnosis.

Pool sizing

DATABASE_POOL_MAX is per application process. Reserve connections for migration, cleanup, administrators, monitoring, and restore work:

(maximum app instances × DATABASE_POOL_MAX) + job/admin reserve < max_connections

The maintained Cloud Run configuration uses a pool maximum of 5 and up to 10 app instances, or 50 possible app connections before the reserve. Recalculate both sides before changing capacity or pool settings.

Secret and certificate rotation

Create one new immutable secret version at a time, point the runtime at it, deploy or stage the revision, smoke test, and only then disable the old version. Rotating SESSION_SECRET invalidates sessions; rotating STATE_ENCRYPTION_KEY invalidates outstanding opaque LTI/OAuth state; rotating the LTI signing key requires Canvas and JWKS coordination.

For a certificate-encryption identity, distribute and validate the replacement private identity first, then deploy the replacement public certificate, verify the public-key hash, require fresh configurations, and retire the old client identity after the planned overlap. Do not introduce plaintext compatibility mode as an incident response shortcut.

Monitoring and release smoke checks

Alert on application 5xx and /ready failures, migration/cleanup failures, database connection saturation, storage growth, backup age/failure, stale cleanup, and certificate expiry when encryption is enabled. On a VPS also monitor host security updates, disk capacity, container health, and off-host backup transfer.

After every release, check:

curl -fsS "${TOOL_URL}/health"
curl -fsS "${TOOL_URL}/ready"
curl -fsS "${TOOL_URL}/.well-known/jwks.json"
curl -fsS "${TOOL_URL}/lti/config"
curl -fsS "${TOOL_URL}/js/canvas-seb-detector.js" | head
curl -fsS "${TOOL_URL}/js/canvas-seb-theme-loader.js" | head

Then run the relevant Canvas and SEB acceptance sequence. Public route checks establish service health; they do not prove that Canvas registration, OAuth, a real client, or assessment exit work.