Safe Online Exam
Troubleshooting

Deployment

Service-level problems — startup failures, /ready never passing, migration and cleanup job failures, and connection saturation.

These are problems with the running service itself, independent of Canvas. Start here when /health or /ready fail.

The service fails to start

In a hardened runtime (production, or any Cloud Run service), startup fails before listening if configuration is unsafe or incomplete — this is intentional.

Check the configuration:

  • All required values are present: database, Canvas, LTI, OAuth, secrets, URL, and a valid encryption certificate when encryption is enabled. See Required application values.
  • TOOL_URL and CANVAS_DOMAIN are HTTPS origins with no path/query/credentials.
  • SESSION_SECRET and STATE_ENCRYPTION_KEY are each ≥32 chars and different from each other.
  • LTI_PRIVATE_KEY is a valid RSA 2048+ JWK.
  • You did not set both a direct secret and its _FILE form (that is rejected).
  • Hardened runtimes reject USE_IN_MEMORY_STORE=true and APP_DEBUG_ENABLED=true. They require a valid X.509 certificate whenever SEB_CONFIG_ENCRYPTION_ENABLED=true.

/ready never passes

/ready reflects PostgreSQL reachability and whether all checked-in migrations are applied.

Check:

  • PostgreSQL is reachable from the service (host, port, SSL mode, credentials).
  • The migration job/service ran and succeeded against this exact image before traffic. A new image whose migrations have not run will fail readiness by design.
  • On Cloud Run, the Cloud SQL attachment and socket path (/cloudsql/PROJECT:REGION:INSTANCE) are correct.
  • On Docker, the migrate service completed — app waits on its success condition.

Migration job fails

Check:

  • Checksum validation — migrations are forward-only and checksummed; if an already-applied migration was edited, validation fails. Add a new forward migration instead of editing history.
  • The migration ran with the intended image and against the correct database.
  • On Cloud Run, review the migration job execution logs; the service deploy waits for the migration job and will not shift traffic if it failed.

Cloud Run upgrade cannot verify a tagged candidate

If an upgrade stages a Ready candidate but reports no usable tagged URL, confirm the old revision still has traffic and inspect the run.googleapis.com/default-url-disabled service annotation. A current release bundle verifies the configured custom origin, temporarily enables the generated Cloud Run URL for candidate checks, and restores the disabled policy after cutover or failure. Do not leave the generated URL enabled as a manual workaround.

Cleanup job fails or stops running

Expired sessions, transient state, and locks accumulate if cleanup stops.

Check:

  • The schedule exists and is firing (Cloud Scheduler job, or the systemd timer/cron entry).
  • The scheduler identity has invoker permission on the cleanup job (Cloud Run).
  • Review the cleanup log/executions. Alert on stale or failed executions.

See Operations → Scheduled cleanup.

PostgreSQL connection saturation

Check the pool math:

(max app instances × DATABASE_POOL_MAX) + job/admin reserve < database max_connections

Recalculate before changing DATABASE_POOL_MAX or the instance count. The Cloud Run configs use a pool max of 5; ten production instances use at most 50 application connections before the reserve. See Pool sizing.

Verifying a deployment quickly

curl -fsS "${TOOL_URL}/health"
curl -fsS "${TOOL_URL}/ready"
curl -fsS "${TOOL_URL}/.well-known/jwks.json"
curl -fsS "${TOOL_URL}/lti/config"
curl -fsS "${TOOL_URL}/js/canvas-seb-detector.js" | head

If these pass but users still fail, the problem is in Canvas or on devices — return to the symptom index.

On this page