# Deployment

> Service-level problems — startup failures, /ready never passing, migration and cleanup job failures, and connection saturation.

[Canonical HTML page](https://safeonlineexam.com/docs/troubleshooting/deployment)

These are problems with the running service itself, independent of Canvas. Start
here when `/health` or `/ready` fail.

## The service fails to start [#the-service-fails-to-start]

In a hardened runtime (production, or **any** Cloud Run service), startup fails
before listening if configuration is unsafe or incomplete — this is intentional.

**Check the configuration:**

* All required values are present: database, Canvas, LTI, OAuth, secrets, URL, and
  a valid encryption certificate when encryption is enabled. See
  [Required application values](/docs/deploy/configuration#required-application-values).
* `TOOL_URL` and `CANVAS_DOMAIN` are HTTPS origins with no path/query/credentials.
* `SESSION_SECRET` and `STATE_ENCRYPTION_KEY` are each ≥32 chars and **different**
  from each other.
* `LTI_PRIVATE_KEY` is a valid RSA 2048+ JWK.
* You did not set both a direct secret and its `_FILE` form (that is rejected).
* Hardened runtimes reject `USE_IN_MEMORY_STORE=true` and `APP_DEBUG_ENABLED=true`.
  They require a valid X.509 certificate whenever
  `SEB_CONFIG_ENCRYPTION_ENABLED=true`.

## `/ready` never passes [#ready-never-passes]

`/ready` reflects PostgreSQL reachability **and** whether all checked-in migrations
are applied.

**Check:**

* PostgreSQL is reachable from the service (host, port, SSL mode, credentials).
* The **migration job/service ran and succeeded against this exact image** before
  traffic. A new image whose migrations have not run will fail readiness by design.
* On Cloud Run, the Cloud SQL attachment and socket path
  (`/cloudsql/PROJECT:REGION:INSTANCE`) are correct.
* On Docker, the `migrate` service completed — `app` waits on its success condition.

## Migration job fails [#migration-job-fails]

**Check:**

* **Checksum validation** — migrations are forward-only and checksummed; if an
  already-applied migration was edited, validation fails. Add a **new** forward
  migration instead of editing history.
* The migration ran with the **intended image** and against the correct database.
* On Cloud Run, review the migration job execution logs; the service deploy waits
  for the migration job and will not shift traffic if it failed.

## Cloud Run upgrade cannot verify a tagged candidate [#cloud-run-upgrade-cannot-verify-a-tagged-candidate]

If an upgrade stages a Ready candidate but reports no usable tagged URL, confirm
the old revision still has traffic and inspect the
`run.googleapis.com/default-url-disabled` service annotation. A current release
bundle verifies the configured custom origin, temporarily enables the generated
Cloud Run URL for candidate checks, and restores the disabled policy after cutover
or failure. Do not leave the generated URL enabled as a manual workaround.

## Cleanup job fails or stops running [#cleanup-job-fails-or-stops-running]

Expired sessions, transient state, and locks accumulate if cleanup stops.

**Check:**

* The schedule exists and is firing (Cloud Scheduler job, or the systemd
  timer/cron entry).
* The scheduler identity has invoker permission on the cleanup job (Cloud Run).
* Review the cleanup log/executions. Alert on stale or failed executions.

See [Operations → Scheduled cleanup](/docs/operate/operations#scheduled-cleanup).

## PostgreSQL connection saturation [#postgresql-connection-saturation]

**Check the pool math:**

```text
(max app instances × DATABASE_POOL_MAX) + job/admin reserve < database max_connections
```

Recalculate before changing `DATABASE_POOL_MAX` or the instance count. The Cloud
Run configs use a pool max of 5; ten production instances use at most 50
application connections before the reserve. See
[Pool sizing](/docs/operate/operations#postgresql-pool-sizing).

## Verifying a deployment quickly [#verifying-a-deployment-quickly]

```bash
curl -fsS "${TOOL_URL}/health"
curl -fsS "${TOOL_URL}/ready"
curl -fsS "${TOOL_URL}/.well-known/jwks.json"
curl -fsS "${TOOL_URL}/lti/config"
curl -fsS "${TOOL_URL}/js/canvas-seb-detector.js" | head
```

If these pass but users still fail, the problem is in Canvas or on devices — return
to the [symptom index](/docs/troubleshooting).