Skip to main content

Troubleshooting runbook

A symptom-first guide for support and operations. Each entry lists what the user sees, the likely cause, and where to look or what to fix.

Sign-in and access​

Everyone is suddenly logged out / every request returns 401​

  • In production: ADJUSTSQUARE_JWT_SECRET is unset or wrong, so no token validates. The backend logs a loud auth.secret_missing error on startup.
  • In local mode: check that APP_ENV=local and LOCAL_AUTH_USER_EMAIL are set correctly in .env. The backend logs auth.local_bypass when bypass is active.
  • Where to look: backend logs for auth.jwt_error, auth.secret_missing, or auth.local_bypass.

A user sees "Sign-in failed" on the callback page​

  • Likely cause: the token did not validate (/api/auth/me returned 401), often a secret mismatch or an expired token.
  • Where to look: the callback page shows the error detail outside production; backend logs show the reason.

A user cannot see a claim they expect (403 / not found)​

  • Likely cause: the claim belongs to a different team. Access is team-scoped.
  • Where to look: confirm the user's team (/api/auth/me) matches the claim's team. This is by design, not a bug.

Uploads and processing​

A large video upload fails, hangs, or never creates a job​

  • Likely cause: the file exceeds a CDN/tunnel request-body cap. Cloudflare's tunnel rejects bodies over ~100 MB on Free/Pro, well below the app's MAX_UPLOAD_SIZE_MB (2 GB). If no job appears in /api/jobs, the upload never reached the backend.
  • Mitigation (built in): the frontend uploads video and audio in small resumable chunks (POST /api/uploads/init, then PUT .../chunks/{i}, then complete), so each request stays under the cap. Confirm the deployed frontend is current. See the chunked upload protocol.
  • If it still fails: test a small clip to isolate size; check docker compose logs nginx backend for 413; verify MAX_UPLOAD_SIZE_MB and nginx client_max_body_size both allow the size.

A job is stuck or "frozen"​

  • Self-healing (already in place): pipelines run in the backend process, so a restart (e.g. a deploy) used to freeze in-flight jobs at their last progress forever. Now services/job_recovery.py fixes this automatically: on boot, every job still in a processing status is requeued and restarts from scratch (partial items are cleared first; the up-front charge is keyed on the job id, so no double-billing). A watchdog also sweeps every minute for jobs whose row has been quiet for STUCK_JOB_MINUTES (default 15) and requeues them, or fails them if their in-process task hung. A job only gets one automatic retry; a second interruption fails it with a clear message, refunds the charge, and posts the failure to Slack.
  • Where to look: the job status and the Logs page (/logs, admins only; system-wide with team/user filters), or GET /api/jobs/{id}/logs. Finished jobs keep their logs in the job_log_archives table, so the detail survives restarts. Map the status to a stage using the table in the user journey. A job that says "Restarting after an interruption" was recovered by the sweep (recovery_attempts on the job row counts the retries).
  • Likely causes by stage:
    • analyzing for a long time: this is the slow stage; large videos take minutes. Check for OpenRouter rate-limit retries in the logs.
    • transcribing: Deepgram is slow or the audio is long.

A job ended in error​

  • Where to look: the job's error_message and the pipeline logs (the failing stage is logged as [STAGE FAIL]).
  • Common causes: FFmpeg missing or a corrupt upload (extract stage), an unreadable image (photo upload), or an AI call exhausting its retries.

Photos uploaded but nothing is processing​

  • Likely cause: photo jobs wait in awaiting_grouping. The user must group photos and press Process; AI does not start automatically.
  • Fix: finish grouping in the modal and start processing. See the photo pipeline.

"Upload failed" when uploading many or large photos​

  • Likely cause: historically the frontend sent every selected photo in one request, and the Cloudflare Tunnel caps a single request body at roughly 100MB. A big set of camera JPEGs blew past that and Cloudflare returned an HTML error page the UI could not parse, so it showed a bare "Upload failed".
  • Fix (already in place): photos now upload in size-bounded batches (80MB budget), the first to POST /api/upload-photos and the rest to POST /api/upload-photos/{id}/append, all on one job. A single photo over the 100MB hard limit is caught in the browser with a clear message. See batched photo upload.

The inventory is "visual only" with no spoken details​

  • Likely cause: no DEEPGRAM_API_KEY, so transcription was skipped (the pipeline logs a warning and proceeds visual-only).
  • Fix: set the Deepgram key.

Progress bar never updates in the UI​

  • Likely cause: the SSE stream is being buffered. In production this is handled by nginx (proxy_buffering off); in dev the local SSE route handler owns the stream.
  • Where to look: confirm nginx.conf is the deployed one and the cloudflared/nginx containers are healthy. See deployment.

Output quality​

Items look wrong, duplicated, or missing​

  • Likely causes:
    • Quiet or unclear narration: transcription drives naming. Encourage clear narration.
    • Photos not grouped: ungrouped same-item shots can become separate items.
    • Low-confidence items never reviewed: items below 0.6 arrive flagged and need a human.
  • Where to look: the item's confidence and evidence fields, and the job transcript.

Structural items appear (or expected items are excluded)​

  • By design: the AI is contents-only. It excludes walls, floors, built-in plumbing, and building systems, and includes movable property, cabinets, and appliances. A team can nudge behavior with a custom prompt add-on.

Duplicate items across a chunk boundary in a video​

  • Mechanism: cross-chunk dedup merges near-boundary duplicates within VIDEO_DEDUP_WINDOW_SECONDS. Genuine duplicates far apart in time are not merged. Reviewers can merge them manually.

Submission to Adjust Square​

"Your team has no Adjust Square API token" (409)​

  • Likely cause: with per-user auth, the team's team_token is missing (it is set from the SSO login token).
  • Fix: ensure the user logged in via SSO so the token was captured; confirm the team record has a team_token. See the Adjust Square integration.

Submission returns 502​

  • Likely cause: Adjust Square rejected the request or was unreachable.
  • Where to look: backend logs for the AdjustSquareError status and body. Confirm ADJUST_SQUARE_BASE_URL is set and reachable.

Some items submitted, some did not (re-submission)​

  • By design: re-submission posts items one at a time; a failed item stays unsubmitted so it can be retried, while the rest proceed. Re-run the submission for the stragglers.

Photos did not appear on the Adjust Square claim​

  • Likely cause: photo upload happens in a background task after the items are created; a failure there is logged but does not fail the submission.
  • Where to look: backend logs for photo upload warnings. Re-submitting re-attempts only unsubmitted items, so badly-failed photos may need manual attention.

Deployment and ops​

A deploy comes up unhealthy​

  • Where to look: docker compose -f docker-compose.prod.yml logs -f backend. The frontend waits for the backend's /health to pass.
  • Common causes: a missing required env var (AI key, JWT secret) or a bad .env.

After adding a model column, production errors on that column​

  • Cause: create_all only creates whole new tables; existing tables need an ALTER TABLE line in init_db().
  • Fix: add the ALTER TABLE ... ADD COLUMN statement. See data model.

Data loss after a rebuild​

  • Cause: the SQLite database and per-job files live in the storage (and uploads) Docker volumes. If those volumes were removed, data is gone.
  • Prevention: back up the storage volume regularly.

Where the signals live​

SignalWhere
Live job progress and logs/logs page, GET /api/jobs/{id}/stream, GET /api/jobs/{id}/logs
Job timings, tokens, costGET /api/jobs/{id}/metrics
Durable traces, metrics, and logsGrafana (Tempo / Prometheus / Loki) when telemetry is enabled
Who changed what (and undo)Settings activity log, GET /api/activity
Backend errorsdocker compose logs backend / uvicorn output
LivenessGET /health

Reconstructing a past job after a restart​

The in-memory /logs buffer is lost on restart, but when telemetry is enabled the full history persists in Grafana. Open the job.process trace in Tempo (search by job.id or claim.id), read the stage timeline, and follow the trace id into Loki for the log lines. Step-by-step queries are in the observability runbook.

Last updated on