Observability & activity

ContentsVision has two distinct "what happened" systems: pipeline observability (the technical record of a processing job) and the activity log (the human record of actions a user took). This page covers both. For external product analytics, which reuses the activity-log hook to mirror events into PostHog, see analytics.

Pipeline observability

core/observability.py instruments every pipeline run. For each job it tracks timings, token usage, and a rolling log buffer, all keyed by job_id in an in-memory registry.

Stage timing

Each pipeline stage runs inside a stage_timer context manager that logs [STAGE START] / [STAGE DONE] and records the elapsed seconds per stage (extract, transcribe, segment, analyze, synthesize). On failure it logs [STAGE FAIL] with the elapsed time and re-raises. The result is a per-stage breakdown you can read to see where time went.

Token and cost tracking

Every AI call's input and output token counts are accumulated into a TokenUsage for the job, which estimates the dollar cost using the model's per-million-token rates (input and output priced separately). This is the internal cost estimate, entirely separate from customer billing, and is for understanding our own AI spend.

When a job finishes, the pipeline's finalize hook copies these counters onto the job row (processing_jobs.ai_cost_usd, ai_input_tokens, ai_output_tokens, ai_api_calls) so the spend survives restarts and can be aggregated. The admin dashboard's "AI Cost (today)" stat sums ai_cost_usd over jobs created today.

The log ring buffer and the archive

Each job keeps the last 1000 log lines in memory (a deque). A custom logging handler mirrors the pipeline's logger into this buffer, so log lines are queryable over the API without touching disk. Because it is in memory, the buffer is per-process and is lost on restart, so it only serves live streaming and recent inspection.

For long-term storage, every terminal pipeline path (complete or error, all three pipelines) copies the buffer into the job_log_archives table (one JSON row per job, upserted) via services/job_logs.py. Readers go through get_job_log_entries, which prefers the live buffer and falls back to the archive, so a finished job's logs stay readable after a restart. The write is best-effort and never raises, sitting inside the pipeline's finalize hook (_track_job_outcome) alongside the analytics capture. Jobs that finished before the archive existed have no row and return an empty list.

Where it surfaces

Endpoint	Returns
GET /api/jobs/{id}/logs	The job's log lines (live buffer, else the archive).
GET /api/jobs/{id}/metrics	Stage timings, token usage, and cost summary.
GET /api/jobs/{id}/stream	Live progress, log, and final metrics events (SSE).
GET /api/admin/logs	Admin-only: every job system-wide (all teams) with its logs, filterable by team_id, user_id, status (or active), and q (filename / claim number), paginated with limit/offset.

In the UI, the Logs page (app/logs, admin-only) shows every job across all teams with team/user/status filters and grouping, and ProcessingStatus streams logs live during a run. This is the first place to look when a job behaves unexpectedly.

Error tracking (Sentry)

Sentry captures crashes and handled failures across the whole stack. It is OFF by default and everywhere in the test suite: nothing initialises without a DSN, mirroring the PostHog gate.

Backend (core/sentry.py): a fail-safe wrapper in the analytics.py style. init_sentry(source=...) runs at startup in two processes:

• app/main.py (source="api"), before the FastAPI app object is created so the SDK's Starlette/FastAPI auto-integration hooks unhandled request and BackgroundTasks errors;
• app/worker.py (source="worker"), where sentry-sdk's ARQ auto-integration reports failed jobs.

Errors we catch on purpose and convert into product state never reach those hooks, so they report explicitly via sentry.capture_exception(exc, **context):

• the three pipeline failure handlers (video, photo, grouped photo, each tagged with job_id and pipeline);
• the Contents Capture import batch crash handler (tagged with import_batch_id);
• the Adjust Square submission flows in routes/claims.py: a photo upload that exhausts every retry (flow=adjust_square_photo_upload, with item ids and the portal's status code and response body), a failed createItemForClaim during resubmit (flow=adjust_square_resubmit), and a failed triggerFullLookup (flow=adjust_square_trigger_lookup).

The helper is best-effort and never raises, so error reporting can never break the error handling itself.

Privacy: send_default_pii=False on every init (claims carry insured names and addresses), and the user attached to events is local UUIDs only (user_id, team_id), never email or name.

Frontend (@sentry/nextjs): next.config.js wraps the config with withSentryConfig, which injects sentry.client.config.ts into the browser bundle; src/instrumentation.ts loads sentry.server.config.ts / sentry.edge.config.ts for the server runtimes (via experimental.instrumentationHook, required on Next 14). src/app/global-error.tsx is the root error boundary: it reports the crash and offers a reload. The identity bridge in PostHogProvider.tsx sets/clears the Sentry user on login/logout with the same id-only policy. Tracing and session replay stay off in Sentry (OTel owns tracing, PostHog owns replay), so it spends quota on errors only.

Configuration: backend SENTRY_DSN (+ optional SENTRY_TRACES_SAMPLE_RATE), frontend NEXT_PUBLIC_SENTRY_DSN (public, build-time, passed as a compose build arg). Source map upload during the frontend build is optional and only happens when SENTRY_AUTH_TOKEN (+ SENTRY_ORG / SENTRY_PROJECT) is present. See configuration.

Orphaned-job recovery

In Stage 1 the AI pipeline runs as an in-process BackgroundTask. A backend restart (every deploy) kills any in-flight job. The kill arrives as a SIGTERM / asyncio.CancelledError, which is a BaseException, not an Exception, so the pipeline's except Exception handler never runs: the job is never marked error, never refunded, and never reported. It just freezes at whatever percent it had reached (a classic "stuck at 78%"), and it is invisible to both Sentry (no exception caught) and PostHog (no completion event).

services/job_recovery.py closes this. At startup (lifespan in main.py) it sweeps once, and a watchdog re-sweeps every minute for jobs whose row has been quiet longer than STUCK_JOB_MINUTES (default 15):

• Stage 1 (no REDIS_URL): every processing-state job (pending, extracting, transcribing, segmenting, analyzing, synthesizing) is orphaned by the restart that just happened, so on startup each is requeued automatically: partial items are cleared, the status resets to pending ("Restarting after an interruption"), and the pipeline restarts from scratch. This is safe because inputs are on disk and the up-front credit charge is keyed on the job id (no double-billing). One automatic retry is allowed (recovery_attempts); a job interrupted again is failed instead, with a retryable message and a refund, so a poison job cannot loop. Jobs stuck longer than 24 hours are also failed (with a refund) rather than requeued: silently re-running a weeks-old upload would surprise the customer and burn AI spend nobody is waiting on. Operators can also clean up manually from the Logs page (Mark failed / Delete on each job). The watchdog additionally distinguishes a quiet orphan (requeue) from a quiet job whose in-process task is still alive but hung (fail: a requeue would race the running task) using the in-memory log buffer as a liveness signal.
• Stage 2 (REDIS_URL set): the pipeline runs in a separate ARQ worker, so a processing job may be genuinely live. Sweeps are always stale-only with a floor of 30 minutes (matching the worker job_timeout), and reaped jobs are failed with the retryable message rather than requeued, since the API process cannot know whether a worker still owns them.

A failed job's message tells the user the charge was refunded and to retry the upload; the refund goes through the normal refund_for_job path, the failure posts to Slack, and every sweep that touches anything reports its counts to Sentry (capture_message, info level), so deploy-interrupted jobs stay observable.

Shipping to Grafana (OpenTelemetry)

The in-memory buffer above powers the live UI but is lost on restart and cannot connect a failed job to the request that started it. When telemetry is enabled, ContentsVision also exports OpenTelemetry traces, metrics, and logs over OTLP to the Grafana stack (Tempo, Prometheus, Loki), so you can answer "how did this happen?" long after the fact. It is off by default and only turns on when OTEL_ENABLED=true and an OTLP endpoint are set, so local dev and the test suite never ship anything.

core/telemetry.py wires it up at startup. Setup is a no-op when disabled, and the tracer and metric instruments are OpenTelemetry no-op proxies, so the pipeline call sites cost nothing when telemetry is off.

One trace per job

Each pipeline run opens a root span job.process (a background task does not inherit the request context, so this is the trace root) carrying job.id, job.source_type, room.name, claim.id, and loss_type. Every stage runs in a child stage.<name> span, and because httpx and SQLAlchemy are auto-instrumented, each AI call (OpenRouter), transcription (Deepgram), Adjust Square call, and DB query appears as a nested child span with its own timing. A failed stage or job records the exception on the span and sets its status to error. Opening that one trace in Tempo shows the whole job as a timeline: where the time went, which AI call was slow, and exactly where it failed.

Span and attribute reference

Span	Source	Key attributes
job.process	Manual, pipeline root	job.id, job.source_type (video/photo/audio), room.name, claim.id, loss_type, job.status, job.items
stage.<name>	Manual, per stage (extract, transcribe, segment, analyze, synthesize)	job.id, job.source_type, stage.name
HTTP client spans	httpx auto-instrumentation	OpenRouter, Deepgram, and Adjust Square calls, with method, URL host, and status code
DB spans	SQLAlchemy auto-instrumentation	the SQL statement and timing
HTTP server spans	FastAPI auto-instrumentation	one per request, with route and status (the /health probe and /api/telemetry passthrough are excluded)
documentLoad, web-vital.<NAME>, browser.error, fetch spans	Browser web SDK	page-load timing, Core Web Vitals, JS errors, and /api fetches that propagate traceparent into the backend

Exceptions are recorded on the span (span.record_exception) and the span status is set to error, so a failed job is findable with a status filter and its stack trace is on the span. Only IDs are attached as attributes; tokens, emails, and transcript text are never put on spans.

Logs and metrics, correlated

Every stdlib log record is shipped to Loki stamped with the active trace and span id, so from any span in Tempo you can jump straight to that job's logs, and from a log line back to its trace. Alongside the auto-emitted HTTP server metrics, these domain metrics go to Prometheus:

Metric	Type	Labels
contentsvision.jobs	counter	type, status
contentsvision.job.duration	histogram	type, status
contentsvision.pipeline.stage.duration	histogram	stage, type, status
contentsvision.ai.tokens	counter	model, direction
contentsvision.ai.cost_usd	counter	model
contentsvision.items.produced	counter	type
contentsvision.billing.credits	counter	type

Browser telemetry

The frontend runs the OpenTelemetry web SDK (document load, user interaction, fetch, Core Web Vitals, and uncaught JS errors). The fetch instrumentation adds a traceparent header to /api calls, so a browser action and the backend work it triggers are one trace end to end. The browser holds no ingest credentials: it posts OTLP to the same-origin /api/telemetry path, which the backend forwards to the collector with the server-side auth header. Browser telemetry is built into the bundle only when NEXT_PUBLIC_OTEL_ENABLED=true.

Configuration

Variable	Purpose
OTEL_ENABLED	Master switch for backend telemetry.
OTEL_EXPORTER_OTLP_ENDPOINT	OTLP HTTP collector URL (one endpoint for all three signals).
OTEL_EXPORTER_OTLP_PROTOCOL	http/protobuf.
OTEL_EXPORTER_OTLP_HEADERS	Collector auth, comma-separated key=value.
NEXT_PUBLIC_OTEL_ENABLED	Master switch for browser telemetry (baked at build).
APP_ENV, COMMIT_SHA	Stamped on all telemetry as deployment.environment and service.version.

Runbook: "how did this happen?"

Start from whatever you have (a job id, a claim, "everything is slow today") and pivot across the three signals. Grafana's Explore view is the fastest way in.

1. Find the trace (Tempo). Use TraceQL in the Tempo data source:

{ resource.service.name = "contentsvision-backend" && span.job.id = "<job-id>" }
{ name = "job.process" && status = error }          # every failed job
{ name = "stage.analyze" } | select(duration)        # slow analyze stages
{ span.claim.id = "<claim-id>" }                      # everything for a claim

2. Read the trace. Open the job.process span and read its children top to bottom. The stage spans show where the time went; the httpx child spans show which AI or Deepgram call was slow or errored; a red span is the exact failure, with the stack trace on it.

3. Jump to the logs (Loki). Every log line carries the trace id, so from the trace view click through to logs, or query directly:

{service_name="contentsvision-backend"} | json | trace_id="<trace-id>"
{service_name="contentsvision-backend"} | json | level="ERROR"

4. Check the trend (Prometheus). OTLP metric names arrive dotted-to-underscored, and counters gain a _total suffix. Useful queries:

sum(rate(contentsvision_jobs_total{status="error"}[5m])) by (type)         # failure rate by pipeline
histogram_quantile(0.95,
  sum(rate(contentsvision_pipeline_stage_duration_seconds_bucket[5m])) by (le, stage))   # p95 stage time
sum(increase(contentsvision_ai_cost_usd_total[1h])) by (model)             # AI spend per hour
sum(increase(contentsvision_items_produced_total[24h])) by (type)          # output volume

Local development against the LGTM stack

You can run the whole stack locally with the grafana/otel-lgtm all-in-one image and point the app at it, exactly as in production:

# Grafana on :3001 (the frontend uses :3000), OTLP ingest on :4318
docker run --rm -p 4318:4318 -p 3001:3000 -p 3200:3200 -p 9090:9090 grafana/otel-lgtm

Then enable telemetry and restart the services:

# backend/.env
OTEL_ENABLED=true
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318

# frontend/.env.local (for browser telemetry)
NEXT_PUBLIC_OTEL_ENABLED=true

Open Grafana at http://localhost:3001 (user admin, password admin). Run a job and explore Tempo, Prometheus, and Loki. The backend logs Telemetry active: exporting OTLP to ... on startup when it is on.

How it is wired

• Startup. app/main.py calls setup_telemetry(app) at import, before uvicorn serves, so the ASGI instrumentation wraps the app cleanly. It returns immediately when disabled.
• One endpoint. Traces, metrics, and logs all use OTLP HTTP to OTEL_EXPORTER_OTLP_ENDPOINT; the SDK appends /v1/traces, /v1/metrics, /v1/logs. The grafana/otel-lgtm image fans them out to Tempo, Prometheus, and Loki.
• Browser passthrough. The web SDK posts OTLP to the same-origin /api/telemetry path. app/api/routes/telemetry.py forwards the raw body to the collector with the server-side auth header. /api/* reaches the backend in both dev (Next rewrites) and prod (nginx), so no CORS and no credentials in the browser bundle.
• Deployment. docker-compose.prod.yml injects OTEL_* into the backend via env_file: .env, and bakes NEXT_PUBLIC_OTEL_* into the frontend as build args (browser NEXT_PUBLIC_* values are fixed at build time, so flipping browser telemetry on requires a rebuild). See configuration and deployment.
• Safety. Telemetry is a no-op unless OTEL_ENABLED=true and an endpoint are set, so ./run_tests.sh and local dev never ship anything. Tests assert span and metric emission against in-memory exporters (backend/tests/test_telemetry.py).

Troubleshooting telemetry

Symptom	Likely cause and fix
Nothing in Grafana	Telemetry is off. Both OTEL_ENABLED=true AND OTEL_EXPORTER_OTLP_ENDPOINT are required. The backend logs Telemetry disabled on startup when either is missing.
Collector returns 401/403	The collector needs auth. Set OTEL_EXPORTER_OTLP_HEADERS, for example Authorization=Basic <base64 user:pass>.
Backend traces appear, browser ones do not	NEXT_PUBLIC_OTEL_ENABLED is baked at build time. Rebuild the frontend image with the build arg set to true.
Traces appear but no logs or metrics	All three share one endpoint; confirm the collector accepts /v1/logs and /v1/metrics (the otel-lgtm image does).

Activity log

The activity log is the human-facing audit trail: one row per meaningful action, stored in the activity_log table and shown on the Settings page. It is what powers the per-row Restore (undo) button.

Two views read it: the team-scoped GET /api/activity (Settings page, the caller's team only) and the admin-only GET /api/admin/activity (the admin dashboard's Activity tab, all teams, each row tagged with its team). Restore always goes through the team-scoped restore endpoint, so an admin can only restore rows from their own team without impersonating.

What gets logged

Most mutating actions write an entry via services/activity.py: creating a claim or room, uploading media, editing or deleting an item, bulk edits, approvals, merges, photo changes, settings and prompt changes, sign-in and sign-out. Each entry records who did it (user_id, user_name from the Actor), a machine action_type, human-readable action_label and target, and the affected entity.

Restore (undo)

Restorable actions store a snapshot containing everything needed to undo them (for example, an item's fields before an edit, or a deleted item's full row). Restoring replays that snapshot to revert the change and marks the entry restored.

Some actions are deliberately not restorable because they reach outside the system or write files that cannot be cleanly reversed: uploads (files written, pipeline triggered), sign-in and sign-out, and submitting a claim to Adjust Square (it reaches an external platform). Those appear as read-only history.

The two systems compared

	Pipeline observability	Activity log
Records	Technical job processing	Human user actions
Storage	In-memory for the live UI, plus OpenTelemetry export to Grafana when enabled	activity_log table (durable)
Lifespan	In-memory lost on restart; exported telemetry persists in Tempo/Loki/Prometheus	Persistent
Audience	Engineers debugging a job	Users (and support) auditing changes
Undo?	No	Yes, for restorable actions