Skip to main content

The AI pipeline

The pipeline is the heart of ContentsVision. It takes an uploaded file and produces inventory items. There are three pipelines, one per media type, all orchestrated from backend/app/services/pipeline.py and all sharing the same supporting services. This page traces each one stage by stage.

Where this lives

pipeline.py exposes four entry points: run_pipeline (video), run_grouped_photo_pipeline (photos, the live path), run_audio_pipeline (audio), and the older run_photo_pipeline (legacy batched photos). Each is launched as a FastAPI background task from the upload or process route.

The shared shape​

Every pipeline follows the same rhythm: prepare the input, run AI analysis (in parallel where possible), then synthesize and save items. Each stage is wrapped in a stage_timer from observability so timings and token usage are recorded, and each stage updates the job's status, progress_pct, and current_stage so the UI can stream progress.

Tunable knobs​

The pipeline's behavior is controlled by settings in backend/app/core/config.py (all environment-overridable). The defaults below are tuned together; the configuration reference lists every variable.

SettingDefaultEffect
frame_extraction_fps1.0Frames pulled from video per second.
chunk_duration_seconds30.0Length of each time chunk analyzed by one AI call.
frames_per_minute60Target frame density used when selecting frames per chunk.
max_frames_per_call60Hard cap on frames sent to a single vision call.
video_dedup_window_seconds3.0How close in time two same-named items must be to merge across a chunk boundary.
max_photos_per_call8Photos per call in the legacy batched photo pipeline.
MAX_PHOTOS_PER_ITEM20Hard cap on photos grouped into one item.
openrouter_concurrency6Max simultaneous AI calls per job (one job's own semaphore).
openrouter_global_concurrency16Max simultaneous AI calls across all in-flight jobs in the process. A shared limiter (in claude_analyzer) wrapping every OpenRouter call, so several concurrent jobs share provider capacity smoothly instead of each firing openrouter_concurrency and tripping 429s.
openrouter_modelanthropic/claude-opus-4-5The vision and analysis model.
openrouter_temperature0Sampling temperature for every AI call. Pinned to 0 for deterministic, repeatable results (AS-719).
deepgram_modelnova-2The transcription model.

Video pipeline​

run_pipeline(job_id, db) runs five stages. Progress percentages are shown so support can map a stuck job to a stage.

Stage 1: Extract​

FFmpeg probes the video for duration, fps, and resolution, extracts the audio track to a file, and extracts frames at frame_extraction_fps (1 per second by default). The duration is stored on the job and later drives billing.

Stage 2: Transcribe​

The audio is sent to Deepgram Nova-2, which returns word-level timestamps. Nova-2 is used instead of Whisper because it is far more accurate on brand and product names, which is exactly what the analysis depends on.

The transcript is stored on the job (full text plus per-segment data with word timings). The pipeline also extracts room cues from the narration ("now we're in the master bedroom"), though in the current design each video upload is already scoped to one known room.

Transcription is Deepgram, not Whisper

The running code transcribes with Deepgram Nova-2 (audio_processor.transcribe_audio), and the DEEPGRAM_API_KEY is what matters in practice. OpenAI and Whisper are not used anywhere in the pipeline; the legacy OPENAI_API_KEY and whisper_model config have been removed.

Stage 3: Chunk​

The video timeline is divided into fixed-length chunks of chunk_duration_seconds (30s) by video_processor.compute_chunk_ranges (a pure, unit-tested function). Each extracted frame is assigned to the chunk whose time range contains it. Smaller chunks mean more, smaller AI calls that overlap and finish sooner.

Stage 4: Analyze (the parallel, expensive stage)​

This is where items are actually found. All chunks are launched at once, bounded by a semaphore of size openrouter_concurrency (6), and results are collected as each finishes so progress streams live (55% to 85%).

For each chunk, the pipeline:

  1. Selects frames by density (select_frames_by_density): rather than sending every frame, it picks the best frames up to frames_per_minute/max_frames_per_call, choosing the sharpest frame in each time bucket.
  2. Slices the transcript to just this chunk's time range, so the model sees the narration that belongs to these frames.
  3. Calls Claude vision through OpenRouter (claude_analyzer.analyze_room_segment) with the active system prompt for video.

After all chunks return, cross-chunk dedup merges near-boundary duplicates: if the same item name appears in two adjacent chunks within video_dedup_window_seconds (3s) of the boundary, it is treated as one physical object (a couch straddling the 30s line), not two. The merge is conservative and never collapses items that are far apart in time.

Stage 5: Synthesize​

Items are sorted by video time so numbering matches recording order. For each item the pipeline crops the best frame to the item's bounding box (a close-up) plus a wider context image, writes them to the job's photos/ directory, and saves an InventoryItem with its ItemPhoto rows. Item numbers continue from the claim's current maximum (see numbering). Items with confidence below 0.6 are flagged.

Finally, auto-approval runs if enabled, billing is finalized (video minutes times the per-minute credit rate, recorded for the usage meter), and the job is marked complete. When prepaid billing is enabled (ENABLE_BILLING), the upload did not start the pipeline at all: the job parked in awaiting_confirmation until the user confirmed the quoted cost, and the credits were spent up front at that confirmation (the spend is the gate). So nothing more is charged here; if the job instead fails, that up-front charge is refunded in the error path. All no-ops otherwise. See billing.

Photo pipeline​

Photos work differently from video, and the difference is deliberate. Photos are distinct images, not near-identical frames, and the old approach of letting the AI guess which photos showed the same item produced noisy, duplicate-heavy output. The live pipeline removes that guessing.

How it flows (run_grouped_photo_pipeline):

  1. Upload (POST /api/upload-photos, then POST /api/upload-photos/{id}/append for the rest): every photo is normalized to JPEG and saved; one StagedPhoto row per file, each starting in its own group. The job sits in awaiting_grouping. AI does not start automatically. The frontend splits a large selection into size-bounded batches to stay under the Cloudflare Tunnel's per-request body cap, but all batches land on one job (see batched photo upload).
  2. Group: the user merges photos that show the same item (capped at MAX_PHOTOS_PER_ITEM). The grouping is persisted via the staged-photo regroup endpoint.
  3. Process (POST /api/jobs/{id}/process): the user starts analysis. Soft-deleted photos are removed from disk first.
  4. Analyze: one AI call per group runs in parallel. A multi-photo group uses the "merged" prompt and yields exactly one item; a single-photo group uses the "unmerged" prompt and may yield several items.
  5. Synthesize: items are saved with every source photo attached as ItemPhoto rows. No cross-batch "is this the same item?" guessing is needed.

Described items (AS-705): the user can name an item during grouping, and a named item skips the AI entirely. The analyze step keys off manual_name: if a group has one, the pipeline builds the item straight from the photos, name, and quantity (confidence=1.0, the whole photo as its image) with no AI call, so it costs no credit. This covers both a whole photo / merged group the user named in the Items panel and a named drawn segment. The user sets the name and quantity via the staged-photos/details endpoint.

Manual segmentation (AS-705): during grouping the user can also draw boxes around items inside one photo (the segment endpoint crops each box into a StagedPhoto with is_segment=True, replacing the original). A segment is always a known item: the pipeline never asks the AI to re-split it, and the user's quantity overrides whatever count the model returns. A named segment skips the AI (as above); a blank-named segment is identified by the AI via the "merged" prompt like any known item, with the user's quantity forced onto the result.

Billing for photos is one request per group that hits the AI, not per photo. Described (named) items make no request and are free. See billing.

Merged vs unmerged prompts

The two photo flows use two different system prompts (photo_merged and photo_unmerged), each separately customizable by a team. See automation and prompts.

Audio pipeline​

run_audio_pipeline handles voice memos and call recordings. No frames, no photos: just transcribe and read.

The analyzer records, for each item, the audio_start_seconds and audio_end_seconds of the part of the recording that mentions it, so Guided Review can play back just that slice. If the recording has no speech, the job completes cleanly with zero items and a friendly message rather than failing. Audio is billed per minute, like video.

The vision call​

All analysis goes through claude_analyzer.py, which calls OpenRouter's OpenAI-compatible endpoint (a single routing layer with model fallback and rate-limit headers).

  • Model: openrouter_model, default anthropic/claude-opus-4-5.
  • Sampling: every call pins temperature to openrouter_temperature (default 0) so the same video or photo set yields the same items and names run-to-run (ticket AS-719). With no temperature the model defaults to ~1.0 and samples differently each run, which is what made repeated uploads produce different item counts and names. Temperature 0 makes output near-deterministic; it is not a bit-identical guarantee (provider batching, routing, and floating point still vary, and Anthropic models do not honor a seed). Only temperature is sent, never top_p alongside it, since Claude 4.x rejects both at once.
  • Concurrency: a shared, connection-pooled httpx client; the pipeline limits in-flight calls with a semaphore.
  • Retries: up to 6 attempts per call. On HTTP 429 it honors Retry-After or x-ratelimit-reset; on 5xx and timeouts it uses exponential backoff with jitter.
  • Inspection: every call's final outcome is recorded to the api_request_logs table (request JSON with images redacted, full response, attempts, duration, tokens, real cost) and browsable on the admin Logs page's API Requests tab.
  • Structured output: every prompt ends with a locked OUTPUT FORMAT: block defining the JSON contract the app parses. Custom prompt add-ons are spliced in before that block, so the contract is always preserved.

The built-in system prompts encode the domain rules: be thorough, name items like searchable retail product names, put brand in its own field, document damage, group identical items, include movable property plus cabinets and appliances, and exclude structural building parts entirely. The condition scale is fixed (New, Above Average, Average, Below Average), and confidence is scored by how clearly an item is seen and whether it was named in narration.

Age is only ever taken from something a person said. An estimated age is unreliable and drives item value, so the model may set age_years / age_months only when the age is stated aloud, and never from wear, styling, or technology generation. The video and audio prompts keep the age fields but gate them on a spoken statement (video additionally requires the stating words to appear in audio_evidence). The three photo prompts drop the fields entirely, since a photo set has no narration to quote. analyze_room_segment enforces this by mode: mode="photo" hardcodes both fields to 0 regardless of the response, so a team's custom prompt add-on or an off-format reply cannot seed a guessed age; mode="video" and _audio_item_from_json pass a stated age through. Anything unstated stays 0 for the reviewer to fill in.

Item numbering​

Numbers are unique and sequential within a claim and continue across every job in that claim (numbering.py). AI items are numbered in video-time order. Manual inserts can append (max + 1) or insert between numbers, shifting later items up by one.

Error handling​

Every pipeline wraps its work in a try/except. On any failure the job is set to status=error with the exception type in current_stage and the message in error_message, and the failure is logged to the job's observability buffer. Partial work already committed (for example, items from chunks that succeeded) is kept. The Adjust Square submission path has its own retry and per-item failure handling, covered in its own page.

Next​

Last updated on