Skip to main content

Testing & quality

ContentsVision has one firm, non-negotiable rule, written at the top of the repo's CLAUDE.md:

The hard rule

Run the full test suite after any change to the app, and all tests must pass before the work is considered done. A change with no test is incomplete.

This page explains how to run the suite and why it is built the way it is.

Running the tests​

One command, from the repo root:

./run_tests.sh            # run everything
./run_tests.sh -k billing # only tests matching "billing"
./run_tests.sh -x         # stop at the first failure

run_tests.sh activates the backend virtualenv, installs the test-only dependencies once (pytest, pytest-asyncio), and runs pytest from backend/. Any extra arguments pass straight through to pytest.

Offline by design​

The suite must stay fast, offline, and safe to run anytime. Three guarantees make that true (backend/tests/conftest.py):

  1. A throwaway in-memory database. DATABASE_URL is forced to sqlite+aiosqlite:///:memory: before the app is imported, and each test gets a fresh engine. The real dev database at storage/inventory.db is never opened.
  2. No third-party calls. The suite never reaches the AI on OpenRouter, transcription on Deepgram, the Adjust Square API, or PostHog. Anything that needs an external service mocks that layer. PostHog analytics is simply off in tests (no key is set), so core/analytics.py no-ops without even importing the SDK.
  3. A developer's real credentials cannot leak in. conftest.py sets CONTENTSVISION_IGNORE_MACHINE_ENV=1, which drops ~/.contentsvision/local.env from the settings sources entirely (see where a value comes from), and separately blanks MAILGUN_API_KEY, MAILGUN_WEBHOOK_SIGNING_KEY, and SLACK_WEBHOOK_URL as real environment variables, which outrank both .env layers.
This one is not theoretical

The machine-wide file exists so a developer can set a shared local credential once. The first time a working Mailgun sending key was put in it, ./run_tests.sh picked it up and a test run sent live email through the team's Mailgun account, because test_the_adjuster_invites_the_insured exercises the real send path and had until then only ever seen an unconfigured one.

Opting out of the whole file is deliberate, rather than blanking known keys one at a time: it also covers every credential added to that file in future. test_settings_layering.py guards both halves, and asserts directly that insured_email.email_enabled() and support_email.support_email_enabled() are false inside the suite.

The rule this incident illustrates: a test that exercises a real sender must not depend on the sender being unconfigured by accident. If you add an outbound channel, switch it off in conftest.py in the same change.

The test client (conftest.py) seeds a team and a user and mints a valid AdjustSquare-format token with make_test_token, so auth-protected endpoints work without hitting real SSO. The get_db dependency is overridden to share the in-memory engine, so data seeded directly is visible to API calls.

What the suite covers​

The backend/tests/ directory has broad coverage: claim and room APIs, the inventory API, uploads, jobs, billing (unit and API), auth, auto-approve, numbering, the Adjust Square client, observability, prompt settings, the activity log and service, audio and video helpers, audio chunking, and an end-to-end pipeline test with the AI layer mocked. Tests are configured in pytest.ini (async auto-mode, verbose one-line-per-test output driven by each test's docstring).

Conventions​

  • Add or update tests with every change. A feature or fix includes its test.
  • Show real failures. If a test fails, show the actual failing output. Never claim green when it is red.
  • Keep the default suite offline. Anything that needs a real external service must be mocked, or gated behind an explicit opt-in marker so ./run_tests.sh never touches the network.

Mocking the external layers​

Because the pipeline and Adjust Square client are isolated services, tests mock at their boundaries:

  • The AI analyzer functions (claude_analyzer) are patched to return canned items, so the pipeline can be exercised end to end without a real model call.
  • The Adjust Square client functions are patched to capture the arguments they are called with (for example, asserting the right bearer token and adjuster id are passed) and to return fake responses.

This keeps tests deterministic and instant while still covering the real control flow. For the broader engineering conventions (UI, motion, writing style), see conventions.

Last updated on