Testing

Tests cover every documented input partition, state transition, failure boundary, and publication invariant. “All cases” means all contract classes below, not every possible query string or image byte sequence.

Test layers

Do not add a large test framework before the built-in runners become insufficient. Test data must be tiny, deterministic, redistributable, and free of production or private feedback records.

Ingestion matrix

Network security tests use a controlled local server and resolver stub. They must not contact public sources.

Enrichment and artifact matrix

Run decoder and OCR failure cases inside the same operating-system limits intended for production ingestion.

Retrieval matrix

The golden corpus contains distinct items for these searches:

Also test:

• FTS punctuation, quotes, parentheses, operators, wildcard characters, and Unicode as literal input;

• exact, dense, and hybrid routes independently;

• route weight, candidate-depth, and threshold boundaries;

• stable ties and stable IDs across rebuilds;

• exact duplicate collapse and template-group caps;

• deleted, unavailable, unlicensed, unsafe, and wrong-language exclusion;

• filtered raw top candidates followed by a deeper eligible item, which must refill into the returned list;

• empty corpus and fewer-than-limit result sets;

• dense failure with explicit lexical-only degradation;

• corrupt FTS or manifest causing startup failure rather than changed rankings;

• stale request IDs never replacing a newer UI state.

Protocol matrix

Golden NDJSON fixtures cover:

• compatible and incompatible handshakes;

• valid search and result messages;

• empty result arrays;

• recoverable request error and fatal worker error;

• unknown message type or field policy;

• missing, duplicated, and mismatched request IDs;

• invalid JSON, partial line, oversized line, unexpected stdout text, and premature EOF;

• startup timeout, query timeout, clean shutdown, forced termination, and one explicit restart;

• diagnostic standard error that cannot corrupt protocol output.

Both languages decode the same golden messages. Protocol-version changes require new fixtures rather than silently accepting an incompatible peer.

TUI matrix

Pure state and TestBackend tests cover:

One PTY smoke test launches the real binary, searches, navigates, selects, quits, and verifies that raw mode, cursor visibility, and the alternate screen are restored. A second path kills the worker or triggers a panic and verifies the same restoration.

Evaluator matrix

Small hand-calculated fixtures cover:

• nDCG with perfect, reversed, short, tied, empty, and zero-ideal lists;

• ExactMRR with a grade-3 result at each rank and with none;

• Hit@10 and CorrectEmpty numerator/denominator boundaries;

• errors and timeouts receiving no retrieval credit;

• per-slice macro averaging independent of slice size;

• normalization clipping at each Technical Score boundary;

• every hard gate, score cap, and critical-failure path;

• paired bootstrap determinism under a fixed seed;

• score/report schema and benchmark-version mismatch.

Interaction-event matrix

Test that:

• feedback is off by default and no file is created;

• enabling and disabling take effect immediately;

• the file is owner-only where supported, expires events after 30 days, and supports immediate deletion;

• preview, open, copy, choose, reformulate, and abandon are distinguishable, with only choose counted as success;

• shown item IDs and positions match the rendered list;

• each launch has a new random session ID and no stable user or machine ID;

• raw normalized queries appear only after opt-in and are never replaced with a misleading "anonymous" hash;

• duplicate event IDs are rejected during analysis;

• no chat context, clipboard content, or stable user ID is stored;

• retention cleanup deletes expired local events;

• events from incompatible dataset or retriever versions are not merged silently.

Raw interaction events do not alter retrieval in any test. A later learning experiment receives its own train/evaluation isolation tests.

End-to-end fixture

The first required fixture is:

local cleared manifest
    -> Rust accepts valid text/image items and quarantines invalid ones
    -> second import is idempotent
    -> Python builds and validates FTS plus embeddings
    -> snapshot publishes atomically
    -> worker handshake succeeds
    -> Rust TUI submits "john cena"
    -> expected template group appears by rank 3
    -> navigation selects the expected stable item ID
    -> optional `choose` interaction records the shown rank only when enabled
    -> clean exit restores the terminal

The fixture must run offline and produce the same canonical content identity and rankings in the declared environment.

Performance regression checks

The fixed performance profile measures:

• worker cold start and model/index load;

• engine p50, p95, and p99 search latency;

• end-to-end query-submit-to-results-render latency;

• concurrency-one search throughput;

• steady and peak resident memory;

• SQLite, vector, and total corpus index size;

• ingestion items per second, p95 item time, and peak memory;

• full and incremental rebuild duration.

Use one declared local PTY and terminal backend at an 80x24 viewport, result limit 10, and the frozen corpus/model. Start the end-to-end clock when the TUI accepts the submit key and stop it only after the backend completes drawing the status plus the entire returned list. Repeat every benchmark query equally in shuffled blocks. Run 100 warm-ups followed by at least 1,000 measured searches with result caching disabled. Charge timeouts their full limit and count them as errors. Performance tests report the terminal, hardware, runtime, corpus, and model versions; they are not ordinary unit tests on every commit.

Release gate

A release candidate must pass:

1. Rust and Python unit suites;

2. shared protocol and migration fixtures;

3. ingestion, build, and rollback integration tests;

4. the offline end-to-end fixture;

5. TUI state tests and PTY terminal-restoration smoke tests;

6. benchmark integrity checks and Technical Score gates;

7. rights, provenance, safety, deletion, and artifact-integrity gates.

The exact runnable commands will be added with the first implementation. Documentation-only status must not imply that these tests already exist.

← Back