Data pipeline
The data pipeline converts permitted local source material into a versioned searchable corpus. Rust owns local intake and the first untrusted-input boundary. Python owns deterministic derived-index construction, validation, and publication; remote acquisition and model enrichment are added only when a later stage needs them.
Pipeline
Phases 1 and 2 implement the released corpus-build path; Phase 3 consumes the published result without changing it:
cleared local JSONL manifest and fixture media
-> versioned JSONL source envelope
-> Rust unresolved-manifest gate, bounded parsing, rights checks, decode, hashing, and deduplication
-> content-addressed media + canonical/provenance SQLite rows
-> Phase 1 Python deterministic fielded FTS5 build
-> Phase 2 Python TF-IDF/LSA experiment artifacts
-> integrity, coverage, and checksum validation
-> atomic active.json publication
-> read-only Python worker and Rust TUI
Phase 2 adds a model-free TF-IDF/LSA representation and retrieval artifacts. The paused Phase 4 source design would put a reviewed Commons plan before this same ingest boundary and would supply searchable fields without OCR or generated annotation. Any later enrichment must earn its own measured need and reuse the Phase 1 corpus contract rather than replace it.
Only one build stage writes at a time. The retrieval worker opens the published snapshot read-only.
Implementation status
Phases 1 through 3 are complete and the project is paused. The released ingester accepts bounded local JSON Lines plus static PPM images. It does not implement a Commons client, version-2 deletion records, JPEG/PNG intake, active-snapshot invalidation after deletion, or controlled network-source tests.
Phase 3's optional interaction log is deliberately outside the corpus and snapshot directories at <data-dir>/feedback/interaction-events.jsonl; it cannot change index artifacts or live rankings. Not implemented: OCR, generated captions, pretrained or image embeddings, or thumbnails. The current dense vectors are derived only from stored text fields with TF-IDF and truncated SVD.
Source acceptance
The current product starts with a manually curated manifest and assets whose storage and use are known. If development resumes, the first external source is limited to a finite human-reviewed Wikimedia Commons plan, not a category crawler. Its planned contract is in Wikimedia Commons source.
Every source configuration must record:
API and automated-access permission;
retention, caching, and redistribution rules;
creator, licence, permission, and required attribution;
whether indexing or model training is allowed;
request limits and retry policy;
edit, deletion, and takedown handling;
date on which the source terms were reviewed.
A URL exposed by an API is not proof that its contents may be copied or redistributed. Rights-incomplete items are recorded and quarantined, never published.
Potential sources must be evaluated individually. The paused Phase 4 design considers only the Wikimedia Commons API, and every planned item still needs its own pinned licence and attribution evidence. No external source is implemented. Reddit, Openverse, Imgflip, GIPHY, Tenor, and Know Your Meme have distinct API, storage, ranking, or automated-access restrictions; none is a default crawler target.
Data layers
| Layer | Contents | Rule |
|---|---|---|
| Raw | Permitted source payload, original text, media bytes or external reference, fetch metadata, and checksum | Immutable; a changed fetch creates a revision |
| Canonical | Stable meme identity, searchable metadata, provenance, rights, availability, safety, and processing state | Changed only by a versioned processing run |
| Derived | Search documents, FTS tables, embeddings, ordered IDs, thumbnails, and evaluation artifacts | Rebuildable from raw and canonical inputs |
These are layers of one corpus. Only canonical records are authoritative; the search index can always be rebuilt.
Source envelope
Each source integration maps input into the same bounded envelope without inventing missing source evidence:
| Field | Contract |
|---|---|
schema_version | Cross-language record version. |
source, source_item_id | Source name and stable ID; the pair is unique. |
kind | text or static image in v1. |
source_url, media_url, asset_path | Original page, optional provider media URL, and Phase 1 path relative to the manifest directory. |
fetched_at, source_updated_at | Retrieval time and optional provider revision. |
raw_payload_path | Retained payload or null when retention is forbidden. |
claimed_media_type | Optional declaration; never trusted without inspection. |
title, text, tags | Original title, text-item content, and source tags preserved without search normalization. |
language, safe, reviewed | Required Phase 1 language and explicit safety/review classifications. |
people, template | Source-provided identity metadata, including names such as John Cena. |
creator, licence, permission, attribution | Rights evidence required by the source policy. |
retention_policy, redistribution_policy | What may be stored and shown. |
Phase 1 rejects unknown manifest fields instead of silently discarding them. A resumed Phase 4 must retain the bounded Commons API response separately and map only declared envelope fields. The canonical mapping records whether each value came from the source, a model, or human review.
The released envelope is schema version 1. The paused Stage 4 design proposes a deliberately small version-2 deletion record containing only schema_version, source, source_item_id, source_url, fetched_at, and deleted: true. This contract is not accepted by the current ingester and must ship with its parser, migration behavior, tests, and stale-serving protection.
Rust trust-boundary parsing
Phase 1 local boundary
The local ingester resolves every attempted fixture item to an explicit outcome:
Parse a bounded record and dispatch on
kind; quarantine unknown kinds.Resolve local paths beneath a configured root without following an escape outside it.
Enforce record-byte and decoded-image dimension limits.
For images, compare the declared type with magic-byte detection and bounded static PPM decoding; reject malformed data and dimensions over the configured limit.
For text-only items, require bounded valid Unicode and do not run media checks.
Require stable source identity and complete rights, retention, redistribution, safety, and review evidence.
Calculate a domain-separated SHA-256 digest over the original UTF-8 text or media bytes.
Persist the manifest-path gate, atomically place accepted media in content-addressed storage, sync every newly created directory link, and commit the complete manifest's item, provenance, outcome, and processing-run rows in one SQLite transaction.
No Phase 1 code accepts a URL as media input or performs a network request.
Planned Phase 4 remote boundary
If resumed, the first approved external source must extend the same outcome model. Production accepts only HTTPS requests to the fixed Commons API and original-media host/path, resolves and rejects private, loopback, link-local, multicast, unspecified, and other forbidden addresses, pins the validated addresses for connection, and repeats URL and address validation after every manually followed redirect. It disables ambient proxies and automatic redirects/retries, caps redirects and attempts at three, applies bounded deadlines, and caps API and media bodies.
The planned command sends serial requests with a required contact User-Agent, MediaWiki maxlag=5, bounded Retry-After handling, and a durable plan-hash cursor. Use one reused synchronous HTTP client. A rate-limited source does not justify an async runtime or generic adapter framework.
Outcomes and retries
| Outcome | Meaning | Next action |
|---|---|---|
accepted | New or changed valid item | Queue canonical mapping and enrichment |
unchanged | Same source revision and content identity | Skip without rewriting derived data |
duplicate | Existing canonical content with new provenance | Attach the provenance record |
retryable — planned Stage 4 | Timeout, rate limit, transient DNS failure, or provider 5xx | Bounded backoff without advancing past the item |
quarantined | Invalid schema, forbidden target, MIME/decode failure, missing rights, or policy failure | Retain reason for review |
deleted — planned Stage 4 | Explicit provider deletion, reviewed-plan removal, or permission revocation | Detach provenance, invalidate affected snapshots, and rebuild |
fatal_batch | Schema mismatch, corrupt state, invalid configuration, or failed publication | Stop without advancing the cursor |
A fatal error, read failure, database failure, or process interruption rolls back the whole manifest transaction, so no valid prefix can later become publishable. It also leaves an unresolved marker keyed to the canonical manifest path. Until that same path succeeds, a different manifest cannot ingest and Python cannot publish; this prevents unrelated work from reviving stale content after a failed rights change. The separate failed-run audit row records the attempted outcome counts without making its items visible. A rerun of the fixed fixture must preserve canonical IDs, content checksums, and outcome counts without creating duplicate search items.
Planned Commons acquisition has its own earlier durability boundary: each verified API payload, media file, and mapped record must be atomically written and directory-synced before its plan cursor advances. Plan-removal reconciliation runs only after every remaining item completes, so an interrupted or partial scan cannot invent deletions. The resulting manifest still enters the whole-manifest SQLite transaction above.
Canonical storage
Ordered language-neutral SQL migrations define four initial entities:
| Entity | Required values |
|---|---|
corpus_state | Singleton unresolved-manifest identity used to fail closed across interruption and fatal batches |
meme_item | ID, kind, title, conditional text or asset URI, language, content identity, availability, safety/review state, people, template group, tags, OCR, caption, search description, and processing version |
source_item | Nullable meme-item ID, source identity and URLs, creator, rights and attribution, policies, payload path, fetch revision, outcome/reason, and deletion state |
processing_run | Stage, input/output versions, cursor, timestamps, outcome counts, tool/model versions, and error summary |
Identical domain-separated content shares one meme_item while retaining every provenance row. Identical image bytes also share one media object. A perceptual hash proposes near-duplicate or template groups for review; it never deletes visually similar variants automatically.
The current schema reserves deletion state but the released ingester does not accept deletion records. Stage 4 must make deletion state-idempotent, detach only the affected provenance, preserve a canonical item with another serving provenance, invalidate serving state before removed content can be returned, and require publication before a worker resumes.
Searchable fields remain separate in canonical storage. Do not flatten title, people, template, tags, OCR, caption, and descriptions until building the derived search document; retrieval needs their identity and weights.
Python index build — Phases 1 and 2
The builder remains local and model-free. It:
opens the ingested candidate database and verifies the schema, SQLite integrity, cleared unresolved-manifest state, and latest completed successful ingestion run;
selects only available, safe, reviewed items with complete serving provenance and removes every other item and provenance row from the serving snapshot;
applies the versioned Unicode NFKC and whitespace-normalization function to retrieval copies while preserving original display values;
creates fielded FTS5 rows from title, people, template, tags, source text, reviewed OCR, caption, and description;
builds a deterministic TF-IDF matrix, truncates it to at most eight LSA dimensions, canonicalizes component signs, normalizes item vectors, and writes ordered IDs, vocabulary, IDF, components, and vectors;
verifies serving-row, FTS, and dense-ID coverage, canonical content identity, bounded media checksums, shapes, finite values, norms, and artifact checksums;
writes a complete candidate snapshot and atomically replaces
active.jsononly after validation succeeds.
It does not run OCR, create annotations, download a model, or answer queries. A failed candidate never replaces the active snapshot. LSA is stored even though the Phase 2 comparison rejected it as the default route; keeping the small experiment artifacts makes the negative result reproducible.
Later Python enrichment boundary
Rust validation does not make a file trusted to a second decoder. Run Python enrichment without network access and with bounded dimensions, subprocess timeouts, temporary output paths, and operating-system CPU, memory, and file limits. A crash or limit breach quarantines the item, not the batch.
For accepted items, Python:
applies one versioned Unicode NFKC and whitespace-normalization function to retrieval copies, never to preserved originals;
runs OCR for images and stores raw text, confidence, tool version, and optional human correction separately;
creates a literal visual caption when useful;
normalizes source-provided people, template, and tags without losing original values;
adds a short search description for semantic matching;
records every annotation's source, confidence, and review status;
builds field-labelled lexical and semantic documents;
writes a new embedding set without replacing earlier model revisions.
Example canonical search fields:
title: John Cena "You Can't See Me"
people: John Cena
template: you can't see me
ocr: you can't see me
caption: wrestler waving a hand in front of his face
description: reaction image about being invisible or unnoticed
tags: wrestling, WWE, invisible, hand gesture
Human review is required for benchmark items. Generated annotations elsewhere remain labelled as generated and are never treated as benchmark truth.
Derived artifacts
Every index-build manifest records the applicable values below. Phase 2 records the SQLite/FTS and LSA artifacts, canonical export, schema, normalization, builder, representation, and runtime values.
Each index build manifest records:
dataset, schema, normalization, and search-document versions;
representation type, currently
fielded_fts5+tfidf_lsa;dense method and dimension, vocabulary and item counts, and NumPy version;
SQLite, matrix, ordered-ID, vocabulary, and canonical-export checksums;
builder, SQLite runtime, and compile options.
The row at index i in dense_vectors.npy belongs to element i of dense_ids.json. dense_vocab.json aligns with dense_idf.npy and the columns of dense_components.npy. Worker startup rejects a missing or extra required artifact, unsupported contract version, missing or duplicate ID, dimension mismatch, non-finite value, unexpected norm, ineligible item, or checksum disagreement.
Canonical content identity hashes a UTF-8 JSON Lines export ordered by item ID, including deterministic serving provenance and rights fields, with sorted keys and LF endings. It excludes fetch timestamps and SQLite page layout. Artifact hashes separately protect the actual files.
Atomic publication
Python builds a complete candidate snapshot in a versioned staging directory. Before promotion it must:
check SQLite integrity, migrations, cleared unresolved-manifest state, the latest completed successful ingestion run, runtime version, and required FTS support;
require rights, availability, and safety eligibility for every serving row;
verify FTS coverage and every artifact applicable to the phase, including thumbnail references and item/vector alignment once those artifacts exist;
verify every checksum and the Phase 2 item/vector alignment, shape, finite-value, and norm invariants;
sync the completed snapshot directory before atomically replacing and directory-syncing the active-manifest pointer.
A pre-replacement failure leaves the previous snapshot untouched. The pointer rename is the commit point; a following directory-sync failure is reported as “publication durability unknown” because the new pointer may already be visible and must not be described as rolled back. Stage 4 must add fail-closed invalidation for deletion, rights drift, or serving-provenance changes before remote acquisition can ship. Permitted raw-layer retention remains governed separately by source policy.
Pipeline measurements
Pipeline health is reported separately from search quality:
100% of served items satisfy rights, provenance, safety, availability, and deletion checks;
100% asset integrity and item/vector alignment in the published set;
100% of otherwise eligible canonical items included in the current index;
required metadata completeness by field and source;
zero exact duplicate rows in the serving index;
quarantine, retry exhaustion, near-duplicate, and deletion-lag counts;
accepted items per second, p95 item time, total CPU time, and peak memory;
corpus size, SQLite size, vector size, and rebuild duration.
A future Commons report must additionally record plan identity and scan freshness, outcome counts, retries, downloaded bytes, duration, sequential throughput, peak memory, and resume/reconciliation evidence. A deletion-lag observation is measured from the source timestamp visible to the integration through completed ingestion and publication; it is not a promise of real-time source monitoring.
Rights or integrity failures block publication. Averaging them into a Technical Score would hide a broken corpus.