Library map

Markdown · Agent index

The library map groups ideas from accepted digests into named concepts, with a 2-D layout the web library draws. The device that leads ingest builds it locally. It uses EmbeddingGemma for idea vectors and the sponge setup --digest model for names, then syncs the result as one immutable object. Building it needs no network and no cloud compute.

Viewing a built map works in every web library. Building one needs the embedding helper, which runs only on Apple Silicon Macs with macOS 26 and is installed by sponge setup --embedding; see Requirements and platforms.

Commands

• sponge map shows when the newest map was made and how big it is. It also says how many accepted digests changed since this device last built, and whether a rebuild is due.
• sponge map build [--force] builds now. Without --force, it does nothing when the input is unchanged.
• Both accept --json.
• A build fails with model-busy while another model server is running, the same as sponge digest --draft.

Inputs

A map is built from items with a current Digest v2 whose gist is accepted. Ideas rejected in review are left out, and the other ideas keep their index in the digest.

Each idea is embedded as a document from its claim + ' ' + detail, capped at 960 UTF-8 bytes. Vectors are cached in idea_vectors, keyed by embedding fingerprint and the sha256 of the text. The cache drops vectors no current idea uses.

Cap: a map covers at most the newest 12,000 ideas, taken a whole item at a time, newest saved first. The kNN step is O(n²·d), which is about two minutes at the cap. Older items beyond the cap are not mapped.

Steps

1. Vectors and graph.
   • Vectors are mean-centred, then L2-normalised.
   • The kNN graph uses k = 10 with a cosine floor of 0.35.
   • It never links two ideas from the same item.
   • Edges are symmetric and keep the larger weight.
2. Concepts.
   • Louvain runs at resolution 2.5. It visits nodes in index order, so the result is deterministic.
   • A concept must span at least 3 distinct items.
   • Concepts are ordered by distinct item count. Members are ranked by closeness to the concept’s centroid. Each member’s ideas are listed strongest first.
3. Ids and names.
   • Each new concept is matched against the previous map’s concepts by item-set Jaccard (≥ 0.5, greedy best-first). A match keeps the old concept’s id, name and description.
   • An unmatched concept’s id is the first 16 hex characters of the sha256 of its sorted idea keys (item#index).
   • Unmatched concepts are named by the local model at temperature 0, seed 1.
   • If the model’s reply is unusable, the concept is named from its top c-TF-IDF terms.
4. Links.
   • Each concept links to its top 3 neighbours by centroid cosine + 0.15 × shared items.
   • Links with a weight of 0.12 or less are dropped.
   • Weights are then normalised into (0, 1].
5. Regions.
   • Louvain runs on the concept link graph.
   • A region of 3 or more concepts gets a one- or two-word name from one model call. If it matches a region in the previous map, it keeps that region’s name instead.
   • Any other region’s name is null.
6. Layout.
   • A seeded force layout (seed 7) is scaled to [0, 1] with a 0.06 margin.
   • An item sits at the mean of its concepts, offset by a golden angle, and is clamped to [0, 1].
   • A region sits at the mean of its concepts.
   • Coordinates are rounded to 4 decimal places.

The model server starts only if some concept or region needs a new name.

Identity and storage

• input_hash is the sha256 of canonical JSON. The JSON holds:
  • the schema version;
  • every shaping parameter (mapParams);
  • the embedding fingerprint and the namer pin;
  • the sorted [itemId, digestDerivationId, ideaIndexes] of every included digest.
• A build is skipped when input_hash equals the newest map’s.
• id = libraryMapId(input_hash, output_json) = sha256(JSON.stringify(['library-map-v1', inputHash, output])).
• output_json is canonical JSON (sorted keys, finite numbers). It must pass checkLibraryMap and stay within 8 MiB.
• producer_json is a JSON object of at most 4096 bytes.
• created_at is an ISO time of at most 64 bytes.
• Rows live in library_maps (migration 016). Each device keeps the newest two.

Sync

• Export writes only the newest map, as {kind:'library-map',version:1,map:<row>} at objects/<sha>.json.
• On import, checkLibraryMapRow checks the row:
  • exact columns, hex hashes and schema_version 1;
  • the producer, canonical output and recomputed id;
  • checkLibraryMap on the output.
• The row is then inserted unless already present, and the newest two are kept.
• Maps are never recorded in sync_observations.

Schedule

The worker runs the map step after the digest drain, under the same guards:

• no pending cleanup;
• digests are not deferred to another device;
• no digests are due.

A build is due when the accepted-digest set changed since this device’s last build, and either 5 or more entries changed or 24 hours have passed. The worker builds at most once every 6 hours. Model and runtime errors postpone the next try:

Error	Postponed for
model-busy	5 minutes
model-unavailable	30 minutes
runtime-missing	1 hour
anything else	6 hours

The due check reads only the store, so the worker starts no model while nothing has changed.