# Library map

The library map groups ideas from accepted digests into named concepts, with a 2-D layout the web library draws. The
device that leads ingest builds it locally. It uses EmbeddingGemma for idea vectors and the `sponge setup --digest`
model for names, then syncs the result as one immutable object. Building it needs no network and no cloud compute.

Viewing a built map works in every web library. Building one needs the embedding helper, which runs only on Apple
Silicon Macs with macOS 26 and is installed by `sponge setup --embedding`; see [Requirements and platforms](/docs/requirements).

## Commands

- `sponge map` shows when the newest map was made and how big it is. It also says how many accepted digests changed
  since this device last built, and whether a rebuild is due.
- `sponge map build [--force]` builds now. Without `--force`, it does nothing when the input is unchanged.
- Both accept `--json`.
- A build fails with `model-busy` while another model server is running, the same as `sponge digest --draft`.

## Inputs

A map is built from items with a current Digest v2 whose gist is accepted. Ideas rejected in review are left out, and the
other ideas keep their index in the digest.

Each idea is embedded as a `document` from its `claim + ' ' + detail`, capped at 960 UTF-8 bytes. Vectors are cached in
`idea_vectors`, keyed by embedding fingerprint and the sha256 of the text. The cache drops vectors no current idea uses.

**Cap:** a map covers at most the newest 12,000 ideas, taken a whole item at a time, newest saved first. The kNN step is
O(n²·d), which is about two minutes at the cap. Older items beyond the cap are not mapped.

## Steps

1. **Vectors and graph.**
   - Vectors are mean-centred, then L2-normalised.
   - The kNN graph uses k = 10 with a cosine floor of 0.35.
   - It never links two ideas from the same item.
   - Edges are symmetric and keep the larger weight.
2. **Concepts.**
   - Louvain runs at resolution 2.5. It visits nodes in index order, so the result is deterministic.
   - A concept must span at least 3 distinct items.
   - Concepts are ordered by distinct item count. Members are ranked by closeness to the concept's centroid. Each
     member's `ideas` are listed strongest first.
3. **Ids and names.**
   - Each new concept is matched against the previous map's concepts by item-set Jaccard (≥ 0.5, greedy best-first). A
     match keeps the old concept's id, name and description.
   - An unmatched concept's id is the first 16 hex characters of the sha256 of its sorted idea keys (`item#index`).
   - Unmatched concepts are named by the local model at temperature 0, seed 1.
   - If the model's reply is unusable, the concept is named from its top c-TF-IDF terms.
4. **Links.**
   - Each concept links to its top 3 neighbours by centroid cosine + 0.15 × shared items.
   - Links with a weight of 0.12 or less are dropped.
   - Weights are then normalised into (0, 1].
5. **Regions.**
   - Louvain runs on the concept link graph.
   - A region of 3 or more concepts gets a one- or two-word name from one model call. If it matches a region in the previous
     map, it keeps that region's name instead.
   - Any other region's name is `null`.
6. **Layout.**
   - A seeded force layout (seed 7) is scaled to [0, 1] with a 0.06 margin.
   - An item sits at the mean of its concepts, offset by a golden angle, and is clamped to [0, 1].
   - A region sits at the mean of its concepts.
   - Coordinates are rounded to 4 decimal places.

The model server starts only if some concept or region needs a new name.

## Identity and storage

- `input_hash` is the sha256 of canonical JSON. The JSON holds:
  - the schema version;
  - every shaping parameter (`mapParams`);
  - the embedding fingerprint and the namer pin;
  - the sorted `[itemId, digestDerivationId, ideaIndexes]` of every included digest.
- A build is skipped when `input_hash` equals the newest map's.
- `id` = `libraryMapId(input_hash, output_json)` = `sha256(JSON.stringify(['library-map-v1', inputHash, output]))`.
- `output_json` is canonical JSON (sorted keys, finite numbers). It must pass `checkLibraryMap` and stay within 8 MiB.
- `producer_json` is a JSON object of at most 4096 bytes.
- `created_at` is an ISO time of at most 64 bytes.
- Rows live in `library_maps` (migration 016). Each device keeps the newest two.

## Sync

- Export writes only the newest map, as `{kind:'library-map',version:1,map:<row>}` at `objects/<sha>.json`.
- On import, `checkLibraryMapRow` checks the row:
  - exact columns, hex hashes and `schema_version` 1;
  - the producer, canonical output and recomputed id;
  - `checkLibraryMap` on the output.
- The row is then inserted unless already present, and the newest two are kept.
- Maps are never recorded in `sync_observations`.

## Schedule

The worker runs the map step after the digest drain, under the same guards:

- no pending cleanup;
- digests are not deferred to another device;
- no digests are due.

A build is due when the accepted-digest set changed since this device's last build, and either 5 or more entries changed
or 24 hours have passed. The worker builds at most once every 6 hours. Model and runtime errors postpone the next try:

| Error               | Postponed for |
| ------------------- | ------------- |
| `model-busy`        | 5 minutes     |
| `model-unavailable` | 30 minutes    |
| `runtime-missing`   | 1 hour        |
| anything else       | 6 hours       |

The due check reads only the store, so the worker starts no model while nothing has changed.