LLM Wikis And Hierarchical Structure - Source Excerpt 04 - Migration strategies
Back to LLM Wikis And Hierarchical Structure
Summary
This source excerpt begins near Migration strategies and preserves the surrounding evidence from Wiki.FFTAC.org/agent-file-handoff/Archive/2026-05-11-improvement-concept-layer/LLM Wikis and Hierarchical Structure.md.
**Source path:** Wiki.FFTAC.org/agent-file-handoff/Archive/2026-05-11-improvement-concept-layer/LLM Wikis and Hierarchical Structure.md
| Storage pattern | Best fit | Strengths | Trade-offs | Anchors |
|---|---|---|---|---|
| **PostgreSQL plus JSONB plus pgvector** | Default stack-agnostic starting point | ACID transactions, joins, JSONB metadata, strong filtering, vectors next to relational data | Graph traversal is less natural than in a native graph DB; large-scale search UX may need a separate search tier | PostgreSQL JSON/JSONB docs; pgvector README. citeturn25view11turn29view0 |
| **Search engine plus vector fields** | Heavy lexical search, relevance tuning, and large searchable corpora | Strong BM25-style ranking, analyzers, filters, and vector fields in the same search system | Harder to use as the primary authored system of record | Elasticsearch dense_vector docs. citeturn29view1 |
| **Graph DB plus vector index** | Entity-heavy, multi-hop, provenance-heavy workloads | Native graph traversals plus embeddings on graph nodes | More operational complexity; overkill if the corpus is mostly page/section retrieval | Neo4j vector index docs. citeturn25view9 |
| **Vector-native hybrid DB** | Fast prototyping and multimodal RAG | Built-in hybrid search, configurable fusion, multimodal support | Often better as a retrieval tier than as the canonical authored wiki store | Weaviate hybrid search docs and product page. citeturn25view10turn35view0 |
For indexing, the safest baseline is **three parallel indexes**. The first is a lexical index over titles, aliases, headings, and chunk text for exact matching and sparse relevance. The second is a vector index over chunks, summaries, and optionally entities for semantic recall. The third is a symbolic index over parent paths, section paths, concept IDs, entity IDs, time ranges, and ACL attributes. Weaviate’s hybrid model makes the vector-keyword combination explicit, and LangChain’s retriever abstractions plus permission-aware wrappers show how metadata and security filters can be layered without rewriting the vector layer itself. citeturn25view10turn27view0turn27view5
For chunking, structural boundaries should come first. LangChain’s markdown-header splitter explicitly argues that chunking should “honor the structure of the document itself,” and Perplexity’s contextualized embeddings similarly assume that chunks from a document should preserve document order and shared context. In practice, that means chunking by page → section → passage, not by arbitrary token windows alone. Tables, infoboxes, and media should be lifted into their own child nodes with modality-appropriate embeddings and back-pointers to the parent page and section. citeturn25view12turn23view11turn29view2turn24view9
A practical metadata schema should carry at least these fields: immutable node ID; node type; page title; aliases; parent path; heading path; source URI; time stamps; authorship; status; tags or concepts; entity IDs; ACL attributes; and provenance pointers down to chunk or span level. Wikidata’s statement-qualifier-reference model is a strong guide for factual metadata, while Notion, Obsidian, and LangChain/LlamaIndex show that structured properties should be first-class rather than squeezed into ad hoc text markup. citeturn24view3turn24view4turn31view2turn24view13turn23view0turn25view13turn26view0
The API surface should mirror those separations. A clean pattern is: one set of write APIs for pages and assets; one set of enrichment APIs for links, entities, and claims; and one set of query APIs for hybrid search, neighborhood traversal, and citation-aware answer assembly. LangChain’s `Document` and vector-store interfaces, plus Notion’s page/data-source split, are good examples of why ingestion and storage contracts should be explicit. citeturn25view13turn27view0turn24view12
A compact example schema for a primary node object is below. It is intentionally broad enough to support page nodes, section nodes, chunk nodes, and structured sidecars.
' ' ' json
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"title": "KnowledgeNode",
"type": "object",
"required": [
"id",
"type",
"title",
"path",
"version",
"acl",
"provenance",
"updated_at"
],
"properties": {
"id": {
"type": "string",
"description": "Immutable node identifier"
},
"type": {
"type": "string",
"enum": [
"space",
"page",
"section",
"chunk",
"entity",
"claim",
"summary",
"asset",
"concept"
]
},
"title": {
"type": "string"
},
"aliases": {
"type": "array",
"items": { "type": "string" }
},
"path": {
"type": "array",
"items": { "type": "string" },
"description": "Ancestor IDs from root to self"
},
"heading_path": {
"type": "array",
"items": { "type": "string" }
},
"body": {
"type": "string"
},
"metadata": {
"type": "object",
"properties": {
"language": { "type": "string" },
"status": {
"type": "string",
"enum": ["draft", "published", "archived", "deprecated"]
},
"tags": {
"type": "array",
"items": { "type": "string" }
},
"concept_ids": {
"type": "array",
"items": { "type": "string" }
},
"entity_ids": {
"type": "array",
"items": { "type": "string" }
},
"modality": {
"type": "string",
"enum": ["text", "image", "table", "audio", "video", "mixed"]
}
},
"additionalProperties": true
},
"version": {
"type": "object",
"required": ["revision_id", "parent_revision_id"],
"properties": {
"revision_id": { "type": "string" },
"parent_revision_id": { "type": ["string", "null"] },
"valid_from": { "type": ["string", "null"], "format": "date-time" },
"valid_to": { "type": ["string", "null"], "format": "date-time" }
}
},
"acl": {
"type": "object",
"required": ["visibility", "principals"],
"properties": {
"visibility": {
"type": "string",
"enum": ["public", "internal", "restricted", "confidential"]
},
"principals": {
"type": "array",
"items": { "type": "string" }
},
"inherit": { "type": "boolean" }
}
},
"embeddings": {
"type": "array",
"items": {
"type": "object",
"required": ["space", "model", "vector_ref"],
"properties": {
"space": { "type": "string" },
"model": { "type": "string" },
"vector_ref": { "type": "string" }
}
}
},
"provenance": {
"type": "object",
"required": ["source_uri"],
"properties": {
"source_uri": { "type": "string" },
"source_span": { "type": ["string", "null"] },
"extracted_by": { "type": ["string", "null"] },
"confidence": { "type": ["number", "null"] }
}
},
"updated_at": {
"type": "string",
"format": "date-time"
}
}
}
' ' '
That schema operationalizes the main design lesson of this report: keep **structure**, **semantics**, **security**, and **provenance** explicit rather than implicit. It is deliberately compatible with Wikidata-style claim provenance, Notion-style properties, Obsidian-style local metadata, and chunk-aware retrieval frameworks such as LangChain and LlamaIndex. citeturn31view2turn24view13turn23view0turn25view12turn26view6
A corresponding ER model for a hybrid document-plus-claim implementation could look like this:
' ' ' mermaid
erDiagram
SPACE ||--o{ PAGE : contains
PAGE ||--o{ PAGE_VERSION : has
PAGE ||--o{ SECTION : contains
SECTION ||--o{ CHUNK : contains
PAGE ||--o{ ASSET : embeds
PAGE }o--|| ACL_POLICY : governed_by
CHUNK ||--o{ CHUNK_VECTOR : indexed_as
CHUNK }o--o{ ENTITY : mentions
ENTITY ||--o{ CLAIM : has
CLAIM }o--|| SOURCE : backed_by
PAGE ||--o{ RELATION : source_of
' ' '
## Migration strategies
Migration should preserve two things above all: **stable identity** and **recoverable hierarchy**. Everything else can be rebuilt. That principle is consistent with Wikidata’s use of immutable item IDs plus mutable labels and aliases, and with the way SKOS positions itself as a low-cost migration path for existing knowledge organization systems. citeturn31view4turn31view3turn25view5