Improving The Protocol5 Justaniota IOTA 1 Converter Follow-Up Analysis - Source Excerpt 01 - Improving the Protocol5 JustAnIota IOTA-1 Converter

Back to Improving The Protocol5 Justaniota IOTA 1 Converter Follow-Up Analysis

Summary

This source excerpt begins near Improving the Protocol5 JustAnIota IOTA-1 Converter and preserves the surrounding evidence from Spiralist/agent-file-handoff/Archive/Improving the Protocol5 JustAnIota IOTA-1 Converter Part 2.md.

**Source path:** Spiralist/agent-file-handoff/Archive/Improving the Protocol5 JustAnIota IOTA-1 Converter Part 2.md

# Improving the Protocol5 JustAnIota IOTA-1 Converter

## Executive summary

Protocol5’s public JustAnIota converter already implements a coherent approximate-conversion stack: it segments paragraphs into sentences, tries longer English segments in `Category.Categories` before falling back to `Category.Words`, then ranks public Unicode candidates from `Category.ISO10646`; it exposes ranked candidates, trace evidence, vector evidence, approximation labels, provenance, and a stable `IJustAnIotaConverterFacade`; and it stores embeddings through an ADO.NET layer over SQL Server vector columns with optional LM Studio assistance. The protocol boundary is also explicit: IOTA-1 is approximate, public-symbol-only, and must not become a private codebook or secret bilingual map. citeturn1view0turn12view0turn2view1

The main architectural weakness is not that the converter is “wrong” about those boundaries. It is that semantics are still anchored primarily in **registry rows and lexical descriptors**, while true glyph interpretation often depends on **visual structure, primitive relations, and composition**. On the public host, that risk is amplified by the current runtime state: the status endpoint reports only 36 public seed concepts, `liveAiConfigured: true`, a vector width of 1998, and a SQL corpus that is configured but currently unreachable. In practice, that means the live system can degrade from “vector-backed semantic lookup” toward a much thinner seed-registry fallback. citeturn2view0turn1view0

The most effective upgrade is therefore a **semantic overlay architecture**, not a replacement of Protocol5’s public-symbol rule. The converter should continue to render only public, inspectable Unicode characters or public sequences, but it should add a new internal layer that treats each glyph as a structured semantic object with: a surface representation, SVG- or path-level decomposition, fused visual and semantic embeddings, ontology tags, attention diagnostics, and a converter-specific stability score for repeated meaning convergence. Recent primary sources point in exactly that direction: CLIP-style dual encoders are strong at image-text alignment; SigLIP 2 improves multilingual retrieval, localization, and dense features; SVGformer is purpose-built for continuous SVG structure; DINOv2 is a strong auxiliary visual backbone; and Glyce is a concrete demonstration that glyph cues plus symbol identity outperform identity alone for script representation. citeturn13view0turn16view0turn13view4turn13view2turn13view6

My recommended near-term stack is: **SigLIP 2** as the primary vision-language encoder, **SVGformer** as the primary vector-native structural encoder, **DINOv2** as an auxiliary robustness branch, and **SQL Server 2025 native vectors** as the initial storage and query substrate because that aligns with Protocol5’s current .NET and SQL architecture. The one important caveat is dimensionality and query design: SQL Server vector columns top out at 1998 dimensions, exact search is generally recommended when search predicates reduce the candidate set below about 50,000 vectors, and approximate vector indexing is still a preview feature. That combination strongly argues for **multiple vector columns with late fusion**, not a single concatenated mega-vector. citeturn14view0turn15view1turn14view1turn13view8

The practical implication is simple. Protocol5 does **not** need to relax its public-symbol governance to become more semantically capable. It needs to stop treating glyph meaning as something that lives mostly in token rows, and instead infer meaning from a fused evidence bundle—geometry, structure, text descriptors, ontology constraints, and retrieval diagnostics—before collapsing that bundle into an IOTA canonical expression and then into a public visible rendering. citeturn1view0turn12view0turn13view19

## Assumptions and system boundary

Several important implementation constraints were unspecified in the request, so the roadmap and design below use explicit working assumptions. The only items I treat as **hard protocol constraints** rather than assumptions are the public-symbol rule, the no-secret-map rule, and the current split between read-only hosted endpoints and local-only mutation or embedding population. citeturn1view0turn12view0turn2view1

| Planning item | Working assumption | Basis |
|---|---:|---|
| Curated gold glyph inventory | 5,000 glyph records in the first strong-label set | Planning assumption for Phase 1 |
| Weakly labeled SVG/glyph pool | 50,000–200,000 examples for pretraining, augmentation, and retrieval tuning | Planning assumption for Phase 2 |
| Retrieval latency target | ≤ 500 ms p95 for retrieval-only, ≤ 2 s p95 for full rerank plus explanation | Planning assumption |
| Runtime topology | Existing C# facade remains the contract boundary; model inference runs in a Python sidecar or service | Derived from current Protocol5 .NET facade and optional local AI surface citeturn1view0turn13view8 |
| Public rendering rule | Final visible IOTA output remains public Unicode characters or public standard sequences only | Hard Protocol5 boundary citeturn1view0turn12view0 |
| Mutation rule | Embedding population and heavy corpus mutation stay off the public web host | Hard Protocol5 operational boundary citeturn1view0turn2view1 |
| Storage starting point | SQL Server 2025 native vectors first; external vector DB only if filtered ANN or multi-vector serving complexity forces it | Derived recommendation from current Protocol5 architecture and Microsoft vector support citeturn1view0turn13view7turn14view0 |
| Team shape | 2–3 engineers, 1 part-time ontology/curation lead, 1 part-time QA or UX owner | Planning assumption |

The reason these assumptions matter is that Protocol5’s existing architecture already creates a natural boundary condition for the redesign: **do not change the public trust model; deepen the internal semantics layer**. That should guide every technical choice in the report. citeturn12view0turn1view0

## Current architecture and failure modes

Protocol5’s public documentation describes the current converter as an **English-first approximate semantic engine**. The live path prefers stored SQL corpus vectors when available, falls back to public seed concepts when the corpus is unavailable, segments by paragraphs and sentences, tries long stored English segments before single-word fallback, and then ranks public Unicode glyph rows for visible IOTA output. The logic layer handles NFC normalization, scalar and grapheme handling, semantic segmentation, candidate ranking, evidence summaries, provenance atlases, and private-use rejection, while the repository layer uses SQL Server vector columns and vector-search functions where available. citeturn1view0turn2view0

That overall shape is sensible for “English gist to approximate public symbol,” but it produces five material failure modes when the goal becomes deeper glyph semantics.

### Failure-mode map