Improving The Protocol5 Justaniota IOTA 1 Converter Follow-Up Analysis - Source Excerpt 03 - Candidate model comparison

Back to Improving The Protocol5 Justaniota IOTA 1 Converter Follow-Up Analysis

Summary

This source excerpt begins near Candidate model comparison and preserves the surrounding evidence from Spiralist/agent-file-handoff/Archive/Improving the Protocol5 JustAnIota IOTA-1 Converter Part 2.md.

**Source path:** Spiralist/agent-file-handoff/Archive/Improving the Protocol5 JustAnIota IOTA-1 Converter Part 2.md

' ' ' mermaid
flowchart LR
    A[Input text or glyph] --> B[Unicode and SVG canonicalization]
    B --> C[Sentence and grapheme segmentation]
    B --> D[SVG path parsing and primitive extraction]
    C --> E[Text embedding tower]
    D --> F[Visual embedding tower]
    D --> G[SVG structural encoder]
    E --> H[Fusion and query vector set]
    F --> H
    G --> H
    H --> I[Filtered retrieval across visual structural and semantic indexes]
    I --> J[Ontology constraint filter]
    J --> K[Composition parser and reranker]
    K --> L[IOTA canonical expression]
    L --> M[Public Unicode rendering]
    K --> N[Trace evidence and diagnostics]
' ' ' 

The practical pipeline should work like this in implementation terms. First, normalize text with the current Protocol5 NFC and grapheme machinery; for glyph input, normalize Unicode sequences and, where available, canonicalize SVG path transforms and equivalent shapes. The W3C SVG specifications matter here because SVG basic shapes can be reduced to path-equivalent forms, which makes primitive extraction and structural indexing stable across sources. citeturn1view0turn13view19

Second, generate **three parallel embedding families**:

- a **vision-language family** from rendered glyph images using SigLIP 2;  
- a **vector-native structural family** from SVG path commands and primitive relations using SVGformer;  
- a **semantic text family** from public descriptors, CLDR/UCD/Unihan metadata, canonical-expression glosses, and curated ontology labels.

Protocol5 already points to UCD, CLDR, Unihan, and emoji data as public metadata sources for its symbol atlas; those are appropriate first-party sources for descriptors and anchors, and Unicode’s official resources document exactly those repositories. citeturn1view0turn13view15turn13view16turn13view17turn10search3

Third, retrieve in multiple spaces and then fuse. Because SQL Server vector columns max out at 1998 dimensions—and the current Protocol5 status endpoint is already configured at that ceiling—the design should **not** concatenate all modalities into one vector. Store separate columns such as `visual_global`, `svg_structural`, `semantic_text`, and `ontology_projection`, then fuse at query time with either reciprocal-rank fusion or a lightweight learned reranker. This is a major architectural point: multi-column late fusion is not just better semantically; it is also directly aligned with the current SQL dimensionality limit. citeturn2view0turn14view0

A workable late-fusion score is:

' ' ' text
score(c | q) =
  α * cos(q_visual, c_visual)
+ β * cos(q_svg, c_svg)
+ γ * cos(q_text, c_text)
+ δ * ontology_match(q, c)
+ ε * relation_match(q, c)
+ ζ * RRF_rank_bonus(c)
- λ * public_rule_penalty(c)
- μ * composition_violation_penalty(c)
' ' ' 

where `ontology_match` and `relation_match` are typed compatibility scores, and `RRF_rank_bonus` is a reciprocal-rank-fusion term computed across the different retrieval lanes.

### Candidate model comparison

The model choice should be based on original papers or official repositories, not on secondary summaries. The table below is a practical synthesis of those primary sources. citeturn13view0turn14view5turn18search0turn16view0turn13view3turn13view2turn13view6turn13view4turn13view5

| Model or tool | What it gives the converter | Main advantages for IOTA-1 | Main limitations for IOTA-1 | Recommendation |
|---|---|---|---|---|
| **CLIP** | Classic image-text dual encoder | Strong zero-shot baseline; easy image/text retrieval alignment | Older baseline; weaker on dense/localization-heavy glyph tasks than newer alternatives | Use as a baseline and benchmark anchor, not as the main production encoder |
| **OpenCLIP** | Open reimplementation of CLIP | Operational flexibility and broader open checkpoint ecosystem | Checkpoint quality and data provenance vary more than single-line official releases | Use when open deployment flexibility is more important than standardization |
| **SigLIP** | CLIP-style model with pairwise sigmoid loss | Better scaling behavior than softmax CLIP in the original paper and strong contrastive performance | Still behind SigLIP 2 on multilingual, localization, and dense-feature needs | Acceptable if you must stay close to original SigLIP components |
| **SigLIP 2** | Multilingual vision-language encoder with stronger localization and dense features | Best fit for glyph regions, multilingual glosses, retrieval, and aspect-sensitive inputs; official open release | More complex than plain CLIP operationally | **Recommended primary vision-language tower** |
| **ViT** | Pure image transformer backbone | Strong general image representation and simple conceptual building block | Image-only; no native text alignment; not specialized for SVG | Useful backbone reference, but incomplete alone for IOTA-1 |
| **DINOv2** | Self-supervised visual features | Robust image features across domains without fine-tuning; good auxiliary tower | No text tower; must be fused with a semantic branch | **Recommended auxiliary visual branch** |
| **Glyce** | Glyph-aware representation combining glyph and ID embeddings | Clear proof that glyph information plus identity is valuable; relevant conceptual precedent | Chinese-character-specific design; not a multilingual public-symbol encoder | Use as architectural inspiration, not as the production core |
| **SVGformer** | Continuous SVG representation learning | Purpose-built for continuous SVG, retrieval, interpolation, and long-range geometric structure | Needs adequate SVG data and tuning; not a drop-in text model | **Recommended primary SVG structural encoder** |
| **DeepSVG** | Hierarchical generative SVG model | Excellent for augmentation, synthetic glyphs, interpolation, and data generation | Better as generation and augmentation than as the primary retrieval backbone | Use for data augmentation and stress testing, not primary semantic retrieval |

### Candidate vector-store comparison

Storage choice matters because the retrieval design here is not just “ANN over one vector”; it is **multiple vectors plus ontology-tag filters plus provenance plus diagnostics**. The table below compares the best options against that requirement set. citeturn13view7turn14view0turn15view1turn13view8turn13view9turn14view2turn14view4turn13view11turn14view3turn13view12

| Storage option | Strengths | Weaknesses | Recommendation |
|---|---|---|---|
| **SQL Server vectors** | Native fit with current Protocol5 stack; exact kNN and DiskANN path; `SqlVector<T>` support in .NET; exact search is recommended under ~50k filtered candidates | 1998-dimension cap per vector column; approximate vector search remains preview; vector-column limitations such as no B-tree index directly and no Always Encrypted support | **Recommended starting point** because it aligns with Protocol5 today |
| **Qdrant** | Production vector DB with payload filtering; filterable HNSW is explicitly optimized for vector search with structured filters | Additional service and operational surface compared with staying inside SQL Server | **Best scale-out option** if filtered ANN and multi-vector search become dominant |
| **pgvector** | Familiar SQL workflow; exact search by default; HNSW and IVFFlat available | More operational change from Protocol5’s current stack; IVFFlat has weaker speed/recall tradeoff than HNSW | Strong alternative if the platform standardizes on PostgreSQL |
| **Milvus** | Built for large-scale distributed ANN with metadata filtering | Higher infrastructure complexity than this use case likely needs early on | Good only if the corpus becomes very large very quickly |
| **Faiss** | Excellent offline benchmark and experiment library; strong GPU support | Library, not a serving database with payload and governance features | Use for offline evaluation and tuning, not as the serving plane |

### Example glyph-record schema

The schema below is intentionally aligned with Protocol5’s public-symbol boundary while adding the missing internal semantics. It is a design proposal, not an existing Protocol5 schema.