Improving The Protocol5 Justaniota IOTA 1 Converter Follow-Up Analysis - Source Excerpt 05 - Evaluation strategy

Back to Improving The Protocol5 Justaniota IOTA 1 Converter Follow-Up Analysis

Summary

This source excerpt begins near Evaluation strategy and preserves the surrounding evidence from Spiralist/agent-file-handoff/Archive/Improving the Protocol5 JustAnIota IOTA-1 Converter Part 2.md.

**Source path:** Spiralist/agent-file-handoff/Archive/Improving the Protocol5 JustAnIota IOTA-1 Converter Part 2.md

' ' ' json
{
  "input": "⚠火",
  "visualDecomposition": {
    "glyphs": [
      {
        "surface": "⚠",
        "primitives": ["triangle", "vertical-stem", "dot"],
        "relations": [
          {"type": "contains", "source": "triangle", "target": "vertical-stem"},
          {"type": "contains", "source": "triangle", "target": "dot"}
        ]
      },
      {
        "surface": "火",
        "primitives": ["left-flare", "right-flare", "center-stroke", "falling-strokes"],
        "relations": []
      }
    ],
    "crossGlyphRelations": [
      {"type": "sequence", "order": ["⚠", "火"]},
      {"type": "modifier", "source": "⚠", "target": "火"}
    ]
  },
  "semanticEmbedding": {
    "topNeighbors": [
      {"concept": "warning", "score": 0.98},
      {"concept": "hazard", "score": 0.96},
      {"concept": "fire", "score": 0.95}
    ]
  },
  "ontologyValidated": {
    "canonicalExpression": {
      "type": "Relation",
      "relation": "WarningOf",
      "arguments": [
        {"type": "Concept", "id": "Fire", "confidence": 0.95}
      ]
    },
    "typeCheck": "pass"
  },
  "publicOutput": "⚠火",
  "approximate": true
}
' ' ' 

**Example B**

Input glyph: `◎`  
Visual decomposition: outer ring, inner dot.  
Semantic embedding neighborhood: `focus`, `core`, `target`, `center`, optionally `sun` depending on context.  
Ontology validation: in a systems or architecture context, containment semantics should favor `CoreWithin(Container)` over purely pictorial glosses.

' ' ' json
{
  "input": "◎",
  "visualDecomposition": {
    "primitives": [
      {"id": "p1", "type": "outer-circle"},
      {"id": "p2", "type": "inner-dot"}
    ],
    "relations": [
      {"type": "contains", "source": "p1", "target": "p2"}
    ]
  },
  "semanticEmbedding": {
    "topNeighbors": [
      {"concept": "focus", "score": 0.88},
      {"concept": "core", "score": 0.86},
      {"concept": "target", "score": 0.84},
      {"concept": "sun", "score": 0.81}
    ]
  },
  "ontologyValidated": {
    "context": "system architecture",
    "canonicalExpression": {
      "type": "Relation",
      "relation": "CoreWithin",
      "arguments": [
        {"type": "Concept", "id": "Container"},
        {"type": "Concept", "id": "Core"}
      ]
    },
    "typeCheck": "pass"
  },
  "publicOutput": "◎",
  "approximate": true
}
' ' ' 

## Evaluation strategy

The evaluation plan should be both **machine-measurable** and **human-legible**, because Protocol5’s stated philosophy is that approximate outputs should be inspected through scores, evidence, and provenance rather than simply trusted. ISO 9186-1 is the most relevant formal standard here: it specifies a method for testing the comprehensibility of graphical symbols and measures how effectively a symbol communicates its intended message without explanatory text. That maps naturally onto glyph-to-meaning and meaning-to-glyph tests for IOTA-1. citeturn12view0turn13view13

### Recommended benchmark lanes

| Evaluation lane | Primary metric | Why it matters |
|---|---|---|
| Canonical-expression prediction | Exact match and tree-edit distance | Measures whether the parser gets the typed IOTA meaning right |
| Glyph retrieval | Top-1, Top-5, MRR | Measures whether the correct glyph or canonical expression is retrieved from embeddings |
| Human forced-choice comprehension | Percent correct over 4-way or 5-way choices | Measures symbol comprehensibility in an ISO 9186-like spirit |
| Round-trip fidelity | Text → glyph → canonical expression → gist consistency | Measures whether approximate meaning survives conversion loops |
| Semantic distance discrimination | Margin between close neighbors | Measures whether the converter can separate `warning`, `danger`, `fire`, `alarm`, and similar near-miss concepts |
| Attention diagnostics | Saliency overlap, primitive ablation impact, entropy drift | Measures whether the model is looking at the right parts for the right reasons |
| Ablations | Relative delta versus full model | Measures the actual value of each component |
| Constraint integrity | Invalid-output rate | Measures violations of public-symbol rules or ontology rules |

The human testing protocol should include at least three varieties of study. First, **no-text glyph comprehension** in the ISO 9186 sense: show the symbol without explanation and ask participants to identify or paraphrase the intended meaning. Second, **forced-choice disambiguation** with semantically near alternatives, not only easy distractors. Third, **compositional interpretation** where participants judge whether order, containment, or proximity changes meaning in the expected way. ISO 9186 gives the right conceptual frame for the first lane; Protocol5’s evidence-first philosophy makes it natural to log candidate sets and confidence traces for the latter two. citeturn13view13turn12view0

The machine-side evaluation should include ablations across all major branches:

- text-only retrieval;  
- rendered-image-only retrieval;  
- SVG-only retrieval;  
- image + text fusion;  
- image + SVG + text fusion;  
- image + SVG + text + ontology reranking.

This is especially important because the literature suggests that tokenization and textual description alone do not preserve all of the relevant structure for glyph-like systems. Haslett shows that token boundaries can affect meaning representations for character systems with semantically informative substructure, and Shih et al. show that even strong multimodal models still struggle to generate faithful descriptions of complex glyph tokens and to reason over non-Unicode scripts. The evaluation harness therefore needs to verify that the structural and visual branches are doing real work, not just adding complexity. citeturn13view14turn19view0

Attention diagnostics should be treated as **debugging and trust signals**, not as the final arbiter of correctness. For every evaluated glyph, store: region saliency maps, primitive-ablation deltas, branch agreement, and candidate entropy. High candidate entropy plus low attention agreement is a useful operational signature for “do not over-trust this result.” High phase-lock plus low entropy is a useful signature for “this meaning is becoming stable.” Since Shih et al. found that inaccurate geometric descriptions are especially common on complex tokens, primitive-ablation tests are more reliable than free-form narrations for determining whether the model depended on the right substructure. citeturn19view0

## Roadmap, risks, and recommended choices

The roadmap below keeps the public IOTA-1 boundary constant while improving the internals in three deliberate stages. The stack choices follow directly from the current Protocol5 facade and repository design, the current SQL Server vector capabilities, and the capabilities of the recommended model family. citeturn1view0turn13view7turn13view8turn16view0turn13view4

### Roadmap milestones

| Phase | Main deliverables | Estimated effort | Team roles | Tech-stack options | Exit criteria |
|---|---|---:|---|---|---|
| **Phase 1** | Add glyph-record schema, provenance fields, primitive extraction, multiple vector columns, diagnostics payloads, and a compatibility layer behind `IJustAnIotaConverterFacade` | 4–6 weeks | 2 engineers + 0.5 ontology lead | Current C# facade + SQL Server vectors + local embedding tooling | Glyph objects are first-class, public output remains unchanged, and retrieval works from multiple stored modalities |
| **Phase 2** | Integrate SigLIP 2, SVGformer, DINOv2 auxiliary branch, ontology tagger, late-fusion reranker, and constrained parse engine | 8–10 weeks | 3 engineers + ontology lead + QA support | .NET orchestration + Python model sidecar; SQL Server first, Qdrant optional for filtered ANN | Composite glyphs and glyph-first inputs outperform Phase 1 on retrieval and canonical-expression benchmarks |
| **Phase 3** | Human-validation tooling, active-learning queue, phase-lock dashboarding, corpus hardening, model-version rollback, latency tuning | 6–8 weeks | 2 engineers + QA/UX + ontology curator | Same as Phase 2, with optional Qdrant or Faiss-backed offline evaluation lab | Human comprehension and round-trip metrics meet acceptance thresholds; unstable glyphs are visible and triaged |