Prophecy Tracking Knowledge Graph Design - Source Excerpt 05 - Visualization and Evaluation

Back to Prophecy Tracking Knowledge Graph Design

Summary

This source excerpt begins near Visualization and Evaluation and preserves the surrounding evidence from Antichrist.net/agent-file-handoff/Archive/2026-05-12-content-reports/Prophecy-Tracking Knowledge Graph Design.md.

**Source path:** Antichrist.net/agent-file-handoff/Archive/2026-05-12-content-reports/Prophecy-Tracking Knowledge Graph Design.md

Use **JSON-LD** for APIs and web clients because it is JSON-friendly while preserving linked-data semantics. Use **Turtle** or **TriG** for standards-based exchange, archival publication, and human-readable semantic data. Use **CSV** for bulk analytics and import into general data tools, with at least `nodes.csv`, `edges.csv`, `passages.csv`, and `assertions.csv`. JSON-LD and Turtle are directly standardized Linked Data serializations, while Jena, Fuseki, and many RDF stacks already support those formats natively. citeturn19view3turn34view2turn34view0

If you adopt nanopublication-style packaging, export each reviewed contradiction or interpretation as a small package containing assertion, provenance, and publication info. That makes high-value claims portable, citable, and inspectable outside the operational application. citeturn29view0

## Visualization and Evaluation

### Visualization and UX

The most useful UX is **claim-centric**, not source-centric. Analysts typically start with one prophecy statement or one interpretation and then fan outward to entities, events, conflicting readings, and sources. The most valuable views are an ego graph around a selected claim, a synchronized timeline, a map, a provenance trail, and a contradiction matrix. Time and geography are especially important in this domain, and the standards ecosystem already supports both through OWL-Time and GeoSPARQL. citeturn19view6turn19view7

Recommended visual components are:

| View | Why it matters |
|---|---|
| Claim graph | shows statement → normalized claim → interpretation → event/entity links |
| Timeline | surfaces competing date ranges and fulfillment sequences |
| Map | reveals location convergence or divergence across interpretations |
| Provenance trail | shows exact passage, source, extractor, and reviewer chain |
| Confidence heatmap | shows where automated outputs are least trustworthy |
| Contradiction matrix | quickly shows which claims conflict and on which facets |
| Faceted filters | filter by tradition, source type, language, confidence, review status, time, place |

The provenance trail is especially important because reviewable AI systems need to expose how a claim was derived, not just what the graph currently says. That is directly in line with provenance standards and human oversight principles. citeturn19view0turn19view13

### Evaluation and validation

Evaluation should be split into **extraction quality**, **contradiction quality**, **operational review quality**, and **corpus coverage**.

For extraction, use entity precision/recall/F1, entity-linking accuracy, relation F1, event extraction F1, temporal normalization exact and relaxed accuracy, and geospatial resolution accuracy. Benchmarks like DocRED, TempEval-3, and entity linking evaluation work are useful starting points—even though none are prophecy-specific. For contradiction detection, use precision/recall/F1, AUROC or AUPRC, calibration error or Brier score, and reviewer override rate. For process quality, measure review latency, adjudication rate, and proportion of high-value claims with complete provenance. citeturn21view7turn21view18turn21view19turn32view0

Use existing public datasets where they fit the subtask: FEVER for evidence-grounded claim verification; SNLI and MultiNLI for generic entailment and contradiction training; DocRED for document-level relations; TempEval-3 for time expressions and temporal relations; and cross-document event extraction datasets such as CLES for assembling event structures from multiple sources. But because no standard benchmark captures the special difficulties of prophecy corpora, you should also create an internal gold set and a synthetic benchmark with controlled perturbations such as negation flips, number changes, date shifts, referent swaps, translation variation, and metaphor-to-literal rewrites. citeturn21view10turn21view12turn21view13turn21view7turn21view18turn23view4

Human-in-the-loop evaluation should be formal, not informal. Use double annotation on a stratified sample of claims, especially metaphor-heavy passages and high-score contradictions. Track inter-annotator agreement and the rate at which reviewers override machine judgments. That matters because recent NLI research shows that models can look strong on mainstream datasets yet fail badly on simple inference cases, and document-level contradiction studies show the long-context version of the problem remains hard. citeturn37view0turn22view0

## Roadmap Ethics and Limitations

### Prioritized implementation roadmap

A practical roadmap with no assumed stack is below.

| Milestone | Scope | Estimated effort |
|---|---|---:|
| Governance and schema foundation | finalize ontology, IDs, provenance rules, review statuses, rights model, sample corpora | 2–4 person-weeks |
| Ingestion MVP | source registry, IIIF/archive connectors, OCR/text extraction, passage anchoring, raw storage | 4–6 person-weeks |
| Core NLP normalization | entity extraction/linking, coreference, claim normalization, time/place normalization | 6–10 person-weeks |
| Contradiction engine | candidate retrieval, rule checks, NLI scorer, calibration, contradiction object model | 8–12 person-weeks |
| Review application | queues, provenance viewer, adjudication, audit log, dispute handling | 4–6 person-weeks |
| Standards export and queries | RDF/JSON-LD/Turtle export, Cypher/SPARQL query library, SHACL rules | 3–5 person-weeks |
| Visualization and evaluation harness | timelines, maps, heatmaps, benchmark runner, human evaluation workflow | 3–5 person-weeks |

That yields about **30 to 46 person-weeks** for a strong first production version. In calendar time, that is roughly **4 to 7 months** for a 2–3 person team, depending on corpus complexity and how much annotation work must be done manually. The largest schedule risk is not storage engineering; it is domain annotation and review policy design.

### Ethical and legal considerations

This project touches sensitive religious and cultural material, so the system should be designed to support respectful comparison rather than adversarial ranking. UNESCO’s AI ethics framework emphasizes human dignity, transparency, fairness, and human oversight, and those principles fit this use case well. In practice, that means labeling what the system is doing as **source tracking**, **interpretation comparison**, and **schema-level contradiction analysis**, not as final adjudication on sacred truth claims. citeturn19view13

On the legal side, preserve rights metadata at the source level. RightsStatements.org is useful for standardized cultural-heritage rights descriptions, and copyright constraints for text and data mining vary by jurisdiction. The U.S. Copyright Office’s 2025 AI training report emphasizes that legal analysis is context-dependent and that international TDM rules differ, including EU exceptions conditioned on lawful access and, in some contexts, opt-outs. For this reason, the system should store rights, access basis, source license, and extraction purpose alongside each source record. citeturn21view0turn19view14turn20view0

A conservative operational policy is: store rich metadata and structural annotations freely; store full text only when rights and access allow; keep display excerpts short when rights are uncertain; preserve source locators so users can inspect authorized originals; and never let machine paraphrases silently replace the original source wording in review interfaces. That is a design recommendation informed by cultural-heritage rights practice and current copyright uncertainty around AI-related text use. citeturn21view0turn19view14

### Open questions and limitations

Three limitations should be treated as permanent design constraints. First, modern contradiction models remain imperfect, including on simple inference problems, so automation should support review rather than replace it. Second, historical place normalization is inherently hard, and modern gazetteers are not enough for ancient or sacred geographies; resources like Pleiades help but do not solve all ambiguity. Third, metaphor, hedge, and translation effects can create apparent contradiction where the underlying issue is semantic indeterminacy rather than factual conflict. citeturn37view0turn33search0turn22view1turn22view2

If those limits are accepted up front, the design becomes much clearer: keep the original text immutable, make interpretations explicit, attach provenance to every important assertion, and reserve final contradiction decisions for a calibrated human-in-the-loop workflow. citeturn19view0turn29view0turn32view0