SCOPE and SCION: A Benchmark and an Auditable Reference Pipeline for Schema Induction and Fusion from Text
Introduces SCOPE, a new benchmark for schema induction and fusion from raw text, built from 24 public information extraction sources. Presents SCION, an auditable reference pipeline for schema construction that operates under strict JSON contracts and focuses on candidate space construction. SCION-lite achieves the highest F1 scores on the SCOPE core suite compared to various baselines, including Text2Onto-style and LLM-only approaches. The SCION-RL variant reduces reliance on proprietary LLM sc
Analysis
TL;DR
- Introduces SCOPE, a new benchmark for schema induction and fusion from raw text, built from 24 public information extraction sources.
- Presents SCION, an auditable reference pipeline for schema construction that operates under strict JSON contracts and focuses on candidate space construction.
- SCION-lite achieves the highest F1 scores on the SCOPE core suite compared to various baselines, including Text2Onto-style and LLM-only approaches.
- The SCION-RL variant reduces reliance on proprietary LLM schema engineers, offering a more compact and open-model solution.
- The work emphasizes normalized typed-edge targets rather than claiming induced schemas surpass human ontology design.
Why It Matters
This research addresses a critical bottleneck in schema-grounded information extraction and knowledge graph construction by providing a benchmark and auditable pipeline for schema induction from raw text. It offers a practical solution for practitioners and researchers to evaluate and improve schema induction processes, reducing dependency on proprietary tools and enhancing transparency in schema construction.
Technical Details
- SCOPE is a train-text-only benchmark for corpus-to-schema induction and optional schema fusion, built from 24 public information extraction sources (15 RE and 9 EE) normalized into evaluation-only gold schema graphs.
- SCION constructs candidate spaces from train text and restricts naming, merging, filtering, validation, and conservative fusion to candidate-linked evidence under strict JSON contracts.
- SCION-lite attains the highest F1 scores on the SCOPE core suite across Literal, Fuzzy, Continuous, and Graph schema-graph metrics.
- The SCION-RL variant is a compact open-model approach that reduces reliance on proprietary LLM schema engineers.
- The release includes evidence-linked outputs, parse/fallback logs, candidate retention/merging logs, run manifests, code, and benchmark packages.
Industry Insight
- The introduction of SCOPE and SCION provides a standardized framework for evaluating schema induction and fusion, which can drive further advancements in information extraction and knowledge graph construction.
- The auditable nature of SCION and its reliance on strict JSON contracts can enhance transparency and reproducibility in schema induction processes, making it easier for practitioners to adopt and adapt these methods.
- The SCION-RL variant's reduced reliance on proprietary LLM schema engineers can lower costs and increase accessibility for smaller organizations and researchers, fostering broader adoption of schema induction techniques.
Disclaimer: The above content is generated by AI and is for reference only.