SHELF: A Synthetic Harness for Multi-Task Bibliographic Benchmarking
SHELF is a Python-based synthetic benchmarking harness designed to evaluate LLM fitness for bibliographic tasks in libraries and archives The first release contains 62,899 model-written documents based on Library of Congress vocabularies, covering classification, clustering, retrieval, pair classification, and instruction retrieval tasks Subject classification achieved strong results (0.8887), while genre-form classification performed poorly (0.2605), with several pair and clustering tasks remai
Analysis
TL;DR
- SHELF is a Python-based synthetic benchmarking harness designed to evaluate LLM fitness for bibliographic tasks in libraries and archives
- The first release contains 62,899 model-written documents based on Library of Congress vocabularies, covering classification, clustering, retrieval, pair classification, and instruction retrieval tasks
- Subject classification achieved strong results (0.8887), while genre-form classification performed poorly (0.2605), with several pair and clustering tasks remaining near chance
- Sparse methods (TF, TF-IDF, BM25) remain competitive on classification, with TF-IDF being the fastest measured approach
- Model rankings transfer more reliably across benchmarks than absolute scores, but SHELF scores do not directly estimate production catalogue accuracy
Why It Matters
This benchmark addresses a critical gap in the AI landscape: the lack of systematic evaluation tools for bibliographic and archival work, which libraries and archives perform with limited staff and computing budgets. For AI practitioners working in information retrieval and digital humanities, SHELF provides a controlled, reproducible framework to understand which methods are suitable for specific bibliographic tasks and what computational resources they require.
Technical Details
- SHELF generates controlled benchmark data by converting labelled taxonomies, writing specifications, and a generation budget into evaluation tasks, using Library of Congress vocabularies as the foundational knowledge base
- The benchmark evaluates multiple methods across different task types: TF, TF-IDF, BM25, popular encoders, and zero-shot decoders (on subject classification only), with each method appearing only on tasks that support it
- The dataset includes 62,899 model-written documents with independently varied bibliographic facets, enabling generation of verifiably unseen documents that fall after a model's training cutoff
- Comparative analysis against LCSHBench and Project Gutenberg demonstrates that while absolute scores are not portable, relative model rankings show more reliable transfer across benchmark domains
- All source code and data are released under permissive licenses on GitHub and Hugging Face for reproducibility and community extension
Industry Insight
- Libraries and archives should adopt synthetic benchmarks like SHELF to make informed decisions about LLM deployment, particularly given resource constraints that make trial-and-error evaluation impractical
- The significant performance gap between subject classification (0.8887) and genre-form classification (0.2605) suggests that bibliographic task design should prioritize well-structured taxonomic categories over nuanced form/genre distinctions for near-term AI integration
- Practitioners should treat benchmark rankings as relative guides rather than absolute predictors of production performance, and validate any selected approach against institution-specific catalogue data before deployment
Disclaimer: The above content is generated by AI and is for reference only.