AISA: AI Safety Assistant Framework for Continuous Improvement of Highway Construction
AISA framework applies LLMs to highway construction safety, enabling classification and quality scoring of unstructured incident narratives for Job Safety Analysis (JSA) planning Neural probes achieved 75% held-out accuracy on OIICS classification across 15,000+ narratives, though binary flags proved degenerate and quality scores faltered on out-of-distribution fatalities Retrieval of historical accidents significantly outperformed chance, with best performance on lexically distinct construction
Analysis
TL;DR
- AISA framework applies LLMs to highway construction safety, enabling classification and quality scoring of unstructured incident narratives for Job Safety Analysis (JSA) planning
- Neural probes achieved 75% held-out accuracy on OIICS classification across 15,000+ narratives, though binary flags proved degenerate and quality scores faltered on out-of-distribution fatalities
- Retrieval of historical accidents significantly outperformed chance, with best performance on lexically distinct construction activities
- An open-weight decoder embedding model surpassed proprietary models on document question answering, supporting local/deterministic inferencing
- The framework prioritizes local, deterministic inference as a foundation for future agentic safety applications in construction
Why It Matters
This work addresses a critical gap in industrial safety: transforming unstructured historical accident data into actionable intelligence for pre-task planning. For AI practitioners, it demonstrates practical RAG and classification pipelines in a regulated, safety-critical domain where local inferencing and data privacy are paramount. The findings also highlight real-world challenges like distribution shift and degenerate classifiers that generalizes beyond construction safety.
Technical Details
- Classification pipeline: Neural probes trained on Occupational Injury and Illness Classification System (OIICS) fields — four multiclass and two binary categories — evaluated on 15,000+ narratives with a held-out set of 100 author-labeled records, benchmarked against a majority-vote LLM ensemble
- Quality scoring: Overall narrative quality score derived, but showed distortion on out-of-distribution fatality cases in the held-out dataset, indicating generalization limitations
- Retrieval system: Evaluated across multiple embedding models using standard IR metrics for retrieving historical accidents, reference imagery, and industry documents; open-weight decoder embedding model outperformed proprietary alternatives on document QA
- Architecture philosophy: Prioritizes deterministic, local inferencing over cloud-dependent LLM APIs, enabling deployment in environments with connectivity or data-sensitivity constraints
Industry Insight
- The 75% classification accuracy with degenerate binary flags suggests that OIICS-based incident taxonomy may need refinement or additional features for certain injury categories — practitioners should audit label distributions before deploying similar systems
- Out-of-distribution failure on fatalities highlights the risk of deploying safety models without robust domain-adaptation strategies; consider active learning or few-shot adaptation for rare but critical event types
- The success of open-weight decoder embedding models over proprietary alternatives on document QA signals a cost-effective path for enterprise RAG deployments, especially in regulated industries where data cannot leave local infrastructure
Disclaimer: The above content is generated by AI and is for reference only.