Using Fine-Tuned LLMs to Identify Indicators of Vulnerability in UK Police Incident Logs
Fine-tuned open-weight LLMs can estimate the prevalence of vulnerability indicators (mental ill health, substance misuse, etc.) in UK police incident logs at a population level. Naive single-pass LLM classification is unreliable and prone to instability and systematic over-assignment of indicators compared to human judgment. Defensible measurements require a resource-intensive multi-stage pipeline including repeated inference, label aggregation, structured human review, and statistical correctio
Analysis
TL;DR
- Fine-tuned open-weight LLMs can estimate the prevalence of vulnerability indicators (mental ill health, substance misuse, etc.) in UK police incident logs at a population level.
- Naive single-pass LLM classification is unreliable and prone to instability and systematic over-assignment of indicators compared to human judgment.
- Defensible measurements require a resource-intensive multi-stage pipeline including repeated inference, label aggregation, structured human review, and statistical correction.
- LLM outputs are unsuitable for individual-level operational decisions due to frequent and unpredictable errors, highlighting significant constraints in applied public sector settings.
Why It Matters
This research provides critical evidence for AI practitioners working in sensitive, high-stakes domains like law enforcement, demonstrating that while LLMs offer scalability for analyzing unstructured text, they cannot replace rigorous methodological safeguards. It underscores the necessity of hybrid human-AI workflows and statistical adjustments to ensure data integrity, warning against the direct deployment of LLMs for decision-making without extensive validation.
Technical Details
- Pipeline Architecture: Utilizes a multi-stage process combining repeated model inference, label aggregation, structured human review, and statistical correction to mitigate bias.
- Model Deployment: Employs a locally hosted open-weight LLM to comply with the strict data security and privacy requirements of UK police forces.
- Data Scope: Analyzes nearly 3,000 de-identified incident logs from a UK police force, adapting a classification pipeline originally developed on open-source US police data.
- Target Indicators: Focuses on identifying four specific vulnerability markers: mental ill health, substance misuse, alcohol dependence, and homelessness.
- Validation Method: Compares aggregated LLM outputs against human judgment to quantify systematic over-assignment and determine the threshold for defensible population-level estimates.
Industry Insight
- Hybrid Workflows are Essential: Organizations should design AI systems that integrate human oversight and statistical correction mechanisms rather than relying on automated outputs for critical decisions.
- Domain Adaptation Challenges: Models trained on data from one jurisdiction (e.g., US) may not transfer directly to another (e.g., UK) without significant adaptation and validation, especially regarding linguistic and procedural nuances.
- Resource Planning for AI Implementation: Deploying LLMs for administrative analysis requires substantial investment in human review and computational resources to achieve defensible accuracy, impacting cost-benefit analyses for such initiatives.
Disclaimer: The above content is generated by AI and is for reference only.