Model-agnostic PII detection with LLMs
A configurable, model-agnostic PII detection system built on LLMs that treats detection logic as instructions rather than fixed code, enabling flexible entity detection without retraining The detector uses instruction-driven design with a uniform inference interface (Inferencer) that supports both managed APIs (Amazon Bedrock) and self-hosted open models Evaluated span-for-span against eight other LLM-based detectors including OpenAI PrivacyFilter across five public PII corpora, demonstrating co
Analysis
TL;DR
- A configurable, model-agnostic PII detection system built on LLMs that treats detection logic as instructions rather than fixed code, enabling flexible entity detection without retraining
- The detector uses instruction-driven design with a uniform inference interface (Inferencer) that supports both managed APIs (Amazon Bedrock) and self-hosted open models
- Evaluated span-for-span against eight other LLM-based detectors including OpenAI PrivacyFilter across five public PII corpora, demonstrating competitive performance
- Customization requires only prompt edits—adding or removing entity types is a one-line change with no retraining or redeployment needed
- The system handles messy, multilingual PII across eight languages without translation, addressing limitations of traditional token-classification models with frozen schemas
Why It Matters
This approach solves a critical problem in AI safety: models fine-tuned on real-world text can memorize and leak personally identifiable information, creating compliance and privacy risks. By making PII detection configurable and model-agnostic, organizations can adapt quickly to new entity types (like employee IDs or crypto-wallet addresses) without costly retraining cycles. The ability to run on either managed cloud APIs or air-gapped infrastructure makes it practical for regulated industries with strict data residency requirements.
Technical Details
- Architecture: Four-component system—(1) prompt template defining 15 entity categories with definitions, do-not-flag lists, and few-shot examples; (2) configurable backend via Inferencer interface accepting any messages-in/text-out model; (3) parsing and offset layer that computes exact character positions from LLM-returned text spans; (4) thin call sequence orchestrating the pipeline
- Model Agnosticism: Supports Amazon Bedrock (Mistral, OSS-GPT) and self-hosted models (e.g., OSS-GPT 20B on EC2 with GPU) through a uniform interface, enabling deployment in VPCs or air-gapped environments
- Prompt Design: Schema defined in
pii_detector/templates.pyas a system prompt with JSON output format specifyingpii_entity_typeandpii_entity_valuefields; LLMs do not return character offsets directly—post-processing recovers them - Evaluation: Span-for-span comparison against 9 LLM-based detectors (including OpenAI PrivacyFilter) across 5 public PII corpora with common ground truth
- Customization: Entity set changes require only one-line prompt edits; no retraining, relabeling, or redeployment needed for new domain-specific identifiers
Industry Insight
- Organizations should adopt instruction-driven PII detection for fine-tuning pipelines to prevent data leakage from training corpora containing names, addresses, SSNs, and other sensitive information
- The model-agnostic design enables hybrid deployment strategies: use frontier models for accuracy-critical scenarios and smaller open models for cost-sensitive or air-gapped environments
- As PII formats evolve (crypto wallets, employee IDs, domain-specific identifiers), the ability to extend detection without retraining provides significant operational agility compared to traditional transformer taggers locked to fixed schemas
Disclaimer: The above content is generated by AI and is for reference only.