Names Can Hurt: Spotting Slopsquatting Risks Caused by Package Name Hallucinations in Local Coding LLMs
The paper introduces "slopsquatting," a supply chain attack where adversaries pre-register PyPI packages with names matching hallucinated package names from code-generating LLMs A two-layer detector combines deterministic PyPI existence checks with a Random Forest classifier trained on ten features from package names and metadata An import name reconciler resolves mismatches between import names and pip install names (e.g., `import cv2` vs `pip install opencv-python`) The pipeline, embedded in a
Analysis
TL;DR
- The paper introduces "slopsquatting," a supply chain attack where adversaries pre-register PyPI packages with names matching hallucinated package names from code-generating LLMs
- A two-layer detector combines deterministic PyPI existence checks with a Random Forest classifier trained on ten features from package names and metadata
- An import name reconciler resolves mismatches between import names and pip install names (e.g.,
import cv2vspip install opencv-python) - The pipeline, embedded in a LangGraph state machine with escalating retry temperatures and cross-model fallback, achieves hallucination-free code on 76% of runs across 300 curated prompts
- Hallucination rates scale linearly with adversarial prompt intensity (0–10% routine to 40–73% on slopsquat baits), and cross-family model pairing significantly improves fallback recovery
Why It Matters
This research addresses a critical and emerging supply chain security threat as local coding LLMs become more widely deployed in development workflows. Slopsquatting exploits a gap where LLM hallucinations can be weaponized by malicious PyPI actors, making this directly relevant to anyone building or relying on AI-assisted code generation in production environments.
Technical Details
- Two-layer detection architecture: Layer 1 performs a deterministic PyPI existence check; Layer 2 uses a Random Forest classifier trained on 10 features derived from package names and PyPI metadata to catch lookalike packages (e.g.,
pil,faiss,tabula,haystack) - Import name reconciler: Bridges the semantic gap between Python import names and PyPI distribution names, preventing security bypasses in cases like
import cv2/pip install opencv-python - LangGraph state machine: The detector is embedded in a retry loop with escalating temperatures; upon repeated failure, it routes to a stronger fallback model
- Evaluation: Tested across 300 curated prompts; 76% hallucination-free rate, with 28.7% of primary model failures exhausting retry budget, ~25% recovered via intra-model retries, and 16.5% via cross-model fallback
- Key empirical findings: ~50% of flagged hallucinations are real PyPI-registered lookalike packages; same-family model fallbacks repeat ~84% of primary failures, supporting cross-family pairing strategies
Industry Insight
- Organizations deploying local coding LLMs should integrate package name validation pipelines into their development tooling before hallucination-driven supply chain attacks become commonplace
- Cross-family model fallback strategies are essential—relying on the same model family for primary and fallback yields minimal recovery gains due to correlated failure modes
- The near-linear relationship between adversarial prompt intensity and hallucination rate suggests that red-teaming coding LLMs with slopsquat-style prompts should become a standard security evaluation benchmark
Disclaimer: The above content is generated by AI and is for reference only.