Ask HN: How do you protect self-hosted websites from LLM scrapers? 问 HN：你如何保护自托管网站免受 LLM 爬虫的侵害？

TL;DR

• The post discusses self-hosted static websites with low traffic and seeks techniques to protect against abusive LLM scraping
• Four existing techniques are mentioned: Anubis AI firewall, LLM poisoning (iocaine), honeypot links, and blocking cloud provider CIDR ranges
• No new technical contributions, benchmarks, or findings are presented

Why It Matters

This is a community discussion thread rather than a substantive AI research or industry article. It reflects growing practitioner concern about LLM scraping but offers no novel analysis, data, or actionable technical depth.

Technical Details

• The post lists four defensive techniques but provides no implementation details, performance metrics, or evaluation results
• No architecture, model specifications, datasets, or benchmarks are discussed
• The content is purely exploratory, asking others for additional techniques rather than presenting findings

Industry Insight

• LLM scraping mitigation is an emerging concern, but the community is still in early exploration without standardized or proven solutions
• The lack of substantive technical content in this post indicates the field lacks mature, well-documented defensive strategies worth citing

Insufficient content

TL;DR

• 讨论自托管静态网站防护LLM爬虫的技术方案，聚焦无CDN场景
• 提及anubis AI防火墙、iocaine投毒项目、隐藏链接陷阱、云厂商IP段封锁等现有手段
• 作者寻求社区分享更多防护LLM及其他爬虫的创新技术

为什么值得看

本文反映了LLM数据抓取对网站运营者带来的实际威胁，以及社区正在探索的多元化防护策略。对于自托管网站运营者和AI安全从业者，提供了可参考的防护思路和技术方向。

技术解析

• Anubis AI Firewall：专门针对AI爬虫的防护方案，可识别并拦截自动化抓取请求
• LLM投毒项目（如iocaine）：通过在网页中注入误导性内容，污染LLM训练数据，使抓取行为对爬虫方产生负面回报
• 隐藏链接陷阱（Bate Links）：在HTML源码中嵌入用户不可见但爬虫可访问的链接，一旦检测到访问即封锁对应IP
• CIDR封锁策略：直接屏蔽AWS、GCP、Azure、Alibaba等主要云提供商的IP段，阻断来自云环境的批量抓取

行业启示

• LLM数据抓取防护正从传统的反爬虫手段向专门针对AI训练场景的防御策略演进，"投毒"等对抗性方法体现了攻防双方的博弈升级
• 自托管和小规模网站运营者面临防护资源有限的挑战，需要轻量级、低维护成本的解决方案
• 云厂商IP段封锁是一把双刃剑，可能误伤正常用户，未来需要更精细化的识别机制而非粗暴的IP封锁