Meta’s AI keeps scraping my community website, and I get charged £25 a month for the privilege Meta的AI不断抓取我的社区网站，而我每月为此支付25英镑的费用

TL;DR

• Meta's AI crawler aggressively scraped a small community wiki, generating over 230,000 requests in just 1.5 days, consuming nearly half the site's bandwidth and costing the individual operator approximately $30/month in cloud fees.
• The crawler ignored standard web conventions (robots.txt), systematically targeting low-value pages like edit histories, diffs, and login screens rather than actual content articles.
• This case exemplifies how large AI companies externalize infrastructure costs onto volunteer-run websites while extracting valuable training data without compensation or regard for site sustainability.

Why It Matters

This incident reveals a critical ethical and economic imbalance in the current AI development model where massive corporations extract data from small-scale community resources without bearing proportional costs. For AI practitioners and researchers, it highlights the need for more responsible crawling practices and sustainable data sourcing models that don't undermine the very communities that contribute to knowledge ecosystems.

Technical Details

• The Meta crawler identified as 'meta-webindexer' made 230,743 requests to the LGBT History Project wiki over 36 hours, representing 28% of total server traffic during that period
• Despite having only 4,814 actual articles, the MediaWiki platform's architecture created hundreds of thousands of unique URLs through revision histories, diff views, and edit interfaces
• The crawler specifically targeted disallowed paths per robots.txt specifications: 226,391 requests (98%) went to /index.php addresses including edit screens, revision diffs, and login pages
• Only 1.5% of requests were for legitimate article content; the single most-requested endpoint was the login screen (113,016 requests)
• The crawler completely ignored the robots.txt file at lgbthistoryuk.org/robots.txt which explicitly prohibited access to these technical endpoints and recommended 10-second request intervals

Industry Insight

AI companies should implement rate-limiting and respect for established web standards when scraping public sites, particularly those maintained by volunteers with limited resources. The industry needs to develop fair compensation mechanisms or infrastructure support for websites that provide valuable training data, recognizing that free data extraction from small operators creates an unsustainable business model. Future AI development should consider partnerships with community organizations rather than unilateral scraping that degrades the services providing their training material.

TL;DR

• Meta的爬虫在不到两天的时间内对作者维护的LGBT历史项目网站进行了230,743次请求，占服务器总流量的近一半。
• 该爬虫无视robots.txt协议，大量抓取低价值的页面如登录界面和编辑历史，导致作者需额外支付约£25/月的带宽费用并升级服务器。
• AI行业通过无偿抓取个人或小团队建设的社区网站数据来训练模型，将成本外部化到网站所有者身上。
• 此类行为不仅造成直接经济损失，还因资源挤占影响了真实用户的访问体验。
• 这反映了当前AI数据获取模式中存在的伦理与商业责任失衡问题。

为什么值得看

这篇文章揭示了大型科技公司（如Meta）在构建AI系统时，如何通过自动化爬虫无偿使用个人或小社区运营的网站内容，却由后者承担相应的计算和带宽成本。对于AI从业者而言，这是一个关于数据来源伦理、资源分配公平性以及行业可持续发展的重要案例研究；它促使人们反思：在追求技术进步的同时，是否应当建立更合理的机制来补偿那些为互联网知识生态做出贡献的个体创作者？

技术解析

• 爬虫行为分析：Meta的meta-webindexer bot在短时间内发起极高频率的请求（平均每分钟数百次），远超正常人类浏览或友好型搜索引擎爬虫的行为模式。其目标并非仅限于核心文章页面，而是深入挖掘了Wiki平台特有的冗余结构（如版本差异diffs、旧修订版、编辑表单等）。

• 协议规避策略：尽管站点根目录下存在标准的robots.txt文件明确指示禁止访问特定路径（例如/index.php相关地址）并要求请求间隔至少10秒，但该爬虫完全未读取也未遵守此规则，显示出对既有网络规范的 disregard。

• 成本量化方法：作者基于亚马逊云服务（AWS）的计费逻辑估算出仅Meta一家公司的单次抓取行动就产生了相当于每月£25的数据传输费，若加上其他类似AI厂商的重复操作及被迫进行的硬件升级支出，实际负担更为沉重。

• 架构影响评估：由于并发访问量激增且包含大量无效请求，原有服务器处理能力被严重挤压，导致 legitimate user experience下降——表现为页面加载缓慢甚至暂时不可用，从而触发进一步的投资以增强基础设施韧性。

行业启示

• 重新审视训练数据的合法性与代价：随着生成式AI快速发展，企业对海量文本语料的需求日益增长。然而，目前主流做法仍倾向于“先抓取后授权”或直接忽略版权约束，这种模式难以为继。未来可能需要探索更加透明、合规且具备经济激励的数据采集方案，比如设立专项基金支持小型知识库建设者，或者采用联邦学习等方式减少对外部资源的依赖。

• 推动制定新的数字契约框架：鉴于现有法律法规滞后于技术创新步伐，业界应联合政府机构共同商讨适用于人工智能时代的新型合作协议范本。其中应包含清晰界定各方权利义务的内容，特别是针对非营利性或个人性质网站的保护措施，确保他们在贡献自身成果的同时不会遭受不合理损失。

• 倡导负责任的AI开发文化：鼓励科技巨头主动承担起社会责任，在产品设计阶段就考虑到潜在的社会影响，包括但不限于优化算法效率以降低不必要的网络负载、提供API接口供受控访问替代粗暴爬取、以及公开披露所用数据集来源等信息。唯有如此，才能赢得公众信任并实现真正的共赢局面。