Creepy crawlies
Abusive web crawlers generate so much traffic that git.kernel.org spends more CPU cycles rendering commits as HTML for scrapers than on all legitimate access combined At any given time, 14 CPU cores across 5 geo-distributed nodes are exclusively dedicated to rendering git commits as HTML for these crawlers The problem is described as "background radiation" — a persistent, low-grade but resource-intensive form of crawler abuse The issue extends beyond Git infrastructure, with similar concerns rai
Analysis
TL;DR
- Abusive web crawlers generate so much traffic that git.kernel.org spends more CPU cycles rendering commits as HTML for scrapers than on all legitimate access combined
- At any given time, 14 CPU cores across 5 geo-distributed nodes are exclusively dedicated to rendering git commits as HTML for these crawlers
- The problem is described as "background radiation" — a persistent, low-grade but resource-intensive form of crawler abuse
- The issue extends beyond Git infrastructure, with similar concerns raised for Datasette, a tool serving a large number of crawlable web pages
Why It Matters
The escalating cost of serving abusive crawlers represents a growing operational burden for open-source infrastructure and public-facing data platforms. As AI companies increasingly scrape repositories and datasets for training data, the resource imbalance between legitimate users and automated scrapers threatens the sustainability of critical open-source projects.
Technical Details
- git.kernel.org operates 5 geo-distributed nodes, with 14 CPU cores simultaneously dedicated solely to HTML rendering of git commits for crawler traffic
- The infrastructure serves both legitimate users (git clones, etc.) and automated scrapers, with scraper traffic consuming disproportionate compute resources
- The problem is not limited to Git hosting; Datasette, a tool for serving crawlable web pages from datasets, faces similar scalability and resource concerns
- The "background radiation" metaphor suggests this is a persistent, ambient problem rather than sporadic attacks, making it difficult to mitigate with traditional rate-limiting or blocking approaches
Industry Insight
- Open-source infrastructure maintainers should consider implementing stricter crawler policies, API-based access, or paid tiers to protect resources for legitimate users
- AI companies relying on large-scale web scraping should anticipate increasing costs and friction as infrastructure providers push back against abusive crawling patterns
- The trend signals a broader shift where the cost of free data access is becoming unsustainable, potentially accelerating the move toward licensed or structured data APIs for AI training pipelines
Disclaimer: The above content is generated by AI and is for reference only.