Cloudflare CEO says the web's future is "pay to crawl" as bots overtake human traffic Cloudflare首席执行官称，随着机器人流量超越人类流量，网络的未来将是“付费爬取”。

The internet has officially crossed a silent, seismic threshold: non-human traffic now constitutes the majority of activity online. Cloudflare CEO Matthew Prince confirms this tipping point has arrived years ahead of his own 2027 forecast, and he pins the explosion squarely on the rise of AI agents—automated software performing tasks, gathering data, and interacting with services at a scale and speed no human could match. His prognosis for the web’s economic model is blunt: “Clearly it’s going to be pay to crawl.”

This isn’t just a metric shifting on a dashboard. It’s the moment the foundational assumption of the web—that it’s a human-centric space we merely inhabit—collapses. For decades, the web’s economy was built on a simple, implicit bargain: content is freely available, and in return, we pay with our attention and data. That model was already fraying, but the bot flood doesn’t just stress it; it drowns it. When the majority of “visitors” to a site aren’t potential customers, readers, or users but automated scrapers, training bots, and monitoring scripts, the very notion of “traffic” as a value proxy becomes meaningless.

Prince’s “pay to crawl” conclusion feels both inevitable and terrifyingly simplistic. Of course gatekeeping will become a primary business model. Why shouldn’t a news organization, a wiki, or a creative portfolio charge for the computational resources required to ingest their work? The expensive, energy-intensive act of generating a large language model response by processing a million copyrighted articles is a service, not a right. This will create a new, direct tollbooth layer on the information superhighway. Expect a proliferation of sophisticated, dynamic billing APIs where every access request comes with a cost, calculated by content type, update frequency, and the requester’s identity.

But here’s the sharp judgment: framing this as a simple “pay to crawl” solution mistakes the symptom for the disease. The core issue isn’t that bots are accessing the web; it’s that the value chain of the web has been fundamentally rewritten by AI. The old model monetized human eyeballs. The new model monetizes data inputs for intelligence systems. The battle isn’t just about charging for access; it’s about who controls the pipeline from raw information to synthetic intelligence. Cloudflare, as a gatekeeper infrastructure provider, is perfectly positioned to become the central toll collector for this new era, and this statement is as much a market positioning as it is an observation.

This shift will bifurcate the internet into two starkly different realms. One is the Pay-to-Access Web, a curated, metered, and likely more expensive ecosystem where high-quality, trusted data flows. Think of it as the financialized, API-driven backbone of the AI economy. The other is the Open, Diluted Web, a chaotic wasteland of low-quality, auto-generated content, spam, and hallucinated information—a kind of digital brownfield where bots talk mostly to other bots, a self-referential echo chamber of synthetic sludge. Human users will increasingly feel alienated from both: one will feel like using a utility meter, the other like wandering through a hall of distorted mirrors.

The true danger isn’t the toll; it’s the potential for垄断. If “pay to crawl” becomes the norm, who sets the prices? The platforms that control the access points. This could consolidate power further in the hands of a few cloud and security giants, creating a new chokepoint. It could also spark a fierce backlash and acceleration toward decentralized alternatives. Protocols like Solid or IPFS, which aim to separate data from applications, might finally gain traction not out of ideological purity but as a practical necessity to avoid the tollbooths.

Furthermore, this milestone obliterates any remaining fiction about the “neutral” web. The infrastructure is now actively negotiating terms with software agents. Your website’s robots.txt file was a polite suggestion; the new reality is a contractual agreement with a billing endpoint. This is the commercialization of the very act of reading.

Matthew Prince is right that the model has to change. But “pay to crawl” is just the first, most obvious chapter in a much longer, more contentious story. It’s the starting pistol for a race to control the economic layer of the AI age. The real question isn’t whether we’ll pay, but what we lose when the free, chaotic, human-driven web becomes a series of gated data utilities. The internet as a public square is being replaced by the internet as a private, metered data mine. We’re not just losing a traffic metric; we’re losing the ghost in the machine—us.

当Cloudflare CEO马修·普林斯说互联网的未来是“付费爬取”时，他恐怕不是在危言耸听，而是在陈述一个已经发生的残酷现实。他晚了几年才得出这个结论——机器人流量在2027年底之前就会超越人类——但这迟到的预警反而显得更加沉重，因为它已经被市场用真金白银和无数个被改写的服务器日志验证了。机器人流量霸占互联网，这早已不是科幻小说里的情节，而是每天都在发生的、静悄悄的政变。

这场政变的主角，就是他口中那些“AI智能体”。它们不再是过去那种笨拙的、只会按规则抓取页面的爬虫。它们更聪明、更高效、更饥渴。它们是为了训练下一代模型而诞生的“数据饕餮”，是为了执行自动化任务而不知疲倦的“数字劳工”。它们24小时不间断地访问、解析、下载，将人类生产的内容——那些文章、图片、代码、对话——源源不断地抽干，倒进某个庞然大物的消化系统里。我们还在讨论版权归属时，智能体们早已完成了内容的搬运、重组与再生产。这场流量战争，从来就不公平。

普林斯那句“显然是付费爬取”，听起来像是一个解决方案，但更像一个早已标好价格的判决书。互联网最初那个开放、平等、共享的乌托邦梦想，在算力和资本的双重碾压下，正迅速瓦解。未来图景已经清晰：一堵由付费墙构筑的数字长城拔地而起。想要数据？拿钱来。想要训练资料？请先签合同并支付许可费。API调用按次计费，模型微调按时长收费。这不再是信息高速公路，而是数据垄断集团的收费站。

这直接引爆了几个更深层次的矛盾。首先，它加速了互联网的“封建化”。大型科技公司——它们既是AI的制造者，也是最优质的付费用户——将轻松获取海量数据，而中小开发者、学术机构、独立创新者则可能被高昂的“数据税”挡在门外。创新的门槛被空前抬高，游戏从“看谁有创意”变成了“看谁有预算”。其次，它可能扭曲我们整个信息生态。当优质内容被锁在付费墙后，公共互联网留下的，是否会是一片由低质量AI生成内容、广告和垃圾信息构成的荒漠？人类创作的独特性和深度，其价值是否会在一次次“付费爬取”的交易中被折算、贬低甚至消失？

最辛辣的讽刺或许在于：正是这些AI智能体，它们赖以生存的“养分”来自于人类过去几十年无偿或低成本分享在网上的知识、智慧与创造力。它们喝着开源的奶长大，现在却要反手给下一代断奶，并修建起收费的水龙头。人类为AI提供了成长的温床，AI长大后却率先改变了人类赖以交流和获取信息的环境的基本规则。这是一场典型的“技术反噬”，技术的受益者，往往也是技术所重塑世界的最被动承受者。

普林斯作为一家网络安全和性能公司的CEO，他的商业嗅觉是敏锐的。Cloudflare早已在为“爬虫管理”提供付费解决方案。他的预言，既是观察，也是推销。这无可厚非，但我们需要清醒地看到，当“付费爬取”从预测变为行业标准协议时，权力已经完成了新一轮的集中。互联网的“公共性”将彻底让位于“商业性”，成为巨头们进行数据资源分割和贸易的私有市场。

所以，别再为“机器人流量超过人类”这个事实本身感到惊讶了。真正的风暴，是这个事实引发的连锁反应。它标志着互联网一个时代的终结，和另一个更昂贵、更封闭、更由算法和算力支配的时代的开启。我们每个人，都即将或已经生活在这个“付费爬取”的新现实里，成为数据流通链条中，那个最被动、也最昂贵的环节——毕竟，我们生产内容，而AI消费并贩卖着这一切。未来的互联网，或许会更高效、更强大，但它一定会更贵，也更冷。