For two decades, the deal between publishers and the web was simple, if lopsided: let the bots crawl, and in return they send readers your way. Search engines indexed your content and rewarded you with traffic; you monetised that traffic with ads and subscriptions. It was never a fair trade, but it was a legible one. AI answer-engines have quietly broken it. They ingest your work, synthesise it into a tidy paragraph, and serve the answer without the click. The crawl still happens; the traffic does not.
Cloudflare, which sits in front of a large slice of the internet’s traffic, has been trying to fix the incentives. Its latest move reframes the whole conversation — away from tolling the bots at the door and toward paying creators when their words actually get used. Here’s what it is, why publishers are watching, and what it could mean for India’s beleaguered content businesses.
The shift
According to The Neuron, citing Cloudflare, the company has launched an effort to make AI search smarter while compensating creators — moving from a ‘Pay Per Crawl’ model toward ‘Pay Per Use’, and charging AI systems when they use publishers’ content in generated answers.
The distinction matters more than it might sound. ‘Pay Per Crawl’ is a tollbooth: it charges an AI company for the act of fetching a page, regardless of whether that page ever influences an output. It treats access as the billable event. ‘Pay Per Use’ moves the meter downstream, to the moment content is actually consumed inside a generated answer. In principle, a publisher gets compensated when a chatbot leans on their reporting to answer a user’s question — the point at which value is genuinely extracted.
What makes Cloudflare unusually placed to attempt this is its vantage point. It already sees an enormous volume of requests and can identify which crawlers and AI agents are hitting which sites. Pairing that network-level signal with better AI search infrastructure — the ability to serve, rank, and account for content as it flows into answer engines — is the technical bet underpinning the model. If Cloudflare can observe both the demand side (AI systems seeking content) and the supply side (publishers offering it), it can plausibly sit in the middle as a clearing house. Whether the accounting can be made precise enough to bill on is the open question, but the architecture is coherent.
Why publishers need it
The urgency is not abstract. As The Neuron notes, the move responds directly to AI answer-engines reducing referral traffic to publishers. The mechanism is now familiar to anyone running a content site: a reader asks a question, an AI system answers it inline using material scraped from across the web, and the reader never visits the source. The information is delivered; the visit is disintermediated.
This is the collapse of the crawl-for-traffic bargain. Under the old arrangement, letting bots read your site was rational because indexing produced discovery, and discovery produced clicks. AI answer-engines keep the first half of the deal — they still read everything — while quietly abandoning the second. They no longer need to send you a reader to extract your value. For publishers whose economics depend on pageviews, that is not a slow decline; it is a structural threat to the business model.
Usage-based compensation is the proposed alternative. Instead of monetising the attention of a reader you can no longer count on receiving, you monetise the utility of your content wherever it is consumed — including inside someone else’s chatbot. In theory, this decouples revenue from traffic. A story that gets cited in a thousand AI answers but generates zero clicks would still earn something. That is a genuinely different content economy: one where being useful to a machine is itself a billable event, rather than a favour you grant in the hope of downstream clicks.
It is worth being clear-eyed. This is a proposal and an early rollout, not a solved market. But it is the first serious attempt by an infrastructure player to price the thing that AI companies have so far taken for free.
The open questions
The idea is elegant. The execution is where it gets hard, and there are at least three unresolved problems.
- Pricing. What is a single use of a publisher’s content actually worth? A paragraph of breaking news, a recipe, a stock quote, and a deeply reported investigation are not economically equivalent, yet a per-use meter risks flattening them. Set the price too low and it is a rounding error that changes nothing; set it too high and AI companies simply route around participating publishers or lean on cheaper sources.
- Attribution and enforcement. To bill for use, you must prove use — that a specific answer drew on a specific source. AI outputs blend many inputs, often without clean citations, and models can restate facts without copying phrasing. Establishing what counts as ‘use’, measuring it reliably, and enforcing payment against companies that would rather not pay is a formidable technical and legal challenge. Cloudflare’s network position helps with detection, but it does not automatically settle disputes.
- Who participates. A payments rail only works if the AI companies plug into it. The largest players have the leverage to negotiate their own bilateral licensing deals with big publishers and may see little reason to accept a third-party meter. Smaller AI startups may lack the budget. If participation is partial, publishers are left with a patchwork of paying and non-paying consumers of their work.
Looming over all of this is the question of open-web access. A world where content is metered by use is safer for publishers but potentially poorer for everyone else — harder to scrape for research, for archiving, for the small builders who made the open web generative in the first place. The healthiest outcome is one that compensates creators without quietly erecting paywalls around human knowledge. That balance is not guaranteed.
The India read
Indian publishers and creators face the same traffic squeeze as their Western counterparts — arguably a sharper version of it. Digital news and content businesses here have long operated on thin margins, heavily dependent on search and social referrals and on programmatic ad rates that are already low by global standards. When AI answer-engines cut into that referral traffic, there is far less cushion to absorb the shock. The vernacular publishing boom, the explosion of regional-language creators, the newsletter and niche-media wave — all of it was built on the assumption that good content earns an audience it can then monetise. AI answers threaten that chain at its first link.
A usage-based model is therefore attractive precisely because it offers a way out of pure traffic dependence. For an Indian creator whose Hindi explainer or Tamil how-to gets absorbed into an AI answer for a user who never clicks through, per-use compensation would convert an invisible loss into visible revenue. It reframes the value of content as its utility, not its ability to win a fickle click — a reframing that suits markets where distribution is dominated by a few large platforms.
But it also raises the stakes on two things Indian publishers have historically underinvested in: content provenance and licensing. To get paid when a machine uses your work, you must be able to prove the work is yours and specify the terms on which it may be used. That means clean rights records, structured licensing signals, and the willingness to treat your archive as a licensable asset rather than just SEO fuel. Publishers who get their provenance and licensing house in order now will be positioned to collect if these markets mature. Those who don’t may find their content used everywhere and paid for nowhere.
Whether Cloudflare’s specific implementation wins is almost beside the point. The bigger signal is that the industry has finally started pricing what AI takes. For publishers everywhere — and especially in India, where the margin for error is slim — that shift is worth watching closely.
