“Best recipe for banana bread.” “How to fix running toilet.” “Is Harry Styles engaged?”
For decades, a well-established business model allowed internet users to answer big life questions like these. Search engines directed readers to content on other websites—and those visits helped support content through advertisements, subscriptions, and product sales.
AI-assisted search is already threatening the once-symbiotic relationship between search engines and content producers. On Google, for example, many queries now result in an AI Overview. These summaries rely on information taken from other websites, but they allow users to get the answers they seek without ever clicking away from Google. Some users are now bypassing search engines altogether, posing questions directly to tools like ChatGPT.
The erosion of website traffic is a problem for everyone, explains Soheil Ghili of Yale SOM. If there is no financial incentive to produce new content, large language models will have only outdated information to draw on for training and/or grounding, and the models’ usefulness will suffer. “A product that doesn’t have a market doesn’t get produced, at least not in an efficient amount,” Ghili says. As a result, “we might just get both lower-quality and lower-quantity content.”
One solution to this problem is a bulk deal. Reddit, for example, has licensed its content to both Google, charging a reported $60-70 million per year for the privilege of crawling its content. But that playbook won’t work in most situations, Ghili points out: “You cannot go around and negotiate a deal with every single small website.”
Pay-per-crawl is a new approach, pioneered by companies including Cloudflare and Tollbit, that allows content producers to charge AI crawlers an access fee for visiting a page. Figuring out how to determine the right fee is the subject of a new working paper by Ghili, Nima Haghpanah of Yale SOM, and graduate student Richard Archer.
Publishers could, of course, charge AI crawlers the same price for every piece of content on their website, but that’s probably not the best strategy—it’s reasonable to assume content varies in its usefulness for large language models. But it’s also not feasible for publishers to manually price every piece of content they produce.
Using real content from a German technology publisher called HardwareLuxx, Ghili and his collaborators developed an AI tool that can, at scale, help publishers find the optimal pay-per-crawl price for each article—one that will maximize revenue while remaining palatable to the crawlers. This agent, which they call the LM Tree, identifies the attributes that make content valuable and sets prices based on those characteristics.
To put the LM Tree to the test, the researchers built a simulated market on top of HardwareLuxx’s real articles. Each article was assigned a hidden value—what an AI crawler would be willing to pay to access it—and the agent never saw that value directly. All it received was binary feedback: at a proposed price, would a crawler buy this article or not? From nothing but the article’s text and a handful of these accept/reject signals, the LM Tree had to work out what each piece of content was worth.
The researchers compared the LM Tree against strategies that price content using only the website’s structured information—the topic categories HardwareLuxx already assigns, like “graphics cards” or “processors.” The LM Tree’s advantage is that it also reads the prose, and from the text it discovers pricing rules that no column in the dataset contains.
One example: HardwareLuxx’s taxonomy has a category for graphics cards, but nothing that singles out flagship graphics cards. The agent learned on its own that articles about high-end, flagship GPUs should command higher prices—even though “this is about a flagship GPU” is not a field anywhere in HardwareLuxx’s data. Strikingly, the word “flagship” never has to be defined in advance for the rule to work. A large language model’s general world knowledge lets it recognize a flagship GPU when it reads one, which means the same rule can automatically tag any future article—including content the agent has never seen—without anyone writing down a list of what counts as “flagship.”
Next, the researchers set the LM Tree loose on a new set of articles and compared its revenue to other pricing strategies. Charging the same price for every article performed worst. Pricing by format (news or review) did better. The LM Tree’s custom, text-based prices did best of all—generating substantially more revenue than a single static price.
Ghili says the LM Tree’s approach could easily be adapted to other settings. “There’s an ocean of content,” he says, and all of it has industry-specific quirks. Within law-focused content, for example, “maybe jurisdiction or the subfield within the legal arena matters—articles on mergers and acquisitions would go for higher prices than immigration law.” The LM Tree makes it possible to identify optimal prices for each kind of article at scale.
Ghili hopes this research can help support an emerging market as it takes shape. The pay-per-crawl industry may be nascent, but it is already showing signs of growth—and provides a solution to a clear market need. “A microtransaction approach is just a sensible one,” he says. “We are hoping to contribute to that process and provide more intelligent approaches to finding the right price.”
“The Yale School of Management is the graduate business school of Yale University, a private research university in New Haven, Connecticut.”
Please visit the firm link to site
