Kagi's Paywall Toggle Makes Content Choice a Discovery Feature
Until now, what your search surface returned was an engineering accident. Kagi's new setting turns it into a product decision.
AI-written, human-edited, never fabricated. How this is made
Most discovery teams I talk to have never opened a crawler config. They write the prompt, ship the ranking tweaks, argue about retrieval over coffee, and assume that whatever lives on the open web will surface in the product. The wall between "what we index" and "what the user actually sees" has been, for years, someone else's problem. A backend default. A tax on relevance.
That wall just became a feature. Last week, Kagi shipped a setting that lets a user exclude paywalled links from their own search results. One toggle. Per user. Per session. It is a small line in a changelog, and it hit 1,252 points on Hacker News, which tells you how starved people are for a product team to admit that content provenance is a choice. Not a backend default, not a quirk of the crawler, but a choice the user gets to make.
Discovery is no longer a search problem
If you build anything that helps someone find a thing on the internet, you already know the feeling. A user asks for "the latest benchmark on retrieval-augmented generation," and your system cheerfully returns a half-dozen pages, of which two are paywalled journals, one is a recycled SEO post, and one is a vendor blog that quotes the other paywalled journal. From the user's chair, every one of those results looked like an answer. None of them were.
Kagi's move forces the question every discovery PM has been quietly dodging: which of those four results did you actually intend to surface? When a user opts out of paywalled links, the system is no longer ranking. It is curating. That is a different product, with different success metrics, different edge cases, and a different kind of bug report when it goes wrong.
The second-order consequence is the one that will eat your quarter. If paywall status is a user-facing control, every downstream summarization step has to respect it too. You cannot have a discovery surface that filters paywalled links out of the list and then cheerfully cites them in the generated answer. The provenance decision is now upstream of the model call. That is not a retrieval tweak; it is a spec change. As we covered last week, provenance has been drifting toward the PM seat for months. This is the moment it sits down and stays.
The new spec lives in your settings page
So what does the Monday-morning version of this look like, for a PM whose roadmap did not budget for it? Three small things, none of which require an engineer to ship.
First, write down the policy. Not the crawler policy, the product policy. For every category of content your surface can return, write one sentence: paid, free, mixed, syndicated, user-generated. If you cannot write the sentence, you have not made the decision, and your users are making it for you, badly. Second, decide what the user gets to override. Kagi's bet is that power users want the dial. Yours might be a quieter default with a discoverable escape hatch. Both are defensible; neither is free. Third, name the failure mode. The default today is "everything looks like an answer." Your new default is "the user opted out of X, and the system respects that across every surface, including the generated summary." Write that down before someone files a bug about it.
None of this requires a new model. It does not require a new vendor. It requires someone on your team to decide what counts as a result, and to write that decision into a place the user can see.
Why this matters beyond search
The interesting pressure is that Kagi is doing this for search, and the lesson travels. Any product that mixes open and gated content, which is most of them, now has a precedent for treating the gate as a first-class control. News readers. Research assistants. Shopping surfaces that mix marketplace and brand-direct results. Code tools that surface public repos alongside enterprise documentation. In each case, the question is the same: is the provenance of what you returned a thing the user can see and change, or a thing you buried in a config file and hoped no one noticed?
The teams that ship this well will look like they thought about it. The teams that ship it badly will be the ones whose AI summarizer quietly cites a paywalled source the user has explicitly told them to ignore. That bug, when it lands, will not be filed as a retrieval issue. It will be filed as a trust issue, and trust issues eat roadmaps.
Paywall status is no longer a crawler's problem. It is your spec.
The fastest way to start is unglamorous: open your discovery flow, point at every result type you surface, and write a one-line product policy for each. Then decide which of those lines the user gets to edit. You will not finish this in a sprint. You will, however, finally know what your product returns and why.