New working paper with Pengxiang (Shawn) Zhou and
@GoliAlii : "LLMs as Gatekeepers: Source Concentration, Factual Quality, and Political Slant in Information Search." (
dadepro.github.io/papers/llm…)
More and more, people ask ChatGPT, Perplexity, Gemini, Claude, or Grok instead of searching. That means these tools increasingly decide which news outlets and media sources people actually see. Focusing on the news and media ecosystem, we looked at how AI compares to regular search — and what happens as publishers start blocking AI crawlers.
A few things we found:
1. AI platforms send traffic to a far more concentrated set of sources than organic search (3–8× higher HHI), but currently of higher average factual quality and comparable political slant.
2. The publishers blocking AI crawlers are disproportionately the largest, most factually reliable, and centrist-to-left outlets. Their exit shifts the available pool toward higher concentration, lower factual quality, and rightward slant.
3. Blocking Perplexity's crawler cuts a domain's referral traffic by 46–48%. That traffic flows back to the largest remaining outlets, reinforcing concentration rather than diluting it.
4. The result: redistribution reinforces concentration while failing to undo the shift toward less factual and more right-leaning publishers.