AI crawlers are the automated bots — GPTBot, ClaudeBot, PerplexityBot, and others — that AI companies send to read and index website content, separately from traditional search engine crawlers like Googlebot. An llms.txt file is one of the few tools that lets a site owner tell those crawlers, directly, what to read and what to skip.
- ▪AI crawlers are bots like GPTBot and ClaudeBot that read your site for AI training or retrieval — distinct from Googlebot.
- ▪Small, choppy demand: 200 US searches/mo, down from an initial 345 spike but holding steady around 210–220 since spring.
- ▪No recorded CPC — a purely technical, not commercial, search — but a fortress top five: GitHub, YouTube, Cloudflare, Reddit, all DR 93+.
- ▪KD 23 — genuinely winnable against that authority, because the content itself, not backlinks, is what’s thin.
- ▪Our edge: we implement llms.txt and crawler-access controls as part of the same tagging work that already governs a site’s data layer.
Before you decide whether an llms.txt file is worth adding to your own site, it helps to know exactly which bots are reading it in the first place — and most site owners genuinely do not.
The emergence
Demand opened at 345 US searches a month last July, dropped hard through the back half of 2025, and has settled into a steadier 200–230 range since spring 2026. That shape — an early spike followed by a plateau — reads as the early-curiosity phase resolving into a smaller, ongoing technical audience.
The commercial pull
There is no recorded CPC — nobody advertises against “AI crawlers.” This is pure technical-reference search: developers and site owners trying to understand what is already hitting their servers, not a purchase decision.
Who’s competing for attention
One of the strongest top fives we track — GitHub (DR 97), YouTube (DR 99), and two separate Cloudflare posts (DR 93–94) alongside a DR-95 Reddit thread. This is infrastructure-provider territory: the companies that see the crawler traffic first are the ones publishing about it.
Growth or decline
Down from an early spike but stable since — the pattern of a topic that found its real, smaller audience rather than one that is dying. Expect this to track AI crawler adoption itself: as more bots launch, the baseline interest should hold or rise slowly.
| llms.txt (the standard) | AI crawlers (what it’s for) | |
|---|---|---|
| What it is | A file at your site root | The bots actually reading that file |
| Question it answers | How do I publish one? | Who is even visiting to read it? |
| Search trend | ▼ 86% from peak | Spiked, then found a plateau |
| Practical first step | Add the file | Check your logs for who’s already crawling |
How PPC Snobs executes here
We treat crawler visibility as part of the same tagging discipline we apply everywhere else — auditing server logs for which AI bots are already indexing a client’s site before we recommend an llms.txt file, a robots.txt update, or nothing at all, because the honest answer is sometimes “you’re already invisible to them for a different reason.”
You cannot write a sensible llms.txt file for crawlers you have never actually looked for in your own logs.