AI Search Visibility (GEO): What Is Actually Documented

Generative Engine Optimization (GEO) is the practice of making a website visible in AI-generated answers: Google's AI Overviews and AI Mode, ChatGPT search, Perplexity, and similar tools. This page collects what the providers themselves actually document, with a source and a capture date for every claim. The short version is uncomfortable for anyone selling GEO services: crawler access and indexing are verifiable prerequisites, but no provider (Google, OpenAI, Anthropic or Perplexity) documents why an eligible page gets cited. Anything that claims otherwise is inference, not documentation.

Everything below was checked against the linked sources on 17 July 2026. Provider documentation changes often (Google's own AI-optimization guide was last updated on 10 July 2026), so treat the capture dates as part of each fact, and re-check the primary sources before making decisions.

Google's documented rules: eligibility, not selection

Google's central reference is AI features and your website (last updated 10 December 2025, checked 17 July 2026). Two statements in it undercut most GEO sales pitches:

• "There are no additional requirements to appear in AI Overviews or AI Mode…" A page qualifies if it is indexed and eligible to show a snippet. Nothing extra.
• "You don't need to create new machine readable files, AI text files, or markup…" No special file unlocks AI visibility.

Eligibility runs through controls you already have:

• The same snippet controls that govern ordinary search results (nosnippet, data-nosnippet, max-snippet, noindex) also govern whether your content can appear in AI features. A stray max-snippet:0 in a template can quietly remove a whole site from AI answers; the meta tags analyzer will show what a page actually declares.
• Google-Extended is a separate robots.txt token that, per the same document, limits "AI training and grounding in some of Google's other systems". Blocking it does not remove pages from Google Search or from Search results' AI features, which are crawled by ordinary Googlebot.

In Optimizing your website for generative AI features on Google Search (last updated 10 July 2026), Google adds qualitative guidance: unique, non-commodity, first-hand content; useful images and other media; the standard technical requirements that apply to normal Search; and monitoring in Search Console. The same guide knocks down three circulating claims in plain words:

• "Structured data isn't required for generative AI search…" (Structured data still earns rich results in classic Search; see the schema markup guide for what it does document.)
• "Google systems are able to understand the nuance of multiple topics on a page", so fragmenting every subtopic onto its own thin page is not a requirement.
• "Seeking inauthentic 'mentions' across the web isn't as helpful as it might seem."

Notice what is missing: none of this says how AI Overviews chooses between two eligible pages. Google documents eligibility plus content-quality guidance, and stops there.

Training bots vs visibility bots: know which one you are blocking

Every major provider now runs at least two kinds of crawler, and confusing them is the most expensive robots.txt mistake in AI search. A training bot collects content to train future models; blocking it is a data-licensing decision. A visibility bot (search bot) indexes content so it can be shown and cited in AI answers; blocking it removes you from those answers. A user-triggered fetcher retrieves a page live when a user asks about it, and may not obey robots.txt at all. The split, as each provider documents it (all sources checked 17 July 2026):

Provider	Training bot (blockable independently)	Visibility bot (blocking removes you from AI answers)	User-triggered fetcher
OpenAI	GPTBot	OAI-SearchBot	ChatGPT-User (robots.txt may not apply)
Anthropic	ClaudeBot	Claude-SearchBot	Claude-User
Perplexity	None documented (Perplexity states PerplexityBot is not used for training)	PerplexityBot	Perplexity-User ("generally ignores robots.txt")
Google	Google-Extended	Googlebot (ordinary Search crawling covers AI features)	None documented

The load-bearing documented details:

• OpenAI (bots documentation): "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links." OpenAI notes that robots.txt changes can take around 24 hours to propagate to its systems, and publishes IP address lists (for example openai.com/searchbot.json) so you can verify which traffic is genuinely theirs.
• Anthropic (Anthropic Help Center): blocking Claude-SearchBot "may reduce your site's visibility and accuracy in user search results". Anthropic states its crawlers respect robots.txt "do not crawl" signals and support the Crawl-delay directive.
• Perplexity (crawler documentation): PerplexityBot builds the index that powers citations and is not used for model training; the separate Perplexity-User agent fetches pages on a direct user request and "generally ignores robots.txt". Perplexity publishes its IP list at perplexity.com/perplexitybot.json.
• Google: there is no separate AI-visibility bot to allow. If Googlebot can crawl and index a page, that page is eligible for AI features, subject to the snippet controls above.

Practical consequence: check your robots.txt bot by bot, not with a single wildcard assumption. Run each user-agent above through the robots.txt tester, and if you need to write or adjust rules, the robots.txt generator produces the per-agent groups (background reading: the robots.txt guide). One more trap: firewall and bot-management rules (Cloudflare and similar) can block these crawlers invisibly, with a robots.txt that looks wide open. The published IP JSON lists exist precisely so you can audit that layer.

What eight well-known sites actually allow: an original dataset

To show how these policies play out in the real world, we fetched each site's live /robots.txt on 17 July 2026 and evaluated access to the site root for six AI-relevant user-agents, following the Robots Exclusion Protocol including shared user-agent groups. The check is reproducible by anyone with the HighSEOTools robots.txt tester.

Domain	GPTBot	OAI-SearchBot	ClaudeBot	Claude-SearchBot	PerplexityBot	Google-Extended
en.wikipedia.org	Allowed	Allowed	Allowed	Allowed	Allowed	Allowed
github.com	Allowed	Allowed	Allowed	Allowed	Allowed	Allowed
highseotools.com	Allowed	Allowed	Allowed	Allowed	Allowed	Allowed
medium.com	Blocked	Allowed	Blocked	Allowed	Allowed	Allowed
stackoverflow.com	Blocked	Blocked	Blocked	Blocked	Blocked	Blocked
www.bbc.com	Blocked	Blocked	Blocked	Allowed	Blocked	Blocked
www.nytimes.com	Blocked	Blocked	Blocked	Blocked	Blocked	Blocked
www.reddit.com	Blocked	Blocked	Blocked	Blocked	Blocked	Blocked

How to read this snapshot:

• medium.com is the training-versus-visibility split in action: it blocks the training bots (GPTBot, ClaudeBot) while allowing the visibility bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot). That is a deliberate policy: no free training data, but stay citable in AI answers.
• reddit.com and stackoverflow.com serve a blanket User-agent: * / Disallow: / to generic clients (both license or whitelist crawler access separately), so every bot reads as blocked in the public file. The public robots.txt is not the whole story for sites with commercial data-licensing deals.
• The two news publishers differ: nytimes.com blocked all six agents at the check date, while bbc.com blocked five but allowed Claude-SearchBot.
• This is a dated snapshot, not a permanent record. robots.txt files change without notice; treat 17 July 2026 as part of every cell in the table.

llms.txt: adoption without consumption

llms.txt is a proposed convention: a plain-text file at the domain root offering AI systems a curated summary of the site. It is heavily promoted in GEO articles, so here is the entire documented evidence base as of 17 July 2026:

• Google's position: the generative-AI optimization guide (10 July 2026) states that an llms.txt file neither helps nor harms a site's visibility in Google's generative-AI features.
• Google's John Mueller, asked about llms.txt in June 2026: "I don't think anyone knows – it's purely speculative for now (the file has existed for years, yet none of the AI systems use it — what does it mean?)" (Search Engine Journal, 2 June 2026).
• The log data: an Ahrefs study published in June 2026, using May 2026 data across 137,210 domains, found that 97% of llms.txt files received zero requests in the month; only about 1,100 of roughly 38,000 valid files got any traffic at all; 96% of the requests that did arrive came from bots, and AI retrieval bots accounted for about 1% of requests.

Neither OpenAI nor Perplexity documents any use of llms.txt, and neither has published an explicit denial; the case against the file rests on absent documentation plus the log data above. Creating one appears harmless. Expecting visibility from it is, on current evidence, unfounded.

How big AI search is, and what it does to clicks

Scale first. In the Google I/O keynote published 19 May 2026 (blog.google, checked 17 July 2026), Google stated that "AI Overviews now has over 2.5 billion monthly active users" and that AI Mode, within about a year of launch, had "surpassed 1 billion monthly active users". Whatever you think of AI answers, they are now a mainstream search surface.

Click evidence is the documented worry. Pew Research Center analysed real browsing data from 900 US adults in March 2025, covering 68,879 Google searches (published 22 July 2025). When an AI summary appeared, users clicked a traditional result link in 8% of visits, versus 15% when no summary appeared. Links inside the AI summary itself were clicked in 1% of the visits where a summary was shown.

Two honest caveats before extrapolating: the Pew panel is US-only and predates AI Mode's 2026 growth, and there is no official Google click data to compare it against, for the reason explained next.

Measuring AI visibility in Search Console

On 3 June 2026, Google launched generative-AI performance reports in Search Console, covering AI Overviews and AI Mode on Search plus a separate report for generative-AI features in Discover (Search Central blog, 3 June 2026). What the reports contain, per Google's help documentation (checked 17 July 2026):

• Impressions only, broken down by page, country, device and date. The help page defines impressions as how many times links to your site were shown to a user in a generative AI feature, and lists no click metric: no clicks, no CTR, no query data.
• No historical backfill before the reports' data begins, per the launch post.
• Limited rollout: Google is rolling the report out to a subset of website owners, and a site also needs enough generative-AI impressions before the report appears at all.

Secondary coverage names a specific mid-May 2026 data-start date, but we could not confirm any exact date in Google's own documentation, so this article does not state one.

Announced alongside the reports on 3 June 2026: a publisher control to opt content out of Google's AI search features, a remedy secured by the UK's Competition and Markets Authority (TechCrunch, 3 June 2026; CMA announcement). It reaches UK publishers first, and per Google the choice "will not be used as a ranking signal".

What to do with this: if you run a Search Console property, check whether the generative-AI report has reached it yet and record a baseline month of impressions, because with no backfill, history starts when you start looking. Our Search Console guide covers property setup. And since indexing remains the prerequisite for everything on this page, the index pages checker is a quick way to see how many of your pages Google currently lists.

A dated, reproducible test with our GEO checker

Our free GEO checker scores a page against deterministic heuristics associated with quotable, extractable content: depth, headings, question-style headings, lists, an answer-first opening, sentence length, concrete numbers, and an early definition. To keep this article honest, we ran it live and publish the raw output rather than a cherry-picked example.

Test run, 17 July 2026. Target: the English Wikipedia article "Search engine optimization". Overall score: 56 / 100.

Signal	Measured	Result
Content depth	6,072 words	Pass
Headings	17	Pass
Question-style headings	0	Fail
Lists detected in extracted text	0	Fail
Answer-first opening	Not detected	Fail
Average sentence length	12 words per sentence	Pass
Statistics and specifics	565 numeric references	Pass
Clear definition near the top	Not detected	Fail

Read the failed rows carefully, because they demonstrate the limits of every heuristic tool in this space, ours included. The Wikipedia article opens with a textbook definition and contains plenty of lists, yet the checker's plain-text extraction, which sees the page roughly the way a simple fetcher does, navigation text included, did not register them. That is exactly why a heuristic score is a prompt for human review, not a prediction of citation. No tool can predict citation, because no provider documents the criteria. Run the GEO checker on your own pages with that framing in mind, and pair it with the broader website SEO score checker for the conventional signals.

What nobody documents: the known unknowns

This section matters more than any checklist above. As of 17 July 2026:

• Citation criteria are undocumented everywhere. Google, OpenAI, Anthropic and Perplexity all document crawler behaviour and (in Google's case) eligibility plus qualitative content guidance. None publishes how an eligible page is selected for citation in an AI answer. Every "AI citation ranking factors" list circulating is correlational inference, not documentation.
• Click impact cannot be measured from official data. Google's generative-AI reports are impressions-only, so third-party panels such as Pew's are currently the only click evidence, with the sampling caveats noted above.
• The Search Console report's exact data-start date is unconfirmed. It is not stated in Google's help documentation; we deliberately omit the date circulating in secondary coverage.
• The AI opt-out's global timeline is unknown. The CMA remedy is UK-publisher-first; there is no announced date for availability elsewhere, India included.
• OpenAI and Perplexity have no stated position on llms.txt. The evidence against the file is log data and absent documentation, not an official denial.
• Bing and Copilot are out of scope here. Microsoft's crawler and AI-answer documentation deserves the same treatment and is not yet covered in the tables above.

The bottom line

The actionable, verifiable core of GEO is short: keep pages indexed and snippet-eligible; decide bot by bot what your robots.txt allows, knowing that training bots and visibility bots are different things; audit the firewall layer against the published IP lists, not just robots.txt; record a Search Console impressions baseline when the generative-AI report reaches you; and treat any promise about "ranking in AI answers" with suspicion, because the selection criteria are published nowhere. Every claim on this page was last verified against its source on 17 July 2026. Provider documentation and dashboards change; this page will be updated when they do.