Agency growth

AI crawlers and robots.txt: a guide for agencies and their clients

Clients are starting to ask their agencies about AI search. One of the simplest and most valuable things you can review with them is which AI crawlers their site lets in. It is also a great way to open a sales conversation.

By Geolytic team · 2 November 2026 · 5 min read

Why AI crawlers matter to your clients

AI assistants need to fetch pages before they can quote them. Each AI company runs its own crawlers and robots.txt decides which ones may visit. A single line copied from a forum years ago can quietly block a client from ChatGPT search or Claude answers. Most business owners have no idea this is happening, which is exactly why it makes such a strong talking point for an agency.

The main AI crawlers you should know

The names below come from each company's public documentation. The key distinction is between crawlers that fetch pages for live answers and crawlers that collect data for model training.

OpenAI

OAI-SearchBot fetches pages for ChatGPT search results.
GPTBot collects content that may be used to train OpenAI models.
ChatGPT-User makes requests when a person asks ChatGPT to visit a page.

Anthropic

Claude-SearchBot fetches pages to improve search answers in Claude.
ClaudeBot collects content that may be used for training.
Claude-User makes requests when a person asks Claude to visit a page.

Google, Perplexity and others

Google-Extended is a robots.txt token that controls whether content may be used for Gemini training. It does not affect Google Search or ranking.
PerplexityBot crawls pages for Perplexity search answers.
Googlebot and Bingbot still matter most for classic search and they also feed many AI search features.

Search access versus training access

This is the conversation most clients have never had. They can allow AI search crawlers so they appear in answers while still blocking training crawlers if they do not want their content used to build models. It is a business decision, not a technical one. Your job is to explain the trade off clearly and implement what the client chooses.

Never switch on training access for a client without their agreement. Some publishers and brands have strong reasons to opt out. Respecting that builds trust.

The most common robots.txt problems we see

A blanket block on all crawlers left over from a staging site that went live.
Blocking GPTBot and assuming that also covers ChatGPT search, or the reverse.
Blocking every AI crawler after a news story, then expecting to appear in AI answers.
Important folders such as blog or services blocked by accident.
No sitemap line, so crawlers have to discover pages the slow way.

How to turn this into a client pitch

Prospects rarely buy because of a long list of generic SEO issues. They respond to one or two clear problems that affect them today. AI crawler access is perfect for this because the evidence is concrete. You can point at a single line in their robots.txt and explain what it blocks.

Run a quick scan on the prospect's site before your first call.
Lead with the most critical finding, such as a blocked AI search crawler or missing structured data.
Show the exact evidence: the robots.txt line or the page where the issue appears.
Explain the business impact in one sentence they can repeat to their boss.
Offer a small, fixed scope first project to fix the critical items.

Keep your claims honest. Fixing crawler access makes a site readable by AI search. It does not guarantee mentions. Prospects trust agencies more when they hear the limits up front.

Using Geolytic for pitch audits

The free Pitch Audit was built for exactly this workflow. Enter a prospect URL and Geolytic runs a rapid three page scan that checks AI crawler access, llms.txt presence, structured data and other critical signals. The summary shows only the failing checks that matter most, each marked Fix Required with its evidence. You can share a read only link that expires after 30 days or download a PDF to send after the call.

When the prospect becomes a client, a full GEO audit gives you a complete crawl with SEO and GEO readiness scores. Agency plans add client workspaces and branded PDF reports so your deliverables carry your logo and colours.

A simple monthly routine for retained clients

Recheck robots.txt after every site release or CMS update.
Rescan key pages and compare scores with the previous month.
Review whether new AI crawlers have been announced by the major providers.
Ask the client whether their stance on training access has changed.
Report findings in plain language with the evidence attached.

A robots.txt review checklist for agencies

Use this checklist on every new client and after every major site change. It takes about fifteen minutes and often uncovers the single most valuable fix in the whole audit.

Open the live robots.txt and save a dated copy for your records.
Look for a blanket block on all user agents. If found, confirm whether it was intentional.
List every AI crawler mentioned and note whether it is allowed or blocked.
Separate search crawlers from training crawlers in your notes.
Check that important folders such as services, blog and product pages are not blocked.
Confirm a sitemap line points to a working sitemap on the same domain.
Ask the client to confirm their stance on training access in writing.
Make the agreed change, then fetch the live file again to confirm it deployed.

Write the client decision down

Record the client choice for each crawler group in your project notes. If someone at the client later asks why their content appears in AI answers, or why it does not, you can point to the documented decision. This protects both of you.

Watch for CMS and plugin changes

Many content management systems and security plugins edit robots.txt automatically. A plugin update can quietly add new blocks. Schedule a monthly check and compare against your saved copy. Small agencies that catch these changes early look very professional to clients.

Pair crawler access with readable pages

Allowing crawlers is only the first step. If key pages hide their content behind scripts that never render, use vague headlines or lack structured data, the crawler arrives and finds little worth quoting. Combine the robots.txt review with a technical SEO audit so access and readability improve together.

Keep learning

If your clients ask what all of this adds up to, share our introduction to generative engine optimization. For questions about the llms.txt file that often comes up in the same conversation, see our llms.txt guide.