<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[Seema Sharma]]></title><description><![CDATA[Notes from a digital marketer learning AI in public. Seema Sharma writes about the technical side of AI search: how AI crawlers read websites, robots.txt and ll]]></description><link>https://seemasharmaai.hashnode.dev</link><image><url>https://cdn.hashnode.com/uploads/logos/6b00d1e93244e4eeb7635c19/39f536ed-652a-43cc-b803-b7a5a4a43d5a.jpg</url><title>Seema Sharma</title><link>https://seemasharmaai.hashnode.dev</link></image><generator>RSS for Node</generator><lastBuildDate>Sun, 22 Nov 2026 16:04:03 GMT</lastBuildDate><atom:link href="https://seemasharmaai.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[robots.txt rules for GPTBot, ClaudeBot and PerplexityBot]]></title><description><![CDATA[Which AI crawlers to allow or block, with ready-made snippets]]></description><link>https://seemasharmaai.hashnode.dev/robots-txt-rules-for-gptbot-claudebot-and-perplexitybot</link><guid isPermaLink="true">https://seemasharmaai.hashnode.dev/robots-txt-rules-for-gptbot-claudebot-and-perplexitybot</guid><category><![CDATA[SEO]]></category><category><![CDATA[webdev]]></category><category><![CDATA[Artificial Intelligence]]></category><category><![CDATA[robots.txt]]></category><dc:creator><![CDATA[Seema Sharma]]></dc:creator><pubDate>Sat, 21 Nov 2026 20:48:56 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6b00d1e93244e4eeb7635c19/05a541f5-2a93-4a87-b24c-4ecf37760c76.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>"Should we block AI bots?" is the wrong question, because there isn't one AI bot. Most AI companies now run separate crawlers for separate jobs, and blocking the wrong one can quietly remove you from AI answers while doing nothing about training.</p>
<p>This post sorts the major AI crawlers by what they do, then gives three robots.txt recipes you can copy. Everything below is taken from each operator's own documentation (links at the end).</p>
<h2>Three jobs, three kinds of crawler</h2>
<p>Think about AI crawlers in three groups:</p>
<ol>
<li><strong>Training crawlers</strong> collect pages that may be used to train future models.</li>
<li><strong>Search crawlers</strong> build an index that the AI product searches when it answers questions, and decide whether you can be shown and cited.</li>
<li><strong>User-triggered fetchers</strong> visit a page because a user asked the assistant about it right now.</li>
</ol>
<p>Here's how the big operators map onto that:</p>
<table>
<thead>
<tr>
<th>Operator</th>
<th>Training</th>
<th>Search / index</th>
<th>User-triggered fetch</th>
</tr>
</thead>
<tbody><tr>
<td>OpenAI (ChatGPT)</td>
<td><code>GPTBot</code></td>
<td><code>OAI-SearchBot</code></td>
<td><code>ChatGPT-User</code></td>
</tr>
<tr>
<td>Anthropic (Claude)</td>
<td><code>ClaudeBot</code></td>
<td><code>Claude-SearchBot</code></td>
<td><code>Claude-User</code></td>
</tr>
<tr>
<td>Perplexity</td>
<td>(none listed)</td>
<td><code>PerplexityBot</code></td>
<td><code>Perplexity-User</code></td>
</tr>
<tr>
<td>Google (Gemini)</td>
<td><code>Google-Extended</code> (token)</td>
<td><code>Googlebot</code></td>
<td>(n/a)</td>
</tr>
<tr>
<td>Apple</td>
<td><code>Applebot-Extended</code> (token)</td>
<td><code>Applebot</code></td>
<td>(n/a)</td>
</tr>
</tbody></table>
<p>A few details that trip people up:</p>
<ul>
<li><strong>OpenAI controls GPTBot and OAI-SearchBot independently.</strong> Allowing one doesn't allow the other. OpenAI says sites that opt out of OAI-SearchBot won't be shown in ChatGPT search answers.</li>
<li><strong>User-triggered fetchers may ignore robots.txt.</strong> OpenAI says robots.txt rules "may not apply" to ChatGPT-User because a person initiated the request. Perplexity says Perplexity-User "generally ignores robots.txt rules" for the same reason.</li>
<li><strong>Google-Extended and Applebot-Extended aren't separate crawlers.</strong> They're robots.txt tokens that control how content already crawled by Googlebot or Applebot may be used. Google says Google-Extended doesn't affect inclusion or ranking in Google Search. But it covers <strong>both</strong> Gemini training <strong>and</strong> grounding (feeding Search results to Gemini at answer time). Block it and you may also drop out of Gemini's grounded answers.</li>
<li><strong>Perplexity says PerplexityBot isn't used to crawl content for AI foundation models.</strong> It exists to surface and link sites in Perplexity's results.</li>
</ul>
<h2>One rule that breaks a lot of robots.txt files</h2>
<p>A crawler that matches a named group <strong>ignores the</strong> <code>User-agent: *</code> <strong>group entirely</strong>. So if you have:</p>
<pre><code class="language-text">User-agent: *
Disallow: /admin/

User-agent: GPTBot
Allow: /
</code></pre>
<p>GPTBot is now allowed into <code>/admin/</code>, because it only reads its own group. Repeat your private-path rules inside every named group.</p>
<h2>Recipe 1: allow AI answers, block training</h2>
<p>This is the most common choice for businesses that want to be recommended by AI assistants but would rather not contribute to training.</p>
<pre><code class="language-text"># Training crawlers and tokens: blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
Disallow: /

# AI search crawlers: allowed, private paths still blocked
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: DuckAssistBot
Allow: /
Disallow: /admin/
Disallow: /account/

# Everyone else
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /account/

Sitemap: https://example.com/sitemap.xml
</code></pre>
<p>Notice that <code>Google-Extended</code> is <strong>not</strong> in the blocked list. Because it also governs Gemini grounding, blocking it is a trade-off: less training use, but possibly less Gemini visibility. Add it only if training opt-out matters more to you than Gemini answers.</p>
<p>Listing several <code>User-agent</code> lines above one set of rules is valid. They all share the group.</p>
<h2>Recipe 2: allow everything</h2>
<p>If you want maximum AI visibility and don't mind training use, you don't need AI-specific rules at all. A plain wildcard group covers every crawler that doesn't have its own group:</p>
<pre><code class="language-text">User-agent: *
Allow: /
Disallow: /admin/
Disallow: /account/

Sitemap: https://example.com/sitemap.xml
</code></pre>
<h2>Recipe 3: block all AI crawlers</h2>
<pre><code class="language-text">User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
User-agent: DuckAssistBot
Disallow: /

User-agent: *
Allow: /
Disallow: /admin/
</code></pre>
<p>Be clear about what this costs: you're opting out of being cited by ChatGPT search, Claude, Perplexity and DuckDuckGo's AI answers. And as noted above, user-triggered fetchers may still visit when someone pastes your URL.</p>
<h2>robots.txt is a request, not a lock</h2>
<p>Well-behaved crawlers read and honor robots.txt, but it doesn't enforce anything. Two consequences:</p>
<ul>
<li><strong>Changes take time.</strong> OpenAI says robots.txt updates can take about 24 hours to apply to OAI-SearchBot, Meta says up to 24 hours, and DuckDuckGo says 72 hours for DuckAssistBot.</li>
<li><strong>Your CDN or firewall can override it.</strong> If a bot-management rule returns a 403 to GPTBot, it doesn't matter what robots.txt says. Always test with a real request using the crawler's user agent, not just by reading the file.</li>
</ul>
<h2>Avoiding the named-group trap</h2>
<p>The rules above are easy to get subtly wrong, especially the "named groups ignore <code>*</code>" behavior. Generating the file is safer than hand-editing it: a <a href="https://presencescout.com/tools/robots-txt-generator">robots.txt generator for AI bots</a> that lets you pick crawlers by what they're used for and repeats your private paths in every group removes the guesswork.</p>
<h2>Sources</h2>
<ul>
<li>OpenAI: <a href="https://developers.openai.com/api/docs/bots">Overview of OpenAI crawlers</a></li>
<li>Anthropic: <a href="https://support.claude.com/en/articles/8896518">Does Anthropic crawl data from the web?</a></li>
<li>Perplexity: <a href="https://docs.perplexity.ai/guides/bots">Perplexity crawlers</a></li>
<li>Google: <a href="https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers">Google's common crawlers</a></li>
<li>Apple: <a href="https://support.apple.com/en-us/119829">About Applebot</a></li>
</ul>
]]></content:encoded></item></channel></rss>