Est. 2019
Est. 2019
Last Updated:
August 4, 2026
Parth Gaurav
Founder & CEO
Quick answer: Before an AI engine can cite your page, its crawler has to reach it. Three things get in the way — robots.txt rules, bot filtering at the CDN in front of your site, and the fact that training crawlers and live retrieval crawlers are separate user agents. Block the retrieval ones and you disappear from AI answers entirely.
By Parth Gaurav, Founder & CEO, Digi Hotshot. Last updated: July 30, 2026.
Most AEO advice is about what you write and how you mark it up. Schema, entity clarity, answer blocks that stand on their own. That work matters. But we keep meeting marketing teams who did all of it properly and still can't find themselves in a single AI answer, and the reason has nothing to do with content. The crawler never got to read the page.
Access is the layer underneath everything else. It's boring, it's invisible from inside the CMS, and on most B2B teams nobody owns it. So it breaks quietly and stays broken.
Googlebot. A file written years ago, or copied from a template, can exclude them without anyone noticing.On Webflow, robots.txt is a site-level file you edit under Site settings → SEO → Indexing, per Webflow's own documentation on setting robots.txt rules. It lives at the top level of your domain, so anyone can read yours right now at yourdomain.com/robots.txt. Webflow adds your sitemap link to it by default.
The starting position is reasonable. Webflow's docs state plainly that traffic from search engine crawlers and AI bots is allowed by default. So a site nobody has touched is usually fine. The problem is that somebody almost always touched it.
Here's the part worth knowing. Webflow gives you Traffic control toggles in that same Indexing panel that auto-generate robots.txt rules for whole groups of bots. There are two groups. The "search engine crawlers" group includes Googlebot, Bingbot and OpenAI's OAI-SearchBot. The "AI bots" group includes GPTBot and CCBot — and it also includes ChatGPT-User, Claude-User and PerplexityBot.
Read that list again, because it's the whole story. A legal or comms conversation ends with "we don't want models trained on our content," someone flips the AI bots toggle off, and the site stops being reachable by the agents that fetch pages to answer questions. Nobody intended that. It just came along in the same switch.
One more Webflow quirk: once a robots.txt has been created, Webflow's docs say it can't be completely removed. You replace the rules, you don't delete the file.
And if you migrated recently — we've done 30+ migrations, 14 off WordPress — robots.txt doesn't travel with your content. It's platform-level, so it gets rebuilt from scratch on the new host. Good moment to get it right, easy moment to paste in whatever the old site had.
Robots.txt is a request, not a wall. A bot-management rule at the CDN is a wall. If your custom domain is proxied through a CDN or a WAF that your security team set up, that layer answers the request before your site ever hears about it, and a perfectly permissive robots.txt makes no difference.
Google names this directly in its own guidance on AI features and your website: the fundamentals include "ensuring that crawling is allowed in robots.txt, and by any CDN or hosting infrastructure." That's Google telling you to go check a layer most marketing teams don't know they have.
The ground is also moving. Cloudflare changed its default posture on AI crawlers on July 1, 2025, and in a July 1, 2026 changelog split AI traffic into three categories — Search, Agent and Training — with new defaults arriving September 15, 2026 for newly onboarded domains. Read those pages rather than take my summary as gospel. This shifts every few months.
Detecting it beats guessing. Request one of your pages with a crawler's user-agent string and look at the response code. A 200 means you're reachable. A 403, a 429 or an HTML challenge page means something in front of the site is deciding on your behalf. Then read the CDN's bot analytics for the same window.
This is the distinction that decides whether you show up in AI answers, and most teams have never been told it exists.
Some crawlers collect content that may be used to train models. Others fetch a page live, at the moment someone asks a question, so the engine can read it and cite it. Several operators now run separate agents for each job — which means you can say no to one and yes to the other. Most teams don't, and end up blocking both while intending neither.
Confirm the current token in the operator's own docs before you write a rule — these change. Primary sources: OpenAI's bots documentation, Anthropic's crawler article, Perplexity's bots guide, and Google's common crawlers list. One nuance that's easy to miss: because a person triggered the fetch, Perplexity documents that Perplexity-User "generally ignores robots.txt rules." Not every agent treats the file as binding.
I'll take a position, because the hedge helps nobody.
If your growth depends on being found and cited, allowing the retrieval and citation crawlers is close to a default yes. Those agents are the mechanism by which a buyer asking an AI assistant about your category ends up looking at your page. Blocking them is the same as asking Google to de-index you, except quieter, and with no dashboard that tells you it happened.
Training is a genuinely different question with a real trade-off, and reasonable companies land on both sides of it. Have that conversation with legal, write the decision down, and apply it to the training tokens specifically. What shouldn't happen is it getting resolved by accident, in a file nobody has opened since launch, in a way that also takes out citation.
And this is the floor, not the ceiling. Princeton's GEO research (KDD 2024) found citing sources lifted visibility around 40% and adding statistics around 37%. Real effects — worth exactly zero if the fetch returns a 403. If you're still working out the content side, start with what AEO actually is and how Webflow handles it.
yourdomain.com/robots.txt in a browser. Read the whole thing. Look for any Disallow line under a user agent from the table above, and for a blanket User-agent: * / Disallow: /..webflow.io subdomain telling crawlers to ignore it, so a staging domain with indexing switched off will typically serve a blanket User-agent: * and Disallow: /. Fetch your own staging URL and you'll see it. That file is meant to look like that — it's the production one that matters.200 is your answer.Webflow sites can also serve a Content-Signal HTTP header, configured in the same Site settings → SEO → Indexing panel. Where robots.txt controls whether a bot can reach the page, Content-Signal expresses what your content may be used for — training, search indexing, and AI-generated answers, each set separately. It looks like Content-Signal: ai-train=no, search=yes, ai-input=no.
Treat it as a signal of intent rather than a control that enforces anything. Webflow's own docs note it's based on a proposed extension to RFC 9309 and isn't an accepted standard yet. Useful to set. Not a substitute for getting the robots.txt right.
Once access is sorted, the next question is whether any of it is working, which is a measurement problem — we walk through that in tracking AI citations on Webflow. And if you want the structural side, entity clarity and llms.txt is the companion piece to this one.
No. Webflow's documentation states that traffic from search engine crawlers and AI bots is allowed by default, and that Webflow adds a sitemap link to robots.txt automatically. But defaults only describe a site nobody has changed. Check your own robots.txt and the Traffic control toggles under Site settings → SEO → Indexing rather than assuming.
A training crawler collects content that may be used to train a model — GPTBot, ClaudeBot and CCBot are examples. A retrieval crawler fetches a page live, when someone asks a question, so the engine can read and cite it. ChatGPT-User, Claude-User, Perplexity-User, OAI-SearchBot and Claude-SearchBot sit in that second group. Blocking the second group removes you from AI answers.
Not on its own. OpenAI documents GPTBot as the training crawler, while OAI-SearchBot handles search indexing and ChatGPT-User fetches pages for user actions inside ChatGPT. They're separate tokens with separate rules. The risk is blocking all of them at once through a category toggle rather than a specific line.
Because it is a different file. Webflow's Staging indexing setting publishes a unique robots.txt on the .webflow.io subdomain that tells crawlers to ignore that domain, which is the correct behaviour — you don't want staging competing with production. Always audit the production domain.
Yes. If a CDN or firewall proxies your custom domain, its bot-management rules apply before the request reaches Webflow. Google's own guidance on AI features tells site owners to confirm crawling is allowed in robots.txt and by any CDN or hosting infrastructure. Fetch a page with a crawler user-agent and check the status code — a 403 or a challenge page means something upstream is deciding for you.
It takes about ten minutes and it's the cheapest thing on your AEO list. We're a Webflow Premium Partner and we've built 50+ B2B sites on the platform since 2019, and in that time the robots.txt has never once been the file anyone brings to a redesign kickoff. It's usually the last thing checked and occasionally the only thing wrong.
If you'd rather have someone else look, our free website audit covers crawler access alongside the rest — structure, schema, performance. No pitch attached. Scope and pricing depend on what we find, and plenty of the time what we find is a two-line fix you can make yourself.
Last Updated:
August 4, 2026
Book a 30-minute discovery call. We'll discuss your current challenges and show you exactly how we can help.
Your competitors aren't stuck in developer queues. They're launching campaigns, testing messages, and capturing market share while you're waiting for simple updates.
Eliminate the bottlenecks. Give your marketing team the infrastructure they deserve—fast, autonomous, built to scale.
