FreeYour browser + GSCAbout 20 min
Check robots.txt and Let AI Search In
robots.txt is a small file at the root of your site. It tells search engines and AI crawlers what they may read. One wrong line can hide your whole site.
Why this matters
If Google can't crawl a page, it can't rank it. If ChatGPT's or Perplexity's search crawler is blocked, those tools can't cite you in their answers. Many sites block them by accident.
✓ How to do it
- Open yourdomain.com/robots.txt in your browser.
- Look for
Disallow: /underUser-agent: *. That line blocks everything. Remove it unless the site is a test copy. - Make sure the file lists your sitemap, for example
Sitemap: https://yourdomain.com/sitemap.xml. - Check that search crawlers aren't blocked: Googlebot, Bingbot, OAI-SearchBot (ChatGPT search), PerplexityBot, and Claude-SearchBot.
- If your site runs through Cloudflare, open AI Crawl Control (under Security or Bots). Cloudflare started blocking AI crawlers by default on new sites in July 2025, so check that the search crawlers are allowed.
- In GSC, open Settings → robots.txt to see the version Google last fetched and any errors.
Done whenrobots.txt blocks none of the search crawlers, lists the sitemap, and GSC's robots.txt report shows no errors.
Tips & pitfalls
- Blocking training crawlers like GPTBot or Google-Extended is a business choice. It does not remove you from Google Search or ChatGPT search results.
- An llms.txt file is optional. No major search engine has said it uses one, so don't spend long on it.