Free Robots.txt Generator (with AI Crawlers)
Choose what search engines and AI crawlers may read, block private pages, and download a ready robots.txt for your site root.
Loading the tool…
What you get
- Private paths
- Block admin, cart, checkout and other pages that should never appear in search.
- AI crawler control
- Block training crawlers without disappearing from ChatGPT, Perplexity or Claude search.
- Sitemap link
- Point every crawler to your XML sitemap.
Questions
Should I block AI crawlers?
Block training crawlers (GPTBot, CCBot, Google-Extended) if you don’t want your content used to train models. Don’t block the search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) if you want AI answers to recommend you.
Does robots.txt hide pages?
No. It asks crawlers not to fetch them, but a blocked page can still be indexed if linked elsewhere. Use noindex to keep a page out of results.
Where does the file go?
At the root of your domain: yourdomain.com/robots.txt.