artenis.alija
ende
Free tools

Free Robots.txt Generator (with AI Crawlers)

Choose what search engines and AI crawlers may read, block private pages, and download a ready robots.txt for your site root.

Loading the tool…

What you get

Private paths
Block admin, cart, checkout and other pages that should never appear in search.
AI crawler control
Block training crawlers without disappearing from ChatGPT, Perplexity or Claude search.
Sitemap link
Point every crawler to your XML sitemap.

Questions

Should I block AI crawlers?

Block training crawlers (GPTBot, CCBot, Google-Extended) if you don’t want your content used to train models. Don’t block the search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) if you want AI answers to recommend you.

Does robots.txt hide pages?

No. It asks crawlers not to fetch them, but a blocked page can still be indexed if linked elsewhere. Use noindex to keep a page out of results.

Where does the file go?

At the root of your domain: yourdomain.com/robots.txt.

Free monthly newsletter

One email a month with an AI automation you can use

Sent at the start of each month, never more often. Real workflows I build for clients, new free tools first, and the n8n templates behind them. Sign up now and get my 6 n8n workflow templates straight away.

Monthly newsletter: one email a month, never more, with free tools and occasional offers of my services. I may also send the odd update on WhatsApp; reply STOP to opt out. Unsubscribe with one click.

Free tool, free extras

Enjoying the tool? Get more like it, once a month

Leave your name, email and WhatsApp number to keep using it without these reminders. You get one email at the start of each month with new free tools and automations, plus my 6 n8n workflow templates right now.

Monthly newsletter: one email a month, never more, with free tools and occasional offers of my services. I may also send the odd update on WhatsApp; reply STOP to opt out. Unsubscribe with one click.