Robots.txt generator
Choose what to allow, block the paths you do not want crawled, decide how to treat AI training crawlers, and copy a valid robots.txt file.
Fill in the fields and press Calculate to see your result.
A badly written robots.txt can remove an entire site from search results with one line. This generator produces safe, valid output and explains what each directive does.
Crawling is not indexing
Robots.txt asks crawlers not to fetch a page. It does not remove a page from search results — a blocked page can still be indexed from external links, with no description. To keep something out of results, allow crawling and use a noindex meta tag.
Should you block AI crawlers?
It is a genuine trade-off. Blocking training crawlers protects your content from being used as training data. It also reduces the chance of your pages being cited in AI-generated answers, which is a growing referral source. The middle option blocks training while allowing search-focused AI crawlers.
Frequently asked questions
In the root of the domain, so it is reachable at yourdomain.com/robots.txt. It will not work anywhere else.
No. It is a public file that anyone can read, and listing a path there advertises its existence. Use authentication for anything genuinely private.
Want the result turned into a plan?
Send us your numbers and we will come back with what we would actually do, in writing.
Or message us on WhatsApp — replies within business hours.