How the robots.txt generator works
robots.txt uses a simple directive format read by well-behaved crawlers: User-agent specifies which crawler a rule block applies to (* means all), Disallow blocks a path from being crawled, and Sitemap points crawlers to your XML sitemap.
Step-by-step guide
- Choose whether to allow all crawlers everywhere, or specify custom disallowed paths.
- If using custom rules, list one path per line (e.g. /admin/).
- Add your sitemap URL.
- Copy the result and save it as
robots.txtin your site's root directory.
Common mistakes to avoid
- Accidentally disallowing
/(blocks your entire site from crawling) - Forgetting to update robots.txt after a site restructure, leaving stale rules
- Using robots.txt as the only method to hide sensitive content — it isn't a security measure
Frequently asked questions
Not reliably — Disallow tells well-behaved crawlers not to crawl a path, but a disallowed URL can still be indexed (without its content) if other pages link to it. To fully prevent indexing, use a noindex meta tag on the page itself instead.
It must be placed at the root of your domain (e.g. example.com/robots.txt) — crawlers won't look for it in a subfolder.
Most major, reputable search engine crawlers do, but robots.txt is a voluntary convention, not enforced access control — it won't stop malicious bots that ignore it.