Robots.txt Generator
Create a correct robots.txt file in seconds — control which crawlers can access which parts of your site, add a crawl-delay and point search engines to your sitemap.
Upload the file to the root of your domain (https://yoursite.com/robots.txt). Remember: robots.txt guides well-behaved crawlers — it is not a security lock. Never list secret folder names in it.
How to Use Robots.txt Generator
- Choose which crawler the rules apply to (or all).
- Pick Allow all, Block all, or enter custom paths to block.
- Optionally add a crawl-delay and your sitemap URL.
- Copy or download the file and upload it to your site's root folder.
Features & Limitations
- User-agent presets or a custom agent, with allow-all, disallow-all or custom path lists
- Optional crawl-delay and sitemap line, generated live as you type
- Copy to clipboard or download the finished robots.txt
- Limit: crawl-delay is ignored by Google (use Search Console instead); the file must be placed at your domain root to work
Frequently Asked Questions
Where does robots.txt go?
In the root of your domain — it must be reachable at yoursite.com/robots.txt. Subfolders do not work.
Does “Disallow: /” remove my site from Google?
It stops crawling, but URLs can still appear in results without descriptions if other sites link to them. For full removal use the noindex tag or Google Search Console removals.
What is Crawl-delay?
It tells polite crawlers how many seconds to wait between requests, reducing server load. Google ignores it (use Search Console crawl settings instead); Bing and Yandex respect it.