Overview
Generate a robots.txt file for your website. Customize crawler access rules, sitemaps, and crawl delay directives with a live preview.
How to Use
Enter the required information below. Results update in real-time with details about your configuration.
Fill in the fields above and click Generate
Example Use Cases
- Block search crawlers from indexing staging or admin areas
- Direct crawlers to your XML sitemap for better indexing
- Set crawl delay for high-traffic sites to prevent server overload
- Allow or disallow specific user-agents (Googlebot, Bingbot) for targeted SEO
Common Issues
Case-sensitive paths: Robots.txt directives are case-sensitive. Ensure your Allow/Disallow paths match the actual URL casing on your server.
Separate image crawlers: Googlebot-Image and Googlebot-Video are distinct user-agents. Add separate rules if you want to control image or video crawling independently.