Robots.txt Builder

Summary

Choose crawlers and rules to build a complete robots.txt file without hand-writing directives.

How it works

1. Add user-agent rules for specific bots or all (*)
2. Set Allow and Disallow paths
3. Add your sitemap URL
4. Copy the final robots.txt content

Use cases

• Blocking crawlers from admin or staging paths
• Allowing only specific search engine bots
• Adding sitemap reference to robots.txt

Frequently Asked Questions