What is a robots.txt file?

A robots.txt file is a plain-text file placed at the root of your website (e.g. https://example.com/robots.txt) that uses the Robots Exclusion Protocol to instruct web crawlers which pages or directories they should or should not access.

Does robots.txt actually block crawlers?

Well-behaved crawlers like Googlebot and Bingbot will respect robots.txt rules, but it is not a security measure. Malicious bots can and often do ignore it. For truly sensitive content, use proper server-side authentication instead.

What is the difference between Disallow and Allow?

Disallow tells a crawler not to access a path or directory. Allow explicitly permits access to a path even if a parent directory is disallowed. Allow rules take precedence over Disallow rules when both match the same URL, making it useful for granting exceptions inside blocked folders.

What does Crawl-delay do?

Crawl-delay tells a crawler to wait a specified number of seconds between successive requests to your server. This is useful for limiting the load caused by aggressive crawlers. Note that Googlebot does not honor Crawl-delay; use Google Search Console to configure Googlebot's crawl rate instead.

Where should I place the robots.txt file?

The file must be at the root of your domain — for example, https://www.example.com/robots.txt. It cannot be placed in a subdirectory. Each subdomain requires its own separate robots.txt file.

Can I have multiple user-agent blocks?

Yes. You can add separate rule blocks for different crawlers in the same robots.txt file. Each block starts with a User-agent line followed by its Disallow and Allow directives. Rules apply only to the crawler named in each block.

Robots.txt Generator

Name: Robots.txt Generator
Availability: InStock
Author: Nham Vu

Build a robots.txt file instantly by configuring user-agent rules, allow and disallow paths, crawl-delay, and your sitemap URL.

Configure Rules

Sitemap URL (optional)

User-Agent Rules

Generated robots.txt

Quick Presets

Summary

Build a robots.txt file instantly by configuring user-agent rules, allow and disallow paths, crawl-delay, and your sitemap URL.

How it works

Choose the user-agent (all bots or a specific crawler such as Googlebot).
Add disallow paths for directories or pages you want to block from crawlers.
Add allow paths to explicitly permit access within a blocked directory.
Optionally set a crawl-delay to limit how fast crawlers access your site.
Enter your sitemap URL so search engines can find it automatically.
Copy or download the generated robots.txt and place it at your domain root.

Use cases

Block search engines from indexing staging or admin directories.
Allow Googlebot to access specific pages inside an otherwise blocked folder.
Limit aggressive bots with a crawl-delay to protect server resources.
Point crawlers to your XML sitemap for faster and more complete indexing.
Prevent duplicate content issues by disallowing filtered or faceted URLs.
Create separate rule sets for different crawlers in a single file.
Protect private or login-only areas from being listed in search results.
Quickly audit and regenerate a clean robots.txt when moving to a new domain.

Frequently Asked Questions

Last updated: 2026-06-10 · Reviewed by Nham Vu