Understand a site's crawl instructions
Test the rule. See why it wins.
Check an exact URL for a named crawler, inspect the matching group and build a straightforward robots.txt file without guesswork.
Live rules test
May this crawler request this URL?
We request the site's root robots.txt file and apply its published rules to the full path you enter.
Browser-based builder
Draft a simple robots.txt file.
Add only the crawl paths you intend to manage. The preview updates in your browser as you type.
This file is public, not a lock. Use sign-in controls to protect private content, and indexing directives when a page should stay out of search results.
Use * for all compliant crawlers, or one crawler product token.
One path per line. Each path must begin with /.
Use a more specific path to open an exception inside a broader disallow rule.
Before you publish
A crawl rule has a narrow job.
Place the file at the site's root, check important URLs for each crawler you care about and keep sensitive paths behind real access controls.
- 01
Publish at /robots.txt
The rules apply only to the matching scheme, hostname and port.
- 02
Test exact URLs
A longer matching path beats a shorter one; Allow wins when matching rules are equally specific.
- 03
Check with the search engine
Platform tools can reveal cached files and crawler-specific extensions this general tester cannot see.
Robots.txt questions
Use crawl controls for crawl decisions.
Can robots.txt keep a page out of search results?
Not reliably. A blocked URL can still be indexed from links or other signals. Use a robots meta tag or X-Robots-Tag on a crawlable URL when indexing is the issue.
Can robots.txt protect private content?
No. The file is public and the protocol is voluntary. Put sensitive material behind authentication and proper access controls.
What does Disallow: / mean?
It matches every path for that user-agent group and asks the matching crawler not to access the site.
Why might a search engine report a different result?
Search engines may cache the file, support additional syntax or observe a different response. Confirm important decisions in that platform's official tools.