Glossary · Search and local visibility
robots.txt
A top-level text file that publishes crawl rules for automated clients following the Robots Exclusion Protocol.
Definition
The Robots Exclusion Protocol defines how a service publishes groups and allow/disallow rules in `/robots.txt`. Cooperative crawlers can use those rules when deciding which paths to request. The file is public and should not list secret paths that would otherwise be unknown.
Why it matters
It helps coordinate crawl behavior and prevent waste, but it must match the site's intended indexing and technical structure.
Common misunderstanding
`robots.txt` is not authentication and does not prevent a person or non-compliant program from requesting a public URL.
Related terms
Related system family: Measurement & Search Signals
Sources
- RFC 9309: Robots Exclusion Protocol (opens in a new tab) · RFC Editor / IETF
Last reviewed July 16, 2026