Glossary · Search and local visibility
Robots directive
An instruction telling compliant crawlers whether content may be indexed or links followed.
Also called: robots directives, meta robots directive
Definition
Robots directives are commonly delivered in a page's robots meta tag or an X-Robots-Tag HTTP header. Directives such as noindex affect indexing behavior, while robots.txt rules primarily govern crawler access.
Why it matters
One unintended directive can remove otherwise healthy pages from search eligibility across an entire release.
Common misunderstanding
robots.txt and a robots meta directive solve different problems; blocking a page in robots.txt may prevent a crawler from seeing a noindex instruction on that page.
Related terms
Related system family: Measurement & Search Signals
Sources
- Robots meta tag and X-Robots-Tag specifications (opens in a new tab) · Google Search Central
- RFC 9309: Robots Exclusion Protocol (opens in a new tab) · RFC Editor / IETF
Last reviewed July 16, 2026