Glossary · Search and local visibility
Web crawling
Automated retrieval and traversal of web resources to inspect, index, monitor, or analyze publicly reachable content.
Definition
A crawler requests URLs, reads responses, follows permitted links, and records selected information. Search engines crawl to discover content; technical tools crawl to find broken links, redirects, metadata, index directives, or patterns across a defined sample.
Why it matters
A crawl can reveal observable structure at scale, but the findings still require interpretation, scope, dates, respectful request rates, and compliance with applicable rules and policies.
Common misunderstanding
Crawling is not authorization to bypass authentication, access controls, provider terms, privacy expectations, or technical safeguards. `robots.txt` is a crawl preference standard, not an access-control system.
Related terms
Related system family: Measurement & Search Signals
Sources
- RFC 9309: Robots Exclusion Protocol (opens in a new tab) · RFC Editor / IETF
- urllib — URL handling modules (opens in a new tab) · Python Software Foundation
Last reviewed July 16, 2026