Glossary · Search and local visibility
Crawler
An automated client that requests web resources and may follow links according to a defined purpose and policy.
Also called: spider, bot
Definition
A crawler is software acting as a web client. It may discover pages for a search index, audit a website, monitor changes, or collect an approved research sample. Responsible crawlers identify their purpose where appropriate, respect request limits, handle errors, and avoid restricted or unnecessary data.
Why it matters
Crawler behavior affects server load, data quality, research ethics, and what a crawl can legitimately conclude.
Common misunderstanding
All bots are not identical. Search crawlers, monitoring agents, accessibility tools, malicious scanners, and private audit crawlers have different purposes and behavior.
Related terms
Related system family: Measurement & Search Signals
Sources
- RFC 9309: Robots Exclusion Protocol (opens in a new tab) · RFC Editor / IETF
- How the web works (opens in a new tab) · MDN Web Docs
Last reviewed July 16, 2026