Crawl test guide
Normal pages
The home page, the about page, this guide, the FAQ, the orphan page, the noindex test page, the canonical test page and the structured-data test page all answer with status 200 and are observed by the crawler.
Planted cases
The address /blocked/ is disallowed in robots.txt, so a correct crawler records it as blocked and never downloads it. The address /old-page answers with a permanent redirect to this guide. The address /gone/ answers with status 404. The noindex test page asks search engines not to index it. The canonical test page names this guide as its canonical version. The structured-data test page contains one valid block and one block that cannot be parsed. The orphan page is in the sitemap, but no page links to it.
Why exact answers matter
A crawler that reports a redirect, a 404 or a robots block as an observed page would give the software false evidence. The test passes only when every case is reported as what it is.