How this site uses robots.txt

Summary

The robots.txt file sits at the root of the site and is the first file a compliant crawler reads. On this site it allows everything except one planted address, and it names the location of the XML sitemap. A crawler that cannot read robots.txt at all must treat permission as unknown rather than assume everything is allowed, while a robots.txt that simply does not exist means everything is allowed.

Detail

Robots rules only control crawling. They do not remove a page from search results by themselves, which is why the noindex test page uses a robots meta tag instead.