# BETTER OFF WITHOUT YOU # PicScout, a relentless image scraper User-agent: psbot Disallow: / # NetSeer scraping to serve contextual ads User-agent: NetSeer/Nutch-0.9 Disallow: / # DontBuyLists trying to monetize a crawl User-agent: DBLBot/1.0 Disallow: / # Internet Archive bot for the wayback machine User-agent: ia_archiver Disallow: / # Sad really User-agent: Atomic_Email_Hunter/4.0 Disallow: / # Snoopy, a php class used mostly for form spam User-agent: Snoopy Disallow: / # ShopWikiBot is both foolish and irresponsible User-agent: ShopWiki Disallow: Crawl-Delay: 5 User-agent: * Disallow: /compare.asp* Disallow: /photos.asp* Disallow: /default.asp?* Allow: /default.asp