# /robots.txt file # 2007/07/23 16:54 User-agent: msnbot Crawl-delay: 30 User-agent: Slurp Crawl-delay: 30 User-agent: Yahoo-NewsCrawler Crawl-delay: 30 User-agent: * Crawl-delay: 30 Allow: /