BTW for anybody interested in the legal relevance of a /robots.txt file (basically zilch) please see http://www.robotstxt.org/faq/legal.html