<table cellspacing="0" cellpadding="0" border="0" ><tr><td valign="top" style="font: inherit;">Let me explain a bit.<br><br><p>To exclude all robots, that respect the robots.txt file:</p><blockquote>
User-agent: *<br>
Disallow: /
</blockquote><p>To exclude just one directory and its subdirectories,
say, the /aggregator/ directory:</p><blockquote>
User-agent: *<br>
Disallow: /aggregator/
</blockquote><p>To disallow specific
robots you need to know what it calls itself, ia_archiver is the wayback machine<br></p><p><br></p><p>To allow the Internet Archive bot you'd make a line like this:</p><blockquote>
User-agent: ia_archiver<br>
Disallow:
</blockquote><p><a name="prevent"></a>To block<span style="font-weight: bold;"> </span>ia_archiver from
visiting:</p><blockquote>
User-agent: ia_archiver<br>
Disallow: /
</blockquote><br>You can have as many lines like this as you want. So you can disallow all robots from everywhere, and then allow only those you want. You can block certain robots from certain parts. You can block directories and sub directories or individual files.. If you have numerous "aggregator" files in various subdirectories you want to block you need to list them all.<br><br>Like this:<br><br>User-agent: *<br>Disallow:/aggregator/<br>Disallow:/foo/aggretator/<br>...<br>
Disallow:/hidden/aggregator/<br><br>Your syntax looks wonky, missing the final "/". <br>User-agent tells who to block and Disallow what to block. This all assumes well behaved robots. This file is useless for those that ignore this file. It is not a security device, just a polite sticky note. <br><br>You might go here for more detailed info. I'm no expert for sure.<br><br>http://www.robotstxt.org/orig.html<br><br>Jack<br><br>--- On <b>Sun, 1/16/11, Jonathan Hutchins <i><hutchins@tarcanfel.org></i></b> wrote:<br><blockquote style="border-left: 2px solid rgb(16, 16, 255); margin-left: 5px; padding-left: 5px;"><br>From: Jonathan Hutchins <hutchins@tarcanfel.org><br>Subject: robots.txt question<br>To: "KCLUG (E-mail)" <kclug@kclug.org><br>Date: Sunday, January 16, 2011, 12:53 PM<br><br><div class="plainMail">I'm wondering about the syntax. The example file from drupal uses the format<br><br>Disallow: /aggregator<br><br>However, it
says in the comments that only the root /robots.txt file is valid. <br><br>From my understanding of the syntax, /aggregator does not <br>block /foo/aggregator, so I need to either prepend "/foo" to everything, or <br>use wildcards per the new google/webcrawler extensions to the protocol.<br><br>If anybody can cite an on-line example that explains I'd be grateful.<br><br></div></blockquote></td></tr></table>