<table cellspacing="0" cellpadding="0" border="0" ><tr><td valign="top" style="font: inherit;">Actually, I was wrong in my previous email.<br><br>"Disallow: /aggregator/" should block any directory at any level that is "aggregator".<br><br>The syntax, "Disallow: /aggregator" should block access to any aggregator directory and any aggregator.html files.<br><br>Without the slash you block directories and html pages by the name. With the final slash you block just directories.<br>There's no reason to add more than one robots.txt file. You should only have one, and put all your rules in there. <br><br>/aggregator/ should block :<br>/aggragator/,<br>/foo/aggregator/,<br>/long/deep/path/to/obscure/folder/aggregator/,<br>etc.<br><br>You can test all of this for yourself. you can use wget to download from your site as a robot. You can also make wget ignore the robots.txt file. You can have wget pretend to be any robot you like, or even make it your own
robot, that you allow to mirror your page.<br><br>Caveat, if you make wget ignore the robots.txt file you should also add a pause to it so you don't hammer the site you are downloading/mirroring. Some sites specifically disallow wget in recursive mode, to keep the site from getting hammered by downloads.<br><br>Jack<br><br>--- On <b>Sun, 1/16/11, Jonathan Hutchins <i><hutchins@tarcanfel.org></i></b> wrote:<br><blockquote style="border-left: 2px solid rgb(16, 16, 255); margin-left: 5px; padding-left: 5px;"><br>From: Jonathan Hutchins <hutchins@tarcanfel.org><br>Subject: robots.txt question<br>To: "KCLUG (E-mail)" <kclug@kclug.org><br>Date: Sunday, January 16, 2011, 12:53 PM<br><br><div class="plainMail">I'm wondering about the syntax. The example file from drupal uses the format<br><br>Disallow: /aggregator<br><br>However, it says in the comments that only the root /robots.txt file is valid. <br><br>From my understanding of
the syntax, /aggregator does not <br>block /foo/aggregator, so I need to either prepend "/foo" to everything, or <br>use wildcards per the new google/webcrawler extensions to the protocol.<br><br>If anybody can cite an on-line example that explains I'd be grateful.<br>_______________________________________________<br>KCLUG mailing list<br><a ymailto="mailto:KCLUG@kclug.org" href="/mc/compose?to=KCLUG@kclug.org">KCLUG@kclug.org</a><br><a href="http://kclug.org/mailman/listinfo/kclug" target="_blank">http://kclug.org/mailman/listinfo/kclug</a><br></div></blockquote></td></tr></table>