robots.txt question

Jonathan Hutchins hutchins at tarcanfel.org
Sun Jan 16 14:53:04 CST 2011


I'm wondering about the syntax.  The example file from drupal uses the format

Disallow: /aggregator

However, it says in the comments that only the root /robots.txt file is valid.  

>From my understanding of the syntax, /aggregator does not 
block /foo/aggregator, so I need to either prepend "/foo" to everything, or 
use wildcards per the new google/webcrawler extensions to the protocol.

If anybody can cite an on-line example that explains I'd be grateful.


More information about the KCLUG mailing list