I'm wondering about the syntax. The example file from drupal uses the format
Disallow: /aggregator
However, it says in the comments that only the root /robots.txt file is valid.
>From my understanding of the syntax, /aggregator does not
block /foo/aggregator, so I need to either prepend "/foo" to everything, or
use wildcards per the new google/webcrawler extensions to the protocol.
If anybody can cite an on-line example that explains I'd be grateful.