16 Jan
2011
16 Jan
'11
8:53 p.m.
I'm wondering about the syntax. The example file from drupal uses the format
Disallow: /aggregator
However, it says in the comments that only the root /robots.txt file is valid.
From my understanding of the syntax, /aggregator does not
block /foo/aggregator, so I need to either prepend "/foo" to everything, or use wildcards per the new google/webcrawler extensions to the protocol.
If anybody can cite an on-line example that explains I'd be grateful.