robots.txt question
Jonathan Hutchins
hutchins at tarcanfel.org
Mon Jan 17 10:21:39 CST 2011
Thanks for at least trying. Yes, I'm aware of robotstxt.org. My question was
not regarding user-agents, but about whether a string such as /foo blocks
only http://website.org/foo, or if it would block
http://website.org/blah/foo. I think it would not, but I'm looking for
documentation, references, or examples that specifically address that.
>From reading Google's reference on robots.txt as extended by Google,
Microsoft, and Yahoo, the most obvious interpretation would be that I need to
block /*foo, but again, no specific confirmation.
As for the trailing slash, again, no specific reference to that format so far.
More information about the KCLUG
mailing list