[ILUG] MSN Search Beta

[Paul Jakma]

On Thu, 11 Nov 2004, John McCormac wrote:

> here.) The msnbot is so badly written that it does not use 304s and puts
> excessive loads on webservers. So many webmasters have complained about it
> that Microsoft even introduced its own robots.txt entry so that webmasters
> can use a delay between pages being fetched by its scrapers.

Ah, where would that be? I followed the URL in the client id it uses 
to http://search.msn.com/webmasters/msnbot.aspx, but there's nothing 
there on how to limit it except the standard robots.txt and robot 
meta tags.

I've sent a mail to their mail address asking them to rate limit, but 
received no reply - at this stage I'm considering barring the MSNBot 
altogether. It's responsible for 40% of hits to a site I maintain..

regards,
-- 
Paul Jakma	paul at clubi.ie	paul at jakma.org	Key ID: 64A2FF6A
Fortune:
A citizen of America will cross the ocean to fight for democracy, but
won't cross the street to vote in a national election.
 		-- Bill Vaughan