Live Search crawler updates from MSN

14th February 2008

MSN has updated its Live Search crawler with a number of new developments to improve efficiency.

Fabrice Canel of the Live Search Crawling Team writes on the Live Search Webmaster Centre Blog that the moves represent "a few steps more in the right direction" for MSN's ability to help webmasters.

Faster transmission time could be possible through HTTP compression, which compresses static files and application responses - this reduces network load between webmaster's servers and MSN's crawler, Canel explains.

Meanwhile, the crawler is able to identify whether pages have been updated since the last time they were scanned, supporting conditional get as defined by RFC 2616.

Including the standard "If-Modified-Since" header and time of the last download in the get request, the crawler will include the "If-None-Match" header when available, using a 304 HTTP response if the content has not changed.

MSN Search launched in 1998 and became Live Search in September 2006.