Usability Views Article Details
 home | timeline | recent | popular | e-reports | userati | books | about 

Observed Web Robot Behavior on Decaying Web Subsites (17 Feb 2006)
We describe the observed crawling patterns of various search engines (including Google, Yahoo and MSN) as they traverse a series of web subsites whose contents decay at predetermined rates. We plot the progress of the crawlers through the subsites, and their behaviors regarding the various file types included in the web subsites. We chose decaying subsites because we were originally interested in tracking the implication of using search engine caches for digital preservation. However, some of the crawling behaviors themselves proved to be interesting and have implications on using a search engine as an interface to a digital library.
Add this article to Del.icio.us

Article URL: http://www.dlib.org/dlib/february06/smith/02smith.html

Read 108 more articles from D-Lib Magazine sorted by date, popularity, or title.
Next Article: Subvocal Speech and the Development of Biological Interfaces
  RSS 0.91  Subscribe with Bloglines  Add to My Yahoo!  Preview in Google Reader
Books about Usability | Information Architecture | Information Visualisation | Technology History

Some of the people who make up the Userati group
This site is a labour of love built by Chris McEvoy




Popular Items
> Top Knitting Sites
> Ajax Sucks
> What if Jakob Nielsen had a blog?
> Kate Bush and Pi
> Simply Google