Hyperactive spiders: Google's no longer number one

Submitted by Walt on Fri, 02/04/2005 - 18:04

I've always been astonished at the sheer number of Googlebot hits at Cites & Insights--averaging 30 a day for a site that has new content around once every four weeks.

(I eventually realized that Googlebot may be crawling the entire site each time, so that it's really more like 60 hits done once every couple of days...still quite a lot.)

In previous statistics, Googlebot was always way out ahead of any other spider.

That's no longer true. Beginning last December (I think), and continuing strong since then, there's a new champion for hyperactivity: Inktomi Slurp.

Actually, for the month of January 2005, Googlebot's third. Here's what I see:

  • Inktomi Slurp: 1420 hits
  • MSN Robot: 479 hits--yes, MSN is getting serious about search
  • Googlebot: 446 hits

After that, it drops rapidly: Gigablast Robot with 85, Turnitin Robot (!?!) with 75, FAST Enterprise Crawler with 68...and 40-odd others, down to SKIZZLE! Distributed Internet Spider and seven others with one each.

In all, spiders seem to account for just over 10% of the hits--but, fortunately, only about 1.5% of the unique visitors.