Mean time to index for blog posts

December 3rd, 2006

stop watchInspired by Matt Hurst’s post and following up on our own note on the mean time to index (MTTI) for blog search engines, we’ve done a simple experiment to measure the speed at which a blog post makes it into various search indexes. We made a test post at 13:20 (GM+5) on Saturday 2 December 2006. We used queries to monitored Google, Google Blog Search and Technorati to try to determines how long it took added to their indexes. Here are the results:

    Lag in H:M:S Where
    00:01:40 Google blog search
    00:08:30 Technorati
    02:02:00 Google blog search alert
    21:00:00 Google search

The post made it into both Google blog search and Technorati very quickly. There was about a two hour delay before the Google alert we had set up was triggered. The post appeared in Google’s main index in under a day. As I had previously mentioned, I have seen many cases in which our blog posts show up in the Google index in about twelve hours.

Its just a single data point, of course, and these numbers will be sensitive to the day of the week and the time of day. We also did this manually, so the times are approximate. We don’t know how Google’s main index is populated. Our suspicion is that one source of new URLs is Google’s blob search system. But it’s speculation. I don’t think that this experiment provides strong evidence for it, but its certainly very consistent with the idea.

Implementing a program to automatically monitor and publish the MTTI of major search engines might be a nice simple project for someone who has an urge to put up a useful Web service.