Google slow to index blog posts?
By Tim Finin on Sunday, February 24th, 2008 at 9:49 am.Last week I noticed that some of our blog posts took a long time to show up in the Google Blog search index. During the past year, Google has been very fast at indexing blog posts, typically taking less than five minutes from the time is made to when it shows up in their blog search index. But this week it seemed that our posts, or at least some of them, took more than twelve hours to be indexed.
Yesterday I tried to watch a post I made on the IT job market which I wrote just before 11:00am (GMT-5). It showed up in Google Feed Reader quickly enough but had not yet appeared in Google Blog Search when I finally went to bed 14 hours later. When I checked at 9:00am today, it was there, so it took sometime between 14 and 22 hours.
It’s not the case that all posts are being delayed — do a Google Blog search for a popular term (e.g., TV) sorted by date and you’ll see posts made in the past few minutes. Nor do I think it’s related to pageRank — their blog search ingest is based on pings rather than crawling. Besides, our blog enjoys a reasonable rank. Finally, it can’t be the case that Google’s systems are being overwhelmed by new blogs — the growth of the Blogosphere has slowed.
So I’m puzzled about what is going on. (goomtitag)
Update 1: Posted at 9:49, in Google Feed Reader at 10:14, indexed by Google Blog Search by ~19:15 and in Google’s main index about the same time. Maybe this is a clue — it used to be the case that a post hit the blog index within a few minutes and showed up in the main index after about twelve hours. This post hit both indexes around the same time — after about ten hours. Maybe there is now just one (logical) index.
Update 2: Hmmm. Another post seems to have made it into Google’s main index before it got into the blog search index. I imagine that Google revisited our blog home page as part of it’s regular crawl and picked up the new post.
Related posts: • Google mean time to index; • Mean time to index for blog posts; • Google’s blog search;
UMBC eBiquity Blog
February 24th, 2008 at 6:58 pm
I’ve noticed the same problem with Google blog search and my own posts. I thought it was just me
February 25th, 2008 at 1:10 pm
We can index new posts within about 60 seconds. The queue for who gets into that is controlled by an approximation of pagerank. I guess ebiquity might be losing page rank that is why it is not near instantaneous. The goal is make very high traffic sites like cnn quickly indexed.
February 25th, 2008 at 11:18 pm
Ouch!
I thought our blog’s pagerank of ~7 was pretty good.
March 6th, 2008 at 1:58 pm
Ah… have been trying to find information on time differences between posts in Google Blog Search and the main Google index. I had the opposite problem - a post I’m watching appeared very quickly in Google Blog Search but (hours later) has yet to appear in the main index. If it can take 12 hours and is normal, I suppose that answers my question! Thank you.
May 1st, 2008 at 1:03 pm
Thats stupid, I’ve had this same problem, one of my posts actually dissapeared from google