Google slow to index blog posts?
Tim Finin, 9:49am 24 February 2008Last week I noticed that some of our blog posts took a long time to show up in the Google Blog search index. During the past year, Google has been very fast at indexing blog posts, typically taking less than five minutes from the time is made to when it shows up in their blog search index. But this week it seemed that our posts, or at least some of them, took more than twelve hours to be indexed.
Yesterday I tried to watch a post I made on the IT job market which I wrote just before 11:00am (GMT-5). It showed up in Google Feed Reader quickly enough but had not yet appeared in Google Blog Search when I finally went to bed 14 hours later. When I checked at 9:00am today, it was there, so it took sometime between 14 and 22 hours.
It’s not the case that all posts are being delayed — do a Google Blog search for a popular term (e.g., TV) sorted by date and you’ll see posts made in the past few minutes. Nor do I think it’s related to pageRank — their blog search ingest is based on pings rather than crawling. Besides, our blog enjoys a reasonable rank. Finally, it can’t be the case that Google’s systems are being overwhelmed by new blogs — the growth of the Blogosphere has slowed.
So I’m puzzled about what is going on. (goomtitag)
Update 1: Posted at 9:49, in Google Feed Reader at 10:14, indexed by Google Blog Search by ~19:15 and in Google’s main index about the same time. Maybe this is a clue — it used to be the case that a post hit the blog index within a few minutes and showed up in the main index after about twelve hours. This post hit both indexes around the same time — after about ten hours. Maybe there is now just one (logical) index.
Update 2: Hmmm. Another post seems to have made it into Google’s main index before it got into the blog search index. I imagine that Google revisited our blog home page as part of it’s regular crawl and picked up the new post.
Ebiquity Blog
February 24th, 2008 at 6:58 pm
I’ve noticed the same problem with Google blog search and my own posts. I thought it was just me
February 25th, 2008 at 1:10 pm
We can index new posts within about 60 seconds. The queue for who gets into that is controlled by an approximation of pagerank. I guess ebiquity might be losing page rank that is why it is not near instantaneous. The goal is make very high traffic sites like cnn quickly indexed.
February 25th, 2008 at 11:18 pm
Ouch!
I thought our blog’s pagerank of ~7 was pretty good.
March 6th, 2008 at 1:58 pm
Ah… have been trying to find information on time differences between posts in Google Blog Search and the main Google index. I had the opposite problem – a post I’m watching appeared very quickly in Google Blog Search but (hours later) has yet to appear in the main index. If it can take 12 hours and is normal, I suppose that answers my question! Thank you.
May 1st, 2008 at 1:03 pm
Thats stupid, I’ve had this same problem, one of my posts actually dissapeared from google
June 1st, 2008 at 5:44 am
Thats actually annoying, its not only a problem with blogs, its also a problem with the normal Google index.
Some websites seam to get their pages indexed the moment they are made, while others need to wait for weaks, or even months.
Submitting a sitemap dosn’t even make google index you pages faster, actually i have sevaral pages from the beginning of may, which havn’t been indexed yet.
Lets be clear about something, google really has some wired ways to work sometimes, picking pages/posts more or less random at times.
July 15th, 2008 at 10:36 am
We have all our posts indexed reasonably quickly for the main index, but have major issues in getting in the blogsearch index. We can ping all we want (feeds, URLs), but it seems that we only get into blogsearch after the main index, or not at all. Worse, a colleagues’ wordpress-hosted (as in, hisblog.wordpress.com) blog pops into the blogsearch index almost immediately. Could that support the pagerank/index queue item above? Could my sites’ content somehow considered spam? It would seem almost certain that I’ll die without resolving this issue.
November 15th, 2008 at 9:45 am
Hi, there are major indexing problems again, it started in the evening of 11/13/08, last all day 11/14/08, and still having problems 11/15/08…..does anyone know what causes these indexing problems? Are they doing maintenence? After I ping, I usually go into blogsearch immediatley and after about 30 mintues get into Google index….but both have been taking over a day in the past few days….its weird!