Here’s my situation: I put up a new article on my site and I went to Google and searched for part of the article’s title, then chose “past hour” to refine my search and got nothing for my site. There’s plenty of results for other sites, but none at all for mine.
Google crawls my site fine, an my robots.txt file isn’t blocking anything. What can be causing this to happen? I guess this has been going on for a while but I only noticed now.
Well from what I can see, it’s taking longer for Google to index my pages. For example, an article I wrote a few days ago is up (Nov. 5) in the results, but nothing from today. I’m wondering why other sites are getting picked up right away.
Exactly as to how often a site is crawled and the content is indexed is far more complicated than any of us can say with 100% certainty. However, the general rule of thumb is that the more popular your content is (backlinks) and the more often you publish new content, the faster your website will be crawled and indexed.
Are you using Webmaster Tools to track the movement of Googlebot and the other spiders through your website?
Sites that Google knows, trusts and rates highly get crawled and indexed more frequently than those that are new, not so highly rated or infrequently updated. If your site is fairly new or fairly small, it shouldn’t be any great surprise that Google doesn’t check it as often as bigger, well-established and more popular sites – that’s just the way it works. As with most things to do with Google, the secret is perseverance – give Googlebot good reasons to keep coming back more frequently, and it will.
I see the article being picked up now, so it took a few hours which is okay for me. I know they go by backlinks, age, trust, etc but that’s what’s surprising because my site has been online for over 10 years and I post content daily. I’m going to keep an eye on the googlebot and see how often it’s picking up my content.
How easy do you make it for Google to find the new content? Is it featured on the home page? Do you have an RSS and/or Twitter feed listing all updates? With those, a key point in terms of getting new articles indexed quickly (which may or may not be your main goal) is to minimise the amount of ‘fluff’ on those feeds and just have the updates and key highlights. That’s particularly true for RSS – if you use both then you can keep RSS for “what’s new” and use Twitter for wider social interaction.
Sorry if this sounds condescending, but the answer seems rather obvious: you either overestimate the frequency of your site being crawled, or Google’s appreciation of this particular article.
Unless you produce new content several times a day, it is unlikely that Google will index (not to mention give a top ranking) your product immediately.