I am confused that why google add following code in blogger robot.txt
User-agent: *
Disallow: /search
Allow: /
If I am not wrong, Disallow: /Search means your blog will be not visible in search results?
Complete robot.txt is as under
User-agent: *
Disallow: /search
Allow: /
And secondly, when I add the custom robot.txt , it get saved but don’t take effect practically. I check it by visiting
[FONT=verdana]
If I am not wrong, Disallow: /Search means your blog will be not visible in search results?
No. That’s not what it means.
What it means is that any pages in the Search sub-folder of your blog won’t be visible in search results (or, more accurately, won’t be accessed by robots).
In Blogger, the Search sub-folder is where the label versions of your posts are stored. For example, if your blog has a label named “recipes”, then any posts with that label will appear in a folder named something like www.MyBloggerBlog.com/search/label/recipes. It’s that folder that the robots won’t access.
But that’s not a problem, because the same posts will also appear in another folder, the URL of which is based on the date of the post. So all that this particular line of the robots.txt is doing is telling the bots not to index the same post twice.
In general, I would suggest you leave the Blogger robots.txt file alone, unless you have a good reason for editing it.
Mike[/FONT]
Thank you Mike for this help. Now I understand the reason behind robot.txt coding.
BTW, many guys are talking on their blogs about how and why to change it. But your approach is more rational. 