codlib — 2010-09-28T02:16:16-04:00 — #1
Is there any importance in the position of sitemap keyword in robots.txt?
Specifically my query is, I have some disallow URL's in my robots.txt. But the same links are appearing in sitemap.xml and sitemap.xml is placed at the end of robots.txt file. Will the links specified in disallow section omitted when crawling? Or do i need to add the disallow links after sitemap declaration?
Also, If i add an old url to disallow section, will it remove from the google search index?
weathor — 2010-09-28T04:34:01-04:00 — #2
You could test it in google masters! It doesnt change your robots.txt, just for test!
codlib — 2010-09-28T02:59:36-04:00 — #3
Thanks for your reply. But still I am not clear about the following.
1.) Will the sitemap command override the disallow command in robots.txt?
2.) If a url is present in both site map and disallow section, which will get precedence?
ralphm — 2010-09-28T02:30:27-04:00 — #4
As I understand it, it doesn't matter where you place the sitemap directive in the .htaccess file. See this page for further reference: