My question is like very simple but for me a very big cause to think .
In robots.txt file if i indicate one of my URL having like [noparse]www.example.com/category/internetmarketing.php[/noparse] , so if do use this particular page not to crawl in Google . Then is it possible that only this URL will be considered for not to crawl as it having main category page also.
According to my it will consider main url like [noparse]www.example.com/category[/noparse] page.
So will please somebody clarify me how this problems will be removed .
[FONT=Verdana]If I’ve understood you correctly, you want Google to crawl your “category” directory, but not the single page within it called internetmarketing.php.
In that case, all you need in your robots.txt file is:
User-agent: Googlebot
Disallow: /category/internetmarketing.php
It will continue to crawl every other page in “category”, (provided it can access them, of course).
If you want to block all search engines, and not just Google, then use:
User-agent: *
Disallow: /category/internetmarketing.php
Remember, this will only stop well-behaved bots whih respect the robots.txt file; it won’t protet you from bad bots.[/FONT]