hayem — 2012-06-05T13:56:40-04:00 — #1
I am using this code on my site.I have added this under root.
trying to exclude ppc folder from search engines.is this correct?
2-also i have a subdomain in which i do not want search engines crawling.I have added this 2nd robots.txt under subdomain.maindomain.com
will these 2 robots.txt work?
technobear — 2012-06-05T14:26:57-04:00 — #2
To exclude the folder ppc from all bots, all you need is
I read somewhere (and I can't now remember where) that Googlebot likes to be called by name, and therefore one should add
I always keep to that practice and have run into no problems, but it's probably redundant.
The robots.txt protocol does not include "Allow", only "Disallow". By default, the whole site will be crawled unless folders/files are specifically excluded.
This page may help.
hayem — 2012-06-12T13:05:05-04:00 — #3