kreut — 2011-07-28T14:29:41-04:00 — #1
I've blocked my site from users (at least I think I have....) as I'm still in development stage . I'm using CloudFlare and according to my reports, I've had tons of robots crawling my site. Can they do this if you need a password just to see the homepage? If it helps, the site is www.myalgebrabook.com.
eastcoast — 2011-07-28T15:37:18-04:00 — #2
The robots won't get past the .htaccess login
kreut — 2011-07-28T16:23:19-04:00 — #3
Thanks for the quick reply!
Then, does this mean that my CloudFlare statistics are based on robots that landed on the page but didn't get through?
eastcoast1 — 2011-07-30T15:25:40-04:00 — #4
aruninfo — 2011-09-12T03:44:39-04:00 — #5
You are right.....htaccess can not get pass through robots.txt file...