Web Security and Robots

Server Config
1

Hello!

I’ve blocked my site from users (at least I think I have…) as I’m still in development stage . I’m using CloudFlare and according to my reports, I’ve had tons of robots crawling my site. Can they do this if you need a password just to see the homepage? If it helps, the site is www.myalgebrabook.com.

Thank you,

Eric

2

The robots won’t get past the .htaccess login

3

Thanks for the quick reply!

Then, does this mean that my CloudFlare statistics are based on robots that landed on the page but didn’t get through?

4

Must be.

5

hello EastCoast,

You are right…htaccess can not get pass through robots.txt file…