steveob200653 — 2012-03-27T08:18:43-04:00 — #1
How do I stop maintenance pages from being indexed by search engines?
I can stop them from being used if the user does not have the correct privileges but it would be better if they weren't indexed to beging with.
stevie_d — 2012-03-27T08:53:17-04:00 — #2
If all the maintenance pages are in the same folder (and nothing else is in there), then you can use robots.txt, with a file containing something along the lines of
Otherwise, you'll need to set it on a file-by-file basis.
Include this line in the <head>:
<meta name="robots" content="noindex, nofollow">
(If you want search engines to follow links from that page then leave off "nofollow").
dklynn — 2012-03-27T17:43:51-04:00 — #3
robot.txt is notorious for being ignored by bots attempting to index (or scrape) your website. If you're concerned about this, either:
Move your maintenance scripts out of the webspace
Unlink them from your website (no link from the website = security by obfuscation, i.e., poor to no security)
Password protect your maintenance folder AND use mod_rewrite in that folder to require authentication, i.e., only provide access to your (fixed) IP Address or via an environmental variable only you have.
The best option is a combination of (Apache) password protection on the subdirectory AND use of a login using a strong password hashed for access.
steveob200653 — 2012-03-27T17:54:10-04:00 — #4
Thank you both.
I will be using a mixture of meta tags to stop them from being indexed and a separate security on each page that checks that a user is logged in and has the correct privileges should a page become indexed by accident, or if search engines ignore the <meta robots>
dklynn — 2012-03-27T19:10:56-04:00 — #5
If the maintenance pages are not isolated in a password protected directory, that's as good as it gets!
BTW, I've been in a conversation with Manuel Lemos, creator of phpclasses.org about hosting classes to break md5 hashed passwords (using a rainbow table lookup hosted by md5cracker.org (or similar)) and the contention of mine is that these lookups shouldn't be available to script kiddies, his is that it can make for a good check on a hashed password. We're both correct but the key to security is to use a STRONG password, i.e., one with uppercase, lowercase, digits, special characters and spaces of sufficient length to make it impossible to crack by brute force in less than a few centuries. Of course, that's if you really need to protect your maintenance pages!)
steveob200653 — 2012-03-27T19:22:08-04:00 — #6
yes. I've become aware that md5 is not good enough.
as for my own admin pages: at the moment it is not that critical. I provide the facility but I also maintain scripts so that if someone does break in and make a mess, I can just rerun scripts to make everything right again.
When I rule the world there will be no need for any of this security malarky ( in any walk of life ).