cpradio — 2013-05-18T20:56:50-04:00 — #1
Okay, so I run two very small sites on a VPS (both are Wordpress using W3 Total Cache) running Apache. I've seen a prior thread very similar to what is happening with my server, but I want to approach the topic differently.
First of all, I want to try and identify the culprit causing the memory usage, but before I get into that, here are a few key details:
2 GHz CPU with 300 MB of RAM
It has 2 wordpress sites, the first had 1224 entries in the access.log for today and the second had a dismal 406 entries (let's face it both are dismal).
The entries for the first wordpress site are primarily trying to hack its way in by guessing an account to automatically login to wp-login.php, the latter site all seem to be legitimate requests for access.
Lastly, the requests are for example.net which is a mirror domain for mydomain.com. example.net doesn't really server any web presence purpose, so I could probably block all requests to that domain and that would kill all of the wp-login.php attacks (example.net is primarily kept for e-mail purposes).
Okay, so what that out of the way, I'm trying to figure out how to identify what is causing the spike in memory usage. Supposedly it spiked at 1:00 AM PST, so they automatically bumped the memory limit by 25% to handle the excess load. They are telling me to throw more RAM on it, but that is the dumbest thing I've heard of. To quote my reply: "That's like stating to throw more wood on a fire because the flame is too high. Sure it will lower the flame for a while, but once it catches up, its not much more fiercer than it was prior."
I really need to identify the cause for the spike. Considering my fire analogy, the reason for the increase flame could be an accelerant (think pine cone, lint, paper, etc) or a pocket of air that became available, feeding the flame. You have to analyze the fire to figure out which it is (sometimes waiting till the fire has died to see remnants of the cause).
So what tools/processes do you use to identify high memory usage on your web servers?
sahostking — 2013-05-19T16:50:19-04:00 — #2
I would use "top" on linux box to start off with and see if it provides you with a common process causing the high load.
If it shows your website user account doing most then its something on the site causing this.
I would recommend starting off installing something like Wordfence security. That would block of brute force attacks first to your login page. From there see if load drops.
Lastly and probably the easiest and most recommended way is to disable all your plugins and start up each one and see what happens. You may find it to be the most unlikely plugin causing the issue.
sahostking — 2013-05-19T16:51:34-04:00 — #3
cpradio — 2013-05-20T06:07:17-04:00 — #4
Hmm... I really don't think it's WordPress (or apache). For it to be one of them, I'd have to see access logs that are hitting heavily and I'm not seeing that. I only have a small set of plugins enabled.
After logging in to the server, I realized I had a few bash scripts setup on cronjobs. I've temporarily disabled them to see if they are the cause, then I'll re-enable them at various times to see if the spike happens again (times when I'm actually awake and could watch the memory usage interactively).
p3 profiler looks really cool though. I think I'll still give that a try and see what turns up.
eastcoast — 2013-05-27T19:53:57-04:00 — #5
Wordpress sites are getting hammered currently by a global bot net. I've had to make modifications to some wordpress sites I host to decrease resource from wp-login getting hammered hundreds of times every minute from a diverse range of ips. Probably worth checking you're not on the receiving end of that. It seems to come in bursts, half an hour to perhaps an hour of sustained attempts at a time.
cpradio — 2013-05-27T20:13:53-04:00 — #6
Just an update:
I did more research and it seemed to be around the time I had a cronjob scheduled to do some "miscellaneous" backup routines. The routines simply created a gzip tar of the sites on the VPS and their MySQL databases. Then it bzip'd them so they could be downloaded from another location. Once I disabled that cronjob, the spikes went away.
So then I decided, well, I really don't want to get rid of my backups... so I changed how it works, and it seems to be running fine still. The biggest change I made was to use zip instead of tar and bzip2 and that must handle memory a lot differently than the other two... as it has yet to spike.
I'll have to keep an eye on it. If zip starts to do the same thing, I may have to use rsync and sync it with a remote location, and handle the archiving on the remote site so the resources are consumed there instead of on the VPS.
eastcoast — 2013-05-28T18:24:43-04:00 — #7
Did you pipe tar to bzip or run the commands sequentially? Piping uses more memory (though less disk i/o). Also, is it an openvz vps?
cpradio — 2013-05-28T18:27:39-04:00 — #8
No, I didn't pipe it. I'm not sure what openvz is, but my VPS is with DreamHost...
serverstorm — 2013-05-29T18:09:33-04:00 — #9
Openvz is a type of virtual machine. Typically KVM and Openvz are the most common types of VM's found on virtual servers. Typically openvz is a better type of VM for linux (uses less server resources in Proxmox (a Virtual Server Eniroment))than KVM
Glad you resolved it re the backup, but it would be good to further understand why the bzip2 is slow and hungry!
cpradio — 2013-05-29T18:22:52-04:00 — #10
Okay, well I'm new to the VPS side of things, so maybe some of you can help me out here. I am on a Linux machine, so I imagine they may be implementing openvz, but they might be using something entirely different too. However, I did see my memory usage spike last night around the backup time. Here is what I'm not fully understanding:
The Cache Memory states 352 MB at 23:07, however, at 22:42, it was 92 MB. Actual Memory at those times were 22 MB and 15 MB respectively.
So my questions: What is cache memory to a VPS and how does my backup script seem to push that to its limits? What is actual memory to a VPS?
As for bzip2, I think that shot the memory sky high because the gzip tar file was HUGE! So it had to load that file into memory and then run it through bzip2. Whereas, if you could write the physical file and add each individual file to it one at a time, that would limit what needed to sit in memory (I think).
I may need to figure out what exactly is causing there to be a large footprint on the site, I'm backing up to see if that is that particular content could be excluded (think cache data for the site -- that can always be regenerated on restore).
cpradio — 2013-05-31T09:48:08-04:00 — #11
So I took a hard look of one of my recent backups and found a few large files that really do not need to be backed up (talking a few MB each, total = 25+ MB). Once I excluded those files from the zip, memory usage dropped dramatically (under 200 MB) during the backup time frame.
That was likely the issue with tar too, and I bet bzip2 still struggled due to large tar file being generated that it had to process (but I may try that method again in a few weeks).
dklynn — 2013-06-01T19:22:17-04:00 — #12
It's my understanding that WP (et al for CMS's) require an immense amount of RAM so running two WP installs may also be problematic (on top of the server overhead for Virtuozzo, WHM and cPanel as well as your mail and security daemons) so bumping your "burst" RAM may help your day-to-day operations. It's the RAM/burst RAM which limit VPS's usability more than the CPU, storage and traffic limitations.
Just food for thought.
cpradio — 2013-06-01T19:33:55-04:00 — #13
I ended up moving one of the WP installs to a different server, so now it just holds one WP and a basic pure static content site. Neither of which get a lot of attention/visits yet. I'm still keeping a very close eye on it and will continue to do so (it has run fine for nearly a year and just recently it has started to spike...)
eastcoast — 2013-06-02T12:37:54-04:00 — #14
Not sure if Dreamhost use Openvz, but when hosts do use it, it is primarily chosen on a server as it is easy to 'oversell' resources. Depending on how heavily oversold the server is, you can see memory warnings and issues that are down to other users on the system. I'd only ever use an openvz vps if I owned the server and knew that resource use of end users was relatively predictable, or was running something trivial that didn't need good uptime. KVM and Xen are more heavyweight virtualisation methods but are generally a lot more reliable. Here's an article that covers the main differences http://hostingfu.com/article/xen-or-openvz