You also have to remember that the end user may be downloading with a torrent client and their bandwidth is choked or the node they are on has more users than the contention ratio which is causing a network issue over speed.
ISP's work on the principle that not everyone will be online all together, so at certain times of the day you will have network slow downs and you will notice a huge service drop at times when TV programs tell people to visit a URL for something like a competition.
Generally you will have network problems with end users because it is the end user and someone on their branch of the node that feeds their service is eating most of the bandwidth ergo site slow downs as they will report.
The other side of the coin is that the server your site is on is over loaded with traffic because of a website that may be sharing or streaming video, you should take the issue up with your host citing that you have had complaints from people visiting your site that it has been slow to respond where it hasn't before, timeouts occur and you spoke to someone in a web forum that has had experience in network administration (me) and they said that you (your ISP) should have a quick look to ensure than no one is running a torrent server or hosting a stream that is pulling the server down and eating excessive bandwidth, it could be possible that the server is overloaded in any case and you can request to be moved to another server which they will do if the server needs it. The biggest cause of server load is PHP and other server sides running excessively long scripts that not only eat memory but also clock cycles. You never know, the server may have a site thats been hacked and someone is having loads of fun!
So give them a poke, because some server farms will park websites that get very low numbers of request on to a server that remains dormant until its woken up.
In the mean time tell your visitors to stop moaning, you are investigating the problem.