I remember having this same situation years ago, but I don’t remember how I solved it at that time. I’m referring to the single ’ and double " keys. I have to hit a space or any other key before one of the two, I would like to use, appears?
What? Are you are saying when you want to press ’ you have to actually hit the Spacebar or something before it prints on screen? Obviously with " you typically would have to press-down SHIFT.
I suspect it’s to do with your “keyboard and language settings” where your quote key might be set to do another action like an accented character.
Maybe: Start > Control Panel > Regional & Language Options.
Have you have enabled an international keyboard layout or something strange?
Thats indeed the case 
I have my keyboard set to Unites States International (Since that is the layout of my keyboard). When I just looked in Start > Control Panel > Regional & Language Options, as you suggested, I saw an extra option int correction (whatever that is). I removed that one but the problem persist.
donboe have you just the plain US option set? If that still isn’t working it’s worth giving the UK option a try too! (you’d never know) 
Well, there’ll most likely be a mismatch somewhere between the keyboard and language settings. If you have another machine spare you could compare them. Though it would explain why you get this weird effect.
Obviously I have a UK Keyboard and it does differ to the US-Keyboard slightly http://en.wikipedia.org/wiki/Keyboard_layout#United_Kingdom If you read the US-International section it says uses keys ', `, ", ^ and ~ as dead keys.
So as my first setting I have:
EN English (United Kingdom) rather than EN English (United States) US and my Key settings are also set as such.
It is clear that the problem is somewhere between keyboard and settings. And the best option is to use that keyboard on another machine and see if it works there, then apply the same settings on your machine too.
I indeed tried that and than everything works fine
There must be a mismatch indeed. Like I said above, when I change it to EN English (United Kingdom) everything is working fine. Thank you for the article by the way.
That could be bit of a problem since I’m talking about a laptop 
That’s great news Donboe, glad everything is working for you now
No worries, and now you’ll know how to diagnose the problem if it reoccurs again.
I’m very glad indeed. Thank you for the input