Hi Zubanoid. Welcome to the forums.
There are some good questions here.
You don't have to get too caught up in the images issue. They are quite responsive if you give them % widths. The only issue is with download sizes for mobiles users, and to help with this, some people have come up with nice solutions (like those you linked to). But it's still a developing area, so we have to "watch this space", as they say.
I find the current state of HTML and CSS overly complex, convoluted and unnecessarily difficult to deal with (referring to versions and browser compatibility and not how to write mark-up and or syntax).
Actually, HTML and CSS are quite simple in nature, so don't get too worried. They also have really good browser support these days in all latest browsers, so things are a lot better than they once were.
How does one learn, let alone teach web design and coding without feeling confused as to what mark-up and CSS version to use?
That's easy. The current standards are HTML 4.01 (or XHTML, if you prefer) and CSS 2.1. Simple as that.
CSS3 and HTML5 are mostly still on the drawing board. Browsers are starting to implement bits of them, and so you can use these bits and pieces, while realizing that older browsers don't support them or providing fallbacks of some sort. But really, you can do so much with CSS2 and HTML4 that all this concern over support for coming technologies is not a valid source of concern.
I end up utterly dumbfounded as to what the future holds and where to focus developing my skills and knowledge.
The future holds HTML5 and CSS3, but it will be a good many years before browsers fully support them. And before that happens, they need to be officially released. Play with them and learn about them, but the reliable tools for current use are CSS2 and HTML4, and these are well supported by all browsers.
When will HTML5 be fully implemented?
I believe it will be released officially in 2014, and will be "fully implemented" officially some time in the 2020s.
Why does IE have so many web standards and compatibility issues?
Because Microsoft fell asleep at the wheel 10 years ago and has never recovered from that. Their browser release policies still suck, and unfortunately we are stuck with that. Just urge people to switch to Chrome, and hopefully M$ will go out of business one day.
Is responsive web design really the way to go for all web sites?
No, not at all. Some sites—especially large ones—are better off with a separate mobile site. And responsive design is just a neat "new thing" that people are enjoying right now. Whether it will last or not is not yet known. Some mobile users prefer to see the desktop site on their browser. At least on the iPhone, it is easy to double tap a region to zoom in and out. So, while responsive design is kind of cool right now and a buzzword, it's not necessary at all. Don't be bullied into using it if you don't want to. I find it fun, but on some sites I don't bother, as it isn't stricty needed.
What about device detection and using separate mobile sites – when should these be used?
Mainly on really big sites like news sites.
I do believe however that web standards and web browsers are making things unnecessarily convoluted and complicated for web designers / developers. Surely things can be consolidated and simplified?
I don't think that's really true. If you learn basic, good practices with HTML and CSS, you can be pretty confident that all modern browsers will render them nicely. The problems usually occur when people don't know how to code properly or try to get way too fancy and push their designs beyond the current limits of HTML and CSS.