There never was any real growth in the "field" of SEO, and I'm afraid that your friends are right in suggesting that there isn't much of a future in it. Over time SEO, social media and many of the other "online" fields will eventually just merge into online marketing as techniques become more refined. SEO is in no way an IT field so its future does not lie there.
SEO is all about domain knowledge, but sadly there are many that lack that knowledge and just spout whatever techniques they read about on forums and blogs. The best SEO guys I've ever met are those that disregard all SEO news and focus on the marketing side of websites and data that from their experience is guaranteed to work.
I'm a programmer by trade, but I've dabbled in SEO for a number of years. I've also had a lot of success over that time, with my main achievement being the acquisition of a company I formerly worked for by a large multi-national brand. During my tenure traffic rose five times over what it previously was.