No, it certainly isn't the end of the world, and clever phone users will know how to patch it.
But...... a lot of phone users are NOT clever, don't know how to patch it and will just be irritated. Irritated users are less likely to stay and less likely to buy. Visitors are getting more and more used to ease of use, and silly, outdated problems like fixed width design give a really bad image to a company that still uses this antique way of desing imho.
Maybe I'm just a difficult customer, but when I try out a shop and there is a queue of people waiting to be served, I'm out. If I visit a website that loads slowly or is hard to navigate, I'm gone. If dealing with the company is pita before they got my money, I really don't want to know how they are after they got my money, and I have a problem with their product.