Yes, attribute selectors are slow, but speed for those is only relevant when they account for more than a few percent of your processing time, which only occurs in very rare situations.
To put it another way, much larger performance gains are to be found elsewhere than in the selectors. The importance of using the attribute selector is to help protect from CSS changes, and to make the code easier for developers to understand.