Organizations like the White House or Disney, what have you ... don't have these huge needs for lighting-fast speeds. Nor are they likely dealing with the kinds of hardware and bandwidth limitations that a lot of mid-sized companies might be dealing with on a shared godaddy account or even your typical $20/month VPS.
With the right caching enabled, there is essentially no difference between what the various CMS's are ultimately cranking out. From the server's standpoint, it's just a static page in a tmp or cache directory. Whatever process was used to produce that page, however ugly or slow, may or may not be relevant.
If you're dealing with Twitter, or an online stock trading service for example, where up-to-the-second information changes need to be reflected immediately ... then it really starts to matter a lot more, and really, just about any CMS system is going to start looking pretty slow and clunky. But then again, it's not very likely one would use Drupal or Joomla for that kind of thing anyway.
If you define "speed" as speed to develop ... then you're getting a lot of bang for the buck with Drupal. You can crank out some pretty complex apps with a lot of advanced functionality using things like organic groups, cck, views and entity/node relationships ... in a good deal less time than it would take to make the same app in Ruby on Rails, for example -- and have a CMS attached to it out of the box. And that's not a knock on ROR. It's when the application is actually "completed" where things get interesting. You might be able to crank something out in no time with Drupal or even Ruby on Rails, but maintaining it is another challenge altogether.
On the other hand ... for a much simpler application with much more modest requirements, Drupal is probably overkill, and you'd probably be able to develop the same app in much less time using something like Wordpress.