I've used both and I prefer Wordpress by far. Google also seems to like Wordpress site better. I think the software is just designed better for SEO.
As for load times, this is a factor the Google takes into account for page rank. So if you site is having this issue, may want to look into optimizing it. Wordpress has some great cache plugins. I don't recall about Joomla, but may guess it that they probably do so as well.