Yes, the content is always served as HTML in the browser. However, if you are using a caching plugin, you are serving static HTML to visitors who are not logged in. The Wordpress engine is not running to generate the output and make database queries as that content has been saved in the cached files. The server transmits the cached HTML files to the browser instead of running the Wordpress engine which requires much less server resources.
Wordpress is a very bloated script. It consumes a lot of server resources to generate one page view. That is why many people using caching plugins. The question now is why your admin area is loading slowly.
If you log in to your Wordpress then view the regular non-admin site pages, does it load slowly? If so, there is a good chance your web server is overloaded. The admin area checks external websites to check for updates to the Wordpress core and plugins. It is possible that could be the cause of the slowdown. But I have a feeling you may be on an overloaded server. That's the first and most simple thing to check.
I have a simple little script in the post linked to below that you can copy and place into a PHP file and then run the script by navigating your browser to it. It will output the server's load average. If the load average is high, we know right away what the problem is. If the load is not high, then further investigation will be required.