What do you mean by "without using 3rd party script"? In order to have the controls show up, you have to explicitly tell the browser to do so...
<audio controls />
...so if you don't want those disparate controls showing up, don't set the "controls" attribute.
Are you sure you don't want to use something like audio.js? You'll save yourself a lot of headaches.