It doesn't slow your website, as such, but video usually takes a while to load on a page, even if it's set to download progressively. If a lot of people view the video and it's quite large, you may find your web hosting bandwidth costs going up.
A really easy way to do this is to upload your video to YouTube, Vimeo or similar and just embed it on your page with the simple code they give you. They are also optimized for faster downloading.
A third option is to buy separate CDN (content delivery network) hosting for your videos and then embed them on your pages like you would with YouTube. CDN hosting is also optimized for fast loading, and is usually pretty cheap (I'm thinking Rackspace or Amazon Web Service etc.)