The reason that crawl errors can hurt you is because Google can't find the pages that it is looking for.
If you have moved pages or reorganised them, you should always try to put a 301 redirect to the new page that gives the best match. That way, search engines and people alike will be taking straight away to the new page. Search engines will notice that the URL has changed, and will (eventually) update their index with the new URL – but in the meantime it doesn't matter, because anyone following a link to the old URL will be redirected!
If search engines hit a 404 error, they will try it again periodically, in case it was just a blip and the page reappears. If you're continually getting 404s then it may well be that there are pages out there linking to your dead URLs, which is why Google keeps following them afresh. And if Google's doing that, real people are likely to do so as well.
If the page continually gets 404 errors then after a while Google will conclude that that content doesn't exist on your site any more. If you have genuinely removed it then this shouldn't have any major impact on your rankings for the content that you do still have ... but if the content is still there, but in a different place, you will be losing rankings, because Google may not be finding it, and certainly isn't giving it the full weight of all the links that should be pointing to it.