There are 120+ possible factors that could effect your website. If you can provide more details, we would be able to help you better than just answering generally.
Most of the time, when your site get penalized, it is pretty much over. Recovering from any penality is a longer process than starting fresh again.
Good luck with your site.