That tells me you probably have something being downloaded that is of some significant size. Have you opened dev tools after loading your site (after clearing cache) to see what the filesizes are?
First I would have to know your website address so I can verify this for you. Optimisation is one thing… then size… host …dns’s… geo location and many many more factors that your loading speed depends on.
Try using a CDN service like Cloudflare. Also, Did you optimize your images ? Try using Pingdom to see what is slowing down your website, it checks the site speed and allows you to see which element is taking long to load etc. If the problem is with the host, try switching, Hostgator works pretty well in my experience, not sure your website needs by the way. Well , I hope it helps you !