Overflow (other than visible) is one of the properties that create a new block formatting context and a block formatting context means that the element must take full control of the contents it holds. (e.g. If you set overflow to scroll then how could the element create a scrollbar if it did not contain its floats.)
Overflow (other than visible) is not the only property that does this but is the most useful in normal use. Floats, absolutely positioned elements, block containers (such as inline-blocks, table-cells, and table-captions) that are not block boxes will all contain floats automatically.
If you don't need visible overflow then overflow:hidden is the simplest to use but is not always the best choice.