There are not many days in Orlando which are literally "freezing", so I assume you mean "uncomfortably cold". And then I will say that I have no way of knowing what you would find uncomfortable.
That said, my advice is to bring swimwear any time of the year, just to be prepared. Florida winters can be unpredictable, with weather ranging from quite warm (70s) down to brisk (30s and below), sometimes during the same week!
Also, the pools are heated.
I think most people would agree that by April, things have warmed up enough that swimming should be just fine.