We went in Jan. 2004, it was great---no crowds, we swam most days, but wore long sleeves some too. It was never freezing. I think MLK day is probably a little busy, but otherwise, you should be fine.
The next year (oddly enough) we went in early Feb. froze to death and there were more crowds!! go figure.