We went last year on the second coldest day of the year! (christmas was colder). It was still wonderful - used full-length wetsuits, swam with two dolphins, stood in the ray pool, etc.
Me, DH, DD-then 11. She wants to go back this year! (not gonna happen for a couple more years, though)
I say - go for it, it is a wonderful experience. And you will have the secret satisfaction of knowing that you are swimming with dolphins who are not kidnapped from the wild.