Absolutely!
Some days, by mid-day, it's warm enough even for those w/"southern blood" to swim...most days, especially by mid-day, it's warm enough for the colder-blooded northerners...
I swim almost every morning, regardless of weather (not when it's 30 or 40...but when it's 50 or 60)...the pools are heated so the only "hard part" is getting out, all wet, into the cold air...I just make sure I have a sweatshirt all laid out and ready to pull on as soon as I towel off...