You can swim pretty much year round in Socal. Of course there are times/days were the weather is much colder then normal or raining where you might not want to.
My kids have swam on every trip we have taken, including December, November, & April. And of course July & August too.