I don't have any experience with this, but I am an RN and my DD had a cast on both legs from surgery at one point, so here's my educated opinion:
Unless the doctor says it's OK, I would not try it. If the cast has a lining, it's basically cotton and if it gets wet, it will tkae a long time (maybe forever) to dry out. In damp conditions, you can get all kinds of problems with the skin and you won't be able to see what is happening.
If he has a fiberglass cast that can get wet, it's really meant to be OK for showering, not swimming. I would be worried that the cast would make him pretty unstable in the water.