We did it a few years ago. With the exception of not telling the driver where we wanted to go, it worked out quite well. We 'assumed' the family that got on before us had told the bus driver to stop at SeaWorld, but evidently they hadn't. So, we passed right by the SW parking lot before anyone realized it. The driver did stop but it was quite a hike back to SW.
Otherwise it is very easy, and inexpensive to do. About $1.50 pp each way (have the correct amount), tell the driver you want to go to SW. It's about 20-25 minutes. You hop off right at the parking lot entrance. When you want to go back to the TTC, just retrace your steps, and cross the street. Look for the bus stop with the 'paw' print on it...it's right across from the parking lot. The bus will come along shortly and off you'll go!