Our family cottage is not-too-far from there in the Kawarthas area North of hwy 7). The drive through TO is manageable because you will quickly figure out what time is the right time to go. We leave at 5 am Saturday, for example. Several of our cottage neighbours have found there's a window between cottage traffic times and regular TO traffic times. Also, the traffic is not so bad once you get past the 400 (it gets bad again near Oshawa-there's always a slow-down there for some reason).
The cons to me are the maintenance & costs. There are a lot of extra costs to owning a cottage, including extra phone service, extra taxes, gas to and from cottage, the cost of owning, operating and maintaining a boat (you have to have a boat with a cottage!), furninshing the cottage, and entertaining (you will have guests!).
The pros are: it's your own place so you can do what you want with it. In the right spot, it's peaceful and relaxing and there's nothing like sitting down on the dock with a cold drink and a book on a sunny summer afternoon. For you, Belleville is a decent town with a mall and lots of grocery stores. You'd be a close drive to Sandbanks Provincial Park in Picton, or could go to Kingston on a rainy day.
One thing to consider: the market. Cottages have gone up a ridiculous amount in the past ten years. Ours was purchased about 15 years ago and the price was $60K. That was for a pretty basic cottage, double lot, no septic but plumbing, hydro and phone. Since then, there has been reno on the cottage and septic installed but the value is appraised at over $250K. I would talk to someone who knows the market and make sure you are getting a good deal because, like most real estate, it's stagnating or even going down.
We love our cottage but it is expensive to own and maintain a cottage.