We do it all the time, and so far we've never had any problems doing everything we wanted to do. Staying onsite makes it considerably easier, but if you're going at a less crowded time of the year, you should have very little problems (especially since you pointed out that you're not big riders - that said, there's a lot of stuff there to ride that is worth your time that's not all "thrilly."
The 1 day 2 park pass is kinda pricy, though I think it's cheaper now than it was last year (I BELIEVE it's $70, but it is worth the extra cash since you are getting two parks out of the deal). As someone else said, it's not a terrible idea to get a 2 day 2 park pass. That second day doesn't expire, so if you ever decide you want to go back, you're set.
You won't be killing yourselves, though. We've worn pedometers to the parks, and found that we only walked about nine miles that day, versus the fourteen miles we walked at the Magic Kingdom (and I doubt not riding the tram to the TTC added that many more miles...)
