I have stayed at the Swan twice, but will be staying at the Dolphin this time because I always seemed to be running over to Tubbi's to get food or a drink. I agree with those that say the Swan feels more "boutique" like, which is odd for a hotel that size. But IMHO, along with that comes the vices of a boutique hotel, mainly it's unwillingness to cater to the budget-minded.
Just try to find a bottle of coke at the Swan at 12:00am after a long day in the parks. If you are unwilling to buy it out of your mini-bar fridge, you are out of luck unless you want to make a trip over to Tubbi's in the Dolphin. Want a quick snack? Tubbi's @ the Dolphin. Not so quick when have to walk to another hotel.
I sound like I don't like the Swan. No true. Love it. But now returning with my 3 and 5 year olds, the Dolphin is the only choice.
I think for those with families, or the budget-minded (like you), the Dolphin is the place to stay.