We also did both in Jan 09.
Tusker House Pros: Inside a park, better food, buffet layout had pods so easier to get to food (no lines) Jungle Juice is excellent.
Tusker House Cons: A bit noisier with the hard floors, character interaction was for shorter periods (similar to other character meals we have done).
Cape May Pros: Very quiet, long character interaction with Goofy and Donald (Minnie was pretty quick).
Cape May Cons: Food is bland, nothing unusual, just standard breakfast fare. Also, if you are not staying right there (we were staying BC), it's not that convenient to get to it.
We will be returning to Tusker House as we loved the food there. Cape May was not terrible but not high on our list to return to.