Wow, what an interesting question.
I have a lot of American friends and the thing we really notice among us is healthcare (we all have young kids so fairly regular contact with health care). I am appalled by the cost of their health care, and that not everyone is covered, and they are appalled by my rationed access. So, it goes both ways.
I lived in the Toronto area about 15 years ago and still visit--my favourite part as another has said is the multicultural aspect. It is such a vibrant city. Not without its racial conflict, but lots of tolerance and respect too.
I quite believe the average Canadian is well off, certainly by international standards, but an affluent American might not have the "user pay" options they are used to up here. I.e., having to wait to see a specialist or have surgery...not having access to private schools (though Toronto is well served as I recall)...and it seems even the public schools in affluent US neighbourhoods are very well-resourced. Here, there's a more equal distribution of resources among schools (though don't underestimate the power of parent fundraisers and lobbyists) and the overall effect is good quality. (I used to teach in the Greater Toronto area.)
Language--I wouldn't worry unless you're working as an editor for the CBC.

Really. Dialect and accent vary among all regions, and I don't think the border makes that much additional difference.
I don't know what your local real estate values are like; I know some US cities are really hot and others not, so you need to compare what YOU would expect to pay/find reasonable for housing. I live in a very cheap area for real estate (small town), but it's reflected in salaries, etc. By Cdn standards, Toronto is not cheap.
As for the disposable income thing, we are much more highly taxed, so there is less money to spend...but then we don't spend it on health insurance or schools the same way. Don't know how that would work in your situation, whether you'd be eligible for OHIP (health plan) or still have to pay...
Don't know if that helped, but it's a very interesting discussion, and I thank you for asking the question! Good luck in your research and decision making.