I know, why get a 4 year old a camera.
It may be elsewhere, but around here, this isn't a silly question at all.
I don't know how to answer, though. When my kids were that age, I didn't have digital, and I didn't want to waste expensive film and processing (unlike with digital pics today which can just be deleted). I did let my kids take pictures using my camera with my help, though.
Today, I'm not sure about what I'd buy, because as you say, the cheap kiddie ones are just awful, and the better ones have a good chance at getting broken.
It was always important to me to teach the kids respect for the camera, regardless of what it was, just like anything else that's expensive/important. So I'd probably buy a decent, simple camera at as low a price I could find, but work with them on using it and try to keep it so that he only uses it with supervision. (Definitely wouldn't be running around a wedding with it.) If there is an accident, then so be it - but I'd try to avoid that if possible by keeping the strap on, paying attention, putting it back in its case, etc.
I bought my son a used Canon S2 IS on Craigslist when he was about 9 and worked with him on using it. He's been respectful with it, so now, two years later, I'm pretty comfortable letting him use my dSLR at home. Every now and then I download pictures and find some that he's taken with it, like this one, from my yard, using macro:
Good luck! I do think that nurturing kids in photography is giving them a great, lifelong gift.