I just got back from 12 days in WDW. I took my DSLR with me every day and everywhere. I have a 2 year old DD. I my my Nikon D3 with a 17-55mm f2.8 lens on it. In my messenger camera bag (Tracy Joy) I carried my 50mm f1.4, 85mm f.14, my Flip video camera, wallet, cell phone, 3 diapers and a package of travel wipes. At night, I also carried an SB-800 speedlight, 2 pocket wizards and then put a small tripod in the stroller. And yes, I carried this EVERYWHERE for 12 days.
This is the advice I give EVERYONE who asks me... What are your intentions with your camera while in the park? Is it to get cool artistic shots? Is it to just snap away at moments with the family? Is it to improve your skill as a photographer while in a different locale?
For me, I carried all this stuff because I wanted to. I don't take snapshots, I use my time in the parks to shoot stuff for me. Since I'm a pro, often I'm shooting for client work, and this was an opportunity for me to stretch my wings and play around with concepts I might not necessarily have time/energy to do at home. But part of my fun is lugging all my equipment and doing these types of shots. For me it's not about going on the rides, or making sure that I have a light pack.
That is why I say that you should sit down and ask yourself what you are trying to accomplish. Having 3 little ones is tough enough... and honestly, the one morning I had DD all by myself while DH went and ran some errands, I left the camera back in the hotel because I couldn't handle her and the equipment/stroller all at the same time at DTD.
If you are too scared that you'll ruin the camera, or you don't want to have to lug everything around, just leave it behind. For scrapbooking memories, a P&S will give you some great shots and give you the freedom to enjoy the moments with a little less baggage. But if you are trying to improve skills, or want super cool/artistic shots, and you aren't worried about the extra equipment and weight, by all means, take your DSLR!
Hope that helps!