We took our daughter (only one at the time) when she was 1.5 years and everyone roasted us for it. I am so glad we did it because she LOVED it! We were so happy watching her wonderment and awe, it was simply awesome! She wasn't scared of the characters (I realize this can vary by child), she liked everything she rode which was basically everything except the rides she was too short for, she LOVED the parade during the day. I was so bummed we couldn't get her to the night parade, but we went in April and it wasn't until 9pm, and although I hoped we could drag her back to the park for it, she wasn't sleeping well or napping well, so it just wasn't happening. The hardest thing for us was : We had a room at the Wilderness Lodge, and she didn't do well falling asleep in the pack and play while we were in the room.... we would try to go on the balcony to leave her alone, but it took her a long time to fall asleep in there. She was definitely tired during the trip. We said that next time we would be looking to have a place with a common area and a bedroom separately that we could close the door to while she fell asleep By the way, she also loved the KidCot stations at Epcot. It was the only thing that got us through the countries, otherwise she wouldn't have stood for that part!
Do not hesitate! All these people said the same things to us, my MIL thought we were crazy and said so to my face. Our thought was, she was free to fly, free to stay, free to eat, free to go to the parks.... it was awesome and I wouldn't change it for the world. Great memories and we talk with her even still about "Mickey's house!" I can't wait to take her back! We have memories for a lifetime and we have pictures to show her since I'm sure she won't remember. If we were going to go SOMEWHERE on vacation... why not Disney when she was free anyway? She wouldn't remember ANYWHERE we took her that year on vaca!
Good luck, ENJOY, and relish the moment when you come back and tell everyone how magical it was for your little one, and how WRONG they were!