If you are realistic about YOUR child and what you want to accomplish, you will have a great time. I, too, have a difficult (DH calls him Sybil as he seems to have multiple personalities at times), with health issues, 3yo right now, in addition to my 2yo and 6yo. We go every year--NEXT WEEK!
We go at our kids pace. We let them call the shots, within reason. We do not go to the parks early, it is much better to let them wake up on their own and set the pace. We leave the parks when it is obvious they are done for the day. We find our kids are not better off going to the hotel room for a nap, but taking in-park breaks. One of our favorites is a late lunch/early dinner. It works miracles. We do not go at peek season.
My oldest who quit napping at two, naps at WDW. We'll see this year though since he's almost 7. Luckily, he's thin though and can still go in the stroller easy, weight-wise.
And remember, kids have melt downs. My oldest is the easiest, most laid back child ever. I can count the number of melt downs he has had since he was born on one hand with fingers to spare. Of course, there are people at WDW when he was 4 who probably are saying "You shouldn't bring young kids to WDW because..." and talking about him. It was a snapshot in time and has nothing to do with the vacation. Then, back to DS3, he has multiple break downs every day. It does not matter where, when, or what is going on. He would have that melt down anywhere he was at that moment in time. I'm not giving him the power to dictate the rest of our lives--I'll just remove from as many other people as possible, same as at home. He is actually much calmer at WDW because of how much is going on he forgets to be himself.
I love the memories I have. My kids remember more than I think they do. And once DS3 went at 15 months, he was, and still is, in love with all things Disney. My favorite ages to take them are as older infants, three and four. I would never leave a child behind on a family vacation--it would not be a family vacation then and my heart would ache too much. My mom left me behind when I was two with my grandmother. My mom says all she kept saying is "M-- would love this." They went back the next day with me.
Make whatever decision that you think will work for you, your spouse, and your kids. If anyone else says anything, just say, "If you have nothing nice to say, say nothing at all" or "Thanks for your opinion, but our minds are made up" and SMILE.