I would absolutely go! My favorite trips with DD were when she was toddler/preschooler. Assuming they aren't scared of characters, at those ages, my DD's favorite thing was just visiting all the characters. I think we had one day at MK that she basically wanted to go from character to character and we hardly rode any rides, and she called it the best day ever. There are a lot of rides that seat 3. Yes, assuming the 4 year old is tall enough and wants to go on the thrill rides, that is an issue, but there is plenty to do without the few rides that the 2 year old will be too short for, or that won't fit 3. Plus, I will say that just because a 4 year old is tall enough for the thrill rides, doesn't even mean they will want to do them. Mine loved most of those rides when she was tall enough to ride them at age 3 and wanted to do them again, and then a couple of years later she refused to ride any of the thrill rides she loved before because they were now scary.
If it were me, since you are staying at the Contemporary and will have monorail access, I would be thinking about both MK and EPCOT if you have 3 days to spend in the parks. You definitely could spend all the time enjoying MK, but EPCOT has Nemo, Frozen, Living with the Land, Figment, lots of characters (if they have all returned) etc.