I have taken kids at 5, 3, and infant age before. We go every year, starting when the oldest were 2 and 8 months, respectively. This year we are going with an 8, 6, 3, and 3 month old. We have always had a great time...but, if you can't manage the kids in a restaurant there is no way I would take them to Disney. We take our kids out at least once a week to a restaurant or somewhere they need to behave. We are all used to it. If you already think of it as overwhelming, it probably will be. Why not wait until they are potty trained?
If you do go, PLAN. Get to parks before rope drop, go to the most recommended parks, make ADRs six months in advance, and take afternoon breaks. Make a detailed packing list. Pack kid's outfits individually in plastic bags to make getting ready easier and always have a spare in your diaper bag. Pick strollers that are easy to collapse for the bus. Understand you will need a lot of breaks and while you need a good plan of attack (FP+, ride order, etc), that it will not go according to plan and your kids may hate characters or be afraid of rides.
Why not plan a trip for a year or year and a half from now? Disney takes a while to plan anyway. At that age, your kids will be able to understand things like standing in line better.