It really depends on your expectations and your family's parenting/family style.
In our case, we wouldn't dream of not going and/or not taking our children at any age. Our 4 children have grown up visiting
Disneyland since they were newborns, and we're making our second trip to WDW this summer. Particularly with children the age of your older child, WDW will be as special to them as you make it

. We have always emphasized that Disney parks are a very, very special place, and they have grown up appreciating it from the time of their earliest memories. They love the family traditions and excitement that surrounds Disney visits and don't remember time when Disney wasn't a big part of their lives.
Of course, with young children, the expectations for what you can accomplish are different, as you need to build in eating and rest time in a way that you don't when they're older, you'll need to do "child switch" on some rides or forego some rides, etc. But if you decide to go, you can build very special memories. As they grow, your children will love looking back at the photos of themselves with Mickey, Tigger, or in front of the castle when they were wee little ones...I have made wonderful montages of the children at varied ages in front of the same places or with the same characters.
Good luck on your decision.