4 months isn't a *bad* time to take a baby (as long as it's not a baby who cries all the time and doesn't sleep well), but 10-11 months is a WONDERFUL time to take a baby.
The weather in October and November is wonderful, and the crowds are low most of those weeks. A 10 month old will really enjoy a lot of the rides but not be so demanding as to make you have to schedule around the child as much as you might have to with an older toddler. You won't need to nurse a 10 month old as often or for as long, and he'll probably be able to nurse discreetly wherever you want to do it.
A lot of 3-4 month old babies would be easy to take to WDW, but you just don't know what kind of a baby you're going to get. If it's a baby with colic, you might not have as much fun. Nursing should be pretty easy by then, but some babies that age like to have nursing sessions that are 20-30 minutes, which means it would be hard to just do a quick feeding during a 3D movie or a ride on the People Mover. I've gone with a 5 month old, and we had a blast, but we planned the trip when he was about 3 months old after we knew what we'd likely be getting into. Plus he was my 5th baby, which is easier than your first.