I put 7 or 8 on the assumption that I wouldn't be able to see my child. If I was close enough to see them, I'd probably be fine at 4 and up. In my experience, you can pretty much always find 2 middle seats behind each other, and if not someone stuck in a middle seat is usually willing to move to another middle seat. My biggest concern with my particular child at 4 - 6 would have been that he'd be too shy to ask to go to the bathroom, and have an accident.
I've been both "that mother" and the woman who refused to move. Once when DS was 4 we were late on a connecting flight. We had A tickets, but got there once the C's were boarding. As soon as the gate agent saw us she called down to the plane and asked them to find us seats together. When we got on the plane, there was a FA sitting in the aisle seat for the last pair of together seats telling people "I'm saving it for a mom with a small child". On the other hand, once I was on SW, when DS was about 5 and a mother got on with a child maybe 7 and was very upset that they couldn't sit together. The FA asked if anyone would give up their adjoining seats and I started to get up, and then thought -- wait, I'm going to volunteer to be separated from my 5 year old so they can sit together? And sat back down. Having said that, who is to say that my 5 year old wouldn't have done better than her 7 year old? Someone else did volunteer, so they were fine.
Having said that, if I knew that not paying the fee was likely to result in my being separated from DS10 on a flight over an hour I'd probably pay the fee. I go on vacation to spend time with my kid, and really treasure those moments when we're "captive" together, I even enjoy waiting in line with him. Our life is a little crazy, and so time to just talk, without needing to worry about getting homework done or other priorities, is precious. I wouldn't, however, deplane if we were denied the chance to sit together.