Raised my own 4 plus helped raise two of my husband's 4. Mine are 32, 29, 28 and 26. Now we're raising my husband's granddaughter (we're her legal guardians), Isabella, who is now 5! She's a delight and is keeping me young. My "kids" adore her and she's wild about them too. They also laugh and say Isabella is taking the pressure off them to produce grandchildren right away.
I always thought I'd have six but first husband (and HIS MOTHER) decided otherwise and he had vasectomy after #4! Can you imagine why we're not married anymore?
I think the right number for you (and your spouse) is the right number whether its 0 or 12. Personally, I thought the third one was the toughest. Suddenly more kids than hands or eyes or parents. After that #4 was easy.