Our policy has been to take the children for their fifth birthdays. It has worked out twice that that meant we brought a younger sibling along who was only one year old. The first time, I think the baby didn't get anything out of the experience at all. I wouldn't say bringing him along was specifically onerous -- it was surely a lot of extra work, but we had the advantage of four adults versus two children, so it was manageable. The second time was different. For starters, she's about twice the size as he was, and although I cannot say that she got all that much out of the trip, but we had a lot of fun watching her reactions to things -- something which didn't happen the first time.
So it basically boils down to "it depends" -- depends mostly on you, your family, and especially, the nature of the individual baby his/herself.