We took our son on his first
ABD at four years old. My husband had been dying to go to Greece and this was the only way we thought it would be possible! My son absolutely loved it! He participated in every activity except the Donkey Ride and kept up beautifully with the group and all his fellow Jr. Adventurers. Since then, we have been on 3 other trips (Ireland, Winter Wonderland, Cowboys and Canyons) and have Knights and Lights and Vietnam booked. It has been wonderful and we have created some of our best family memories.
As for younger kids getting something out of it, his memories of all the trips are fuzzy. He saw a picture of us kayaking in Crete a week ago and asked what trip that was. So, it's clear he did not retain a lot of what he learned when he was there.
However, he does remember the names of all of his guides and most of the Jr. Adventurers whom he considers lifelong friends. I think he got a tremendous amount out of it. He has also become a seasoned traveler and feels comfortable around many different kinds of people and cultures. However, if you ask him what his favorite thing was on any ABD trip, he will say Jr. Adventurers night.
Taking a younger child does mean that you have to enter the trip with a different attitude. The guides are not babysitters and neither are the other kids on the trip. We were always careful to keep our son with us until it was absolutely clear that it was appropriate to send him with the kids/guides.
You also can't have the expectation that they must learn something to appreciate the cultural/historical aspect of the trips because for my son at least, that is not overt. However, it's clear to me that those experiences have been very positive for him in the way he relates to the world and other people.
So, to sum up a long post, JUST DO IT if you think you and your family would enjoy the trip. For our family, it's been worth every penny we've spent.