I highly recommend the GA Aquarium, however, I would bypass the Deepo show. It is a cheap rip off of Nemo, and costs $15 for a 6 minute movie... And that's it. A total rip off, IMO, but the aquarium itself is SO worth seeing!
I would bypass the World of Coke unless you are a HUGE Coke fan. It is expensive (around $30/person last time I was there) and the only truly fun part, for me anyway, is tasting the international sodas which is something you can do in Epcot free of charge, other than park admission.
Check out Lenox Square or Atlantic Station if you want to do some shopping! Oh, and you can't forget Chick-Fil-A if you don't have them where you live.
The Aquarium is a great choice, as is the World of Coke. You can get the Pemberton Place card that allows you access to both for a reasonable price. The World of Coke is $15 for adults, but there are tickets available on My Coke Rewards, we just used those last fall, and they don't require that many points.
If you are only going to have a day or 2 to explore, I wouldn't go for the City Pass, it isn't worth it, especially if you aren't into the sort of choices it gives.
The Atlanta Zoo is really nice and they have quite a few babies that were born earlier this year, including a baby giraffe that was born in the last week or so. We spent at least 2/3 of a day there.
We have been wanting to check out the Fernbank Museum and they have an IMAX theatre. They have Martinis and IMAX on Friday evenings for adults only, DH and I are going to try that one evening.
We didn't care for Stone Mountain, it was really expensive and there just wasn't enough to keep me interested for the price we paid. And I am scared of heights, but rode the cart thing to the top, that was beyond scary.
Atlantic Station is a nice place to go shop or have a nice meal. I want to see the Bodies exhibit, but that is probably not for children and my DH doesn't want to see it, so I guess I won't be seeing it either.