We use the Swan's buses to go to Magic Kingdom and Animal Kingdom all the time. Never had a problem whatsoever. There has never been a mob waiting for either of the buses, the waiting area is shaded with some benches (not a bunch but we've always been able to get one all to ourselves), and they have TVs to keep you occupied (that annoying "seven things to see at Disney World").
We have had the luck to have every single bus EXCEPT the one we are waiting for come through before ours. But we've never had two for another park come before ours did. And we've never had to stand on a bus.
They do share with the Dolphin and the Boardwalk. Sometimes the Boardwalk is first, sometimes the Swan is first.