Buses going to/coming from Magic Kingdom and Downtown Disney (these are 2 separate buses) start at Martinique, and then go clockwise to Old Port Royale, Trinidad North, Trinidad South, Jamaica, Aruba, and then finally Barbados before heading to the destination.
Buses going to/coming from Epcot, Disney's Hollywood Studios and Animal Kingdom (these are 3 separate buses) start at Jamaica, and then go clockwise to Aruba, Barbados, Martinique, Old Port Royale, Trinidad North, and finally Trinidad South before heading to the destination.
No, it doesn't take forever. The bus only holds a certain number of people. Whether they all get on/get off at one stop or seven stops, you still have to wait for them to get on and get off. The additional stops add something like 3 or 4 total minutes to the operation.