Having stayed at both Contemporary and Polynesian, I've never found that sharing buses really adds to transportation time (as compared with, for example, multiple internal bus stops at Moderate resorts). On paper, it would seem like it should take longer, but my actual experiences haven't worked out that way.
That being said, most on-site Disney veterans will agree that Downtown Disney transportation in general takes longer than park transportation, to/from ANY resort. Don't know why, it's all supposed to be on roughly the same schedule, but few experienced Disney folks will argue that DTD bus service isn't slower than park bus service.