The last transportation guide I have gives transferring at Downtown Disney as the primary method of getting from one resort to another. (A newer, different guide may since have been published.)
I would have to consider this long a trip time outside of crowded times worthy of a recoverable complaint unless there is published literature that suggests that this trip time is to be expected.