No, you shouldn't have any additional delay just because of ordering XL. When I checked Epcot to Disney Springs just to get an idea, the pickup and dropoff times were identical.
Riders are not the only consideration. I always order XL for DD, DW and myself if we have luggage.
XL is either an SUV with a capacity of 6 or more riders, or a minivan with that capacity. The fares look like they are about 30-ish% higher for XL. The Epcot - Disney Springs fares were $9.82 for X and $13.00 for XL.
That's actually a bargain; here in Miami, XL fares are more than double X -- like 2.1-2.2 times more. A $9.82 X fare would be a $20+ XL here.