I guess technically you could be dropped off at Cabana Bay. Although they might want to see a room key along that path.
Our last trip was offsite and I hate buses. I didn't find the VB buses to be a problem. There was one waiting when we got there and more buses waiting to load people. So, buses were waiting on us and not the other way around. We got to the parking garage before EA, and they sent us to a different VB-only area of the parking garage. There was two bus loading areas within the garage. Those with early entry and those without. I assume you'll have early entry since your pass is at least Preferred.
ETA: this experience was last August, so post-lockdown. If they had their act together then, it should still be fine.