If there are upgrades available in the category you request, the cast member at the Upgrade desk will tell you which rooms are available, and after you select a room, the cost. (If all of the rooms are in the exact same category, such as 4B, they will have the same cost, but if, for instance, some are 4A and some are 4B, the costs will be different.)