Keep in mind that if you have enough points for four nights at the Dolphin, you actually have five, as the fifth night is free.
I personally much prefer the Swan to the Dolphin for many of the reasons listed above, but not sure I would pay the points premium needed. I value each Starpoint at 2.5 cents each. So to stay at the Swan over the Dolphin, you are basically spending an additional $50 worth of points per night (and actually much more than that if you are visiting during "peak" season).
All that said, although there is usually much more of a convention/business vibe to the S&D, I've never quite seen the raucous scene described by the OP. Given the points required for free nights at each property, I'd probably choose the Dolphin again and hope for the best.