My answer would have been (and has been)..."I'm so sorry, but I already have my travel plans made, and that is when we are going. Would you like me to cancel your room?"
I'm with you on this one, if Brother is going with you as your guest, he should be thankful for anytyhing and everything.
Pick what is best for you and your immediate family and others can come or not at their choice.