OP, Its hard as adults to have just the right relationship with our parents and it just gets harder when there are other family members (like step parents 1/2 siblings, etc etc) Ultimately, each of you need to decide what type of relationship you going to have with your parent based on your individual history and then make that relationship happen. If current wife is making things hard on any of the kids, then the kids needs to have a one on one with current wife. As kids, unless you have mistreated your parent or been disrespectful to the current spouse, you have a right to time with your parent. However, this does not mean that the "terms" of the visit are 100% up to you. you need to work out some sort of compromise w/ the spouse who is the primary caregiver that is realistic.
It sounds like it would be detrimental to your father for him to be taken to brothers house. In this case, brother needs to clearly communicate this to step mom and find out what he has to do to spend time with his father.
I can tell you this. If I truly wanted to spend time with my father, NO ONE would stop me. I would find out what the terms of visitation are, and I would make it happen. However, keep in mind, everyone involved should have the same primary concern - what's best for dad.
A good friend told me once when I was explaining an uncomfortable position I was in with someone, "Well, 2disneyboys, if you knew for a fact that 6 months from today this person would be dead, what would you change about the next 6 months." That is such a simple statement, but it had impact for me. I realized I was so caught up in the history between me & this person that I was wasting our "current" and future. The reality is you don't know if someone has even 6 months left, much less more. Only you can decide how you want to spend what time is left.