I hate to tell you this but the only way they are hypoallergenic is if they are wrapped in plastic with holes for them to breathe. Your brother is probably used to them which is why he doesn't react. As for the cat box, I am sorry but all litter boxes smell. Cat pee and poop is a distinct smell and if you are used to it you won't smell it. Also, nobody is going to tell you that your house stinks like animal. I am not saying you are dirty, I am just saying that non cat people will know the smell.
I have a friend that HATES cats. She came over one day after I had the cat for about a month and said Hey, I thought you guys got a cat. I said we did. She couldn't believe that she couldn't smell anything. So I disagree with you.