Honestly, DS was 6 when we went and had a great time eating in the castle. He wasn't overly thrilled with the princesses but loved the food and the little sword he got.

There is honestly nothing girly, IMO, about the castle interior itself. The princesses are the only girly thing.
I'd say if you want to do it, go for it! This trip we will be opting not to return to CRT due to the cost and instead will do Akershus for breakfast. DS will get his Chef Mickey's experience, so I think all will be happy (and mommy and daddy will have saved a chunk of change by doing two essentially for the price of CRT).