Does it annoy you simply that others feel this way or is it more because people may appear to be judging you for enjoying Disney as an adult/without the kids/etc.?
I don't mind that other people don't enjoy Disney as adults. (Hey, more room for the rest of us!) And when we took DS4 to WDW for the first time, we did make it all about him...we knew we would enjoy everything he wanted to do, and there was no sense in pushing him to total exhaustion/crank-pants time just for the sake of doing MORE things that we had already done on previous kid-free trips. It was totally magical, and I wouldn't have wanted it any other way. However, I have been to Disney more time as an adult than I ever went as a kid, so I'm definitely in the "Disney is fun for all ages" camp! And it *is* annoying if someone makes snide comments to you about something you enjoy just because they don't share your feelings, so I can understand that.