I think you'll be fine. I just returned from a week-long trip with my wife, her parents, and our daughter. One night, they went shopping at IKEA and I went to EPCOT solo. It was awesome and I never felt awkward at all. I had a blast and decided to eat at the Morocco quick-service eatery, which I'd never done. I took breaks and then walked really fast whenever I wanted. It was a different experience and not something I'd want to do for a whole week, but I think it would be great for a few days.
My wife just gave me the go-ahead to start planning my first-ever solo trip for late August or early September. I can't wait! It will just be a few days, but I plan to cram in a serious amount of touring. I think you'll have a blast! I'll likely avoid most of the kids' rides since we just spent a lot of time on them with our daughter, but it's cool either way.