We've had the most fun without our kids. We love JellyRolls, the Food & Wine Festival, all of the Epcot music groups etc. All things which bored our kids.
I've done both. The best time I had with my kids at wdw was when they were older teens. I remember thinking the first time we took them (they were 5 & 7), that we were enjoying much more than they were. Their favorite part was the swimming pool at our hotel during our afternoon breaks.
Now they're older and it's my dh & me. We have way more fun. I also went with a gf and that was a lot of fun too.
Someone said they had a friend who said wdw isn't for kids (or something like that). She may be right