My husband and I don't have kids yet and we have gone to several character meals. As others have said, we've seen other tables of just adults there too. Go and have fun!
My only caution is that sometimes depending on the CMs that are working that day, the characters might not interact with you as much as you might like. We've had great experiences, but we've also had experiences where characters that were lively at other tables with kids came to our table, took a picture, and quickly moved on without interacting with us at all. I hate it when they do that because obviously if we're paying to eat there, we want them to play with us - usually character meals aren't regarded as the best food on property, so we aren't there just for that! But I just recommend going into it knowing this and ready to start interactions with the characters (ask them questions about their story, compliment their clothes, etc.) to show them that you're fun and want them to play with you.