Thankfully, I've never experienced server peevishness, and I'm usually the solo diner at the four-top. Doesn't bother me, I go where I'm seated. If anything, it should teach WDW to have more small tables. I'm usually in and out pretty quickly, unless I get preoccupied with my book or with people watching entertainment.
I have, however, experienced parental entitlement

that goes something like this, "It's not faaaiiiirrrr that she's taking up a whole table while fambuhlees with chiiiiyyylldren are waiting." My response is usually along the lines of, "Wanna see how slowly I can really eat, Fertilla? From the looks of you, it wouldn't hurt you or your brood to skip a few meals."
I'm glad that as a solo diner I am not the only one to experience this. The first time I ate at Jiko, I made my ADR for one, but was seated at a four topper. Not my fault, I went where they took me and had a very nice table right by the window. However, you should have seen the dirty looks I got from some of the families that were seated in the middle. As I was taking my seat, I even heard one lady make a comment about why was I allowed to sit right next to the window at that big table, while they had kids and were stuck in the middle. She said I should have been seated at the bar.
One time when Tusker House was still a CS, I looked around for a smaller table, but none was available, so I took a seat at a bigger table. I had just sat down and was spreading my napkin and silverware out, when a lady suddenly appeared and actually ordered me to move, saying she needed that table for her family and since I was alone, I could eat outside. She didn't ask, she just said, "move, I need this table and you are by yourself, so you can eat outside". Excuse me!

If she had asked I would have gladly shared my table, but no one bullies me. I said, "I don't think so", and saw a couple coming just behind her who was looking for a table, smiled at them, and asked them if they'd like to share my table. The first woman huffed loudly and stomped off and I had a very pleasant lunch with an older couple from Pennsylvania. At that point, even if I hadn't spotted someone to share the table with me, I wouldn't have moved, just on principle. Being a larger group and having kids does not give you any special rights. Should solos not be allowed to dine at a TS restaurant? After all they do not ever have actual tables for one and very few tables for two.
To the OP: book the table for four, let your kids have a good meal, while you and husband order a soda or something, then go enjoy your special couple's meal. As long as it's not a buffet or some other all you can eat meal, it's not a problem, and no one should look at you twice. Again, as a solo dinier I am more often times than not seated at a four topper. I just sit where they place me.