I don't think you are being selfish for wanting a family vacation but I do think you can be more flexible about it. And, I can totally relate to your pain about planning a trip and trying to get schedules to match up. You are definitely correct in the grand scheme of things missing 3 days of school in 1st and 4th grade really won't have a huge effect on the rest of their lives. I totally agree with that.
However, being a parent is tough these days especially when it relates to school, sports, religion and/or work obligations. I can't tell you how hard it is to plan a trip. As your grandchildren age it will get harder. Then, when you tie in the emotional family time, it makes the decision even more difficult. Especially when you read the post above with the person with cancer. But one thing I've learned over the years and my kids are now in 6th and 10th grade that life isn't fair and sometimes we have to make scarfices.
We took my oldest out of school in 2nd grade for a week to go to WDW. It was in May it was a great time to go especially when you think of the savings with airfare. But I gotta tell you it was miserable school wise. You wouldn't think you'd be missing much at the end of the school year. We had filled out the forms to make up the missed worked. This was a packet of all the things that they did in class all week (my kid was in school 5 1/2 hours day and it was a lot of work). I'd say the packet was about 2 inches thick. So, our intentions were to spend the mornings in the park and come back to the room work on homework for 1 hour and then, go back to the park in the evenings. Well, with the 3 hour time difference and traveling taking all your energy it did not work that way. My kids were exhausted from all the walking. In 2nd grade you are too old for a stroller but when you spend all day in the park walking from the room to the bus stop to the park then, back to the bus stop to the room and then, back to the bus stop to the park and then, finally you walk back to the bus stop back to the room. We were staying at BCV so, we could walk to Epcot but then, Epcot was huge. Anyways, my point is that my kid was so exhausted and hot that when we came back to the room she didn't want to do homework she wanted to take a nap or swim. We never did the homework and attempted on the plane ride home and got through maybe half of it. Then, when we got back to home we had to spend the rest of the week finishing the packet and doing all the other crazy things that happen in May like open house, swim practice everyday, etc... Then, my daughter had to spend her recesses inside doing make up work that they didn't send with us. These were things that could only be done in the classroom environment. Finally, the last straw was when she informed me that she missed a field trip while we were gone. I know you think what is the big deal it is just a field trip. Every since that trip I do not miss school for vacation except maybe a day here or there. Also, learned that vacationing at Disney is exhausting and not always the best during school time. But that's our family and every family is different.
While we still love Disney we find other vacations are just as great. Maybe you could plan on renting a house off of VRBO and do something in the middle of everyone. You could have fun hanging out together. We own timeshares in Hawaii and we get our best family time on trips when we go to Hawaii because all we do is sit around and veg. It isn't all go, go.... Or we go somewhere local like Lake Tahoe and have just as much fun.
Just a different perspective and I do hope it works out that you have the best family vacation ever.