We have 3, twins and a younger daughter. It is certainly much more difficult/expensive to travel with a family of 5. I did not realize this until several years after having the third child. The thought of travelling honestly never crossed my mind when having those children. I did think about it when contemplating whether or not to have a fourth child. We decided against but not purely or even mostly for that reason!
I love travelling and have been to Africa, Israel, many places in Europe and the U.S. But most of my foreign travel I did before or without children (I went on two trips with my best friend who is single & has no kids). My thinking is that, say if I'm in Europe, I like to see everything & walk everywhere & go nonstop. The kids (and DH, to be honest) are not up for that. So it's better for me to go without them than to drag them along or to go but do less and nobody's happy. They have the rest of their lives to travel and explore; they don't have to do it all before becoming teenagers-- I didn't.
Still we travel quite a bit:
Overnight trips we took with twins before #3 was born (we lived in Philadelphia and then Miami): Ireland, Boston/New England, MD, Philadelphia, CA, WDW (6 times on the FL resident annual pass), NYC
Overnight trips with all 3 (we live in VT): England (twice), WDW, CA (4 times), Montreal (twice), Boston (numerous), NYC (6 times), MD(numerous), Maine (twice), New York State (twice)
One thing we've done that we've really enjoyed is to trade houses through Intervac. You can look for other families with the same number of kids. You have more room and it's way cheaper than hotels. But you are basically in one spot; you aren't roaming all over the country or continent.
We also camp, stay with friends or family, and look for hotel deals-- I've never paid more than $120 per night except at WDW. A couple of times work has paid for my flight and the hotel room.