If you can afford travel and want to, you should definitely think about it.
The issues with little babies are mainly about comfort. If baby cries before sleep, you'll want to choose a hotel with strong, thick walls, so you're not bothering others. If baby needs to be walked to sleep, as ours did, you'll need enough room in the room to do so. And so on.
For us, though we couldn't afford BIG travel, traveling was easy b/c we were so portable. Baby slept in the same bed as us (still does), baby was exclusively nursed until 11+ months old so we didn't have to worry about feeding him at all (just had to feed and water me!), didn't even own a stroller until he was 2 years old, just wore him in a variety of soft-cloth slings and mei tais, which are very packable. He went where we went, easily.
The only transportation issue is going to be cabs, b/c of differences in carseats. That would bug me, and in fact cab/carseat issues have kept us from some sorts of travel...when we went to downtown portland by train when DS was 3 months old, we walked to our hotel rather than take a cab b/c we didn't want to bring a carseat. That sure threw the Westin people, let me tell you!
Good luck!