I think if I was by myself I'd keep it open. If I had another person (or persons) in the car then I'd roll the window up so we could talk privately.
Limo drivers have seen it all, and won't mind if the window is up. In fact they may prefer it so they don't have to listen to their clients bable!!