I would not plan on using water from the plane. If they don't give you bottled water, then you might be stuck with using water from the plane's tanks and there is no certainty that that water is "clean," particularly for a baby.
Take a couple extra containers of liquid formula, if a screener hassles you, ask for a supervisor and then explain that you are just being prepared for the possibility of a flight delay. Or, if you don't trust TSA to make a reasonable decision, once through the security process get "city" water from the water fountain at the airport.