Your correct in that everyone's taxes are paid to educate all children. I don't see a problem with allowing for the choice to choose how or where those taxes pay for that education. The child gets educated and we're all still paying taxes, what changed?
A few things. So bear with me if this is long.
First, choice in any form diverts dollars from public schools which in turn reduces the available funding for educational expenditures. If a school's fixed expenses are 15% of the operating budget with 500 students, they'll eat up 30% of the total funding if enrollment falls to 250. Because those expenses - building and grounds upkeep, busing, etc. - are hard or impossible to cut, it is education and enrichment that suffer.
Second, some people have a philosophical or moral objection to public funding for religious schools, who would be the biggest recipients of voucher dollars. In my state, any voucher plan would have to clear the hurdle of amending the state constitution because right now it includes a prohibition on state funding for religious schools or organizations.
There is a similar issue at play with charters, only instead of it being funding religious concerns it is a question of whether public educational dollars should be going to profit-seeking enterprises that may prioritize investors' interests over those of the students.
Third, no existing voucher plan has proposed an amount that would fully cover the cost of private school. That makes sense, since public school per-pupil funding is generally lower than private school tuition. But it means that for a family to benefit from school vouchers they would have to have the means to make up the difference. It amounts to welfare for the middle class and fails to address the aspect of education most in need of improvement - the intersection of poverty and school quality.
Fourth, with the exception of choice between public districts, most school choice plans are to some degree discriminatory. The choices they make available are exempt from special ed and ESL mandates, so families of special needs or non-English-speaking children are excluded from the options available to their healthy, native-born peers. This further cripples the public schools, who are left with higher-than-expected percentages of students who require expensive educational supports and who are likely to struggle with performance on the all-important standardized tests that define public schools' success.
