It's more complicated than that and you're misrepresenting the situation.
If you remember the rescheduling of the Olympics went back and forth between ...no..yes..maybe..ok yes we're going to do it but now we need to figure out if it has be at the end of the year of within a year and the IOC was resistant to rescheduling it or canceling it. AND a big reason why it was rescheduled is countries were going to stop their athletes from going. Hard to have Olympics without athletes. I don't think we should make everything out to be "well in the US we wouldn't get our act together" without realizing other countries aren't some benevolent entities either. If Japan still had all the countries' approval for their athletes to still go they may have waffled more.
For a bit more information in-depth (because I was typing off of memory and paraphrasing) you're welcome to read a bit more about the rescheduling of the Olympics:
https://en.wikipedia.org/wiki/2020_Summer_Olympics