Yes, that is the best way to do it. You can also go to Downtown Disney next to DisneyQuest to do it, but last time I was there to do that (a few months ago) their Premiere Passport printer wasn't working... so yes, the Magic Kingdom entrance is the best one.
It's really simple and just takes a few minutes. Just pay the cost difference between the two passes and you'll get the new one.
Then, add your passholder number to the My Disney Experience app right away and it can be used immediately for MyMagic+ stuff as well.
Ethan