Very late changes (e.g. < 7 days) are usually hit with the change fee, no matter the "direction" of the change.
Slightly less late changes (45 - ~7 days out) usually get it waived if upgrading (but NOT all the time!) and charged if downgrading. Been this way for awhile, but the vast majority of late changes we see around here are between the 45 and 7(ish) day mark, so they don't get reported too often (I've seen a few cases of said late change, and every one was at least potentially charged. A few, including the OP, did manage to get it waived, but more than often it was applied).