Just in case HMerritt is still reading, I'd like to offer some general advice (just in case you have not heard it before.)
Read your custody order over and over again, until you are sure you understand exactly what it orders you and your Ex to do. If anything is unclear, write it down.
For instance:
Is there out of state notification for travel, and how much time is needed?
Is contact with the other parent dictated, and how often? (Are you both ordered to give phone access to the other parent, and how often?)
Is there clarity on how "skipped" weekends and weeks are handled?
Is there clarity on how changes to the custody schedule are handled?
If your Ex is violating any of the conditions of the order, document, document, document, and then send a RRR letter to him informing him of the exact violation. Taking him to court over one or two violations would likely do nothing but irritate a judge (even in Dad-hostile states like Georgia).
If he is skipping any of his time with the kids, document that too.
Once you have all this, and any required time has elapsed (most states require a certain ammount of time after a custody order) go back to court for a modification. Ask for specifics, and give clear reasons for those specifics. (Examples: Ask for court ordered 20 minutes of phone time between 8pm and 10pm every night, because between June 10th and June 25th 2011, Ex Hubby refused contact with the children. Ask for $75/month extra support because between January 1 and July 1 2011, ex hubby missed 14 days of ordered custody, and you cared for the children.)
Another thing, since it sounds like the Ex is still getting under your skin, make sure all communication between you is via text or email. Any drop-off/pick-up converstaions need to be followed by an email, or it never happened. If you both agree verbally to swap summer weeks, and he takes the kids for your week, he can turn around and keep them for his, claiming there was no agreement. And if you refuse, then you are in violation of the order, not him.. not fair, but it happens.
You said you let him have extra time for his Disney trip. It sounds like you are trying very hard not to let the kids be negatively impacted by your divorce. If he plays nice as well, then your life will be a lot easier. But if he does not play nice, then document, record, and play by the rules, and your kids will learn a lot about how not to be pushed around.
I hope you have the BEST. DISNEY. VACATION. EVER.