When we went in December 2009, my husband told the CM when he booked that it was an anniversary trip. We got married in July. We wore our anniversary buttons though. The only time it got a little weird was when we were sitting down in Epcot and a family near us said something about coming during the Summer, and how December was such a better time, and my husband says, "Yeah, we came in July on our honeymoon and swore we'd never go during that time of the year again!" The other family didn't mention the fact that we were wearing anniversary buttons in December. So nice of them.
We're going down this year a week after my daughter's birthday, so I'll probably put a birthday button on her, as well as a first visit button, but I personally would feel weird going to celebrate my birthday if it wasn't on my actual birthday (though I'd wear the button during the whole trip!)