The absolute best place for breakfast is Signe's, IMHO. It's a small, local place. You go up to the counter to order, but they bring your food out to you. They don't have alcoholic beverages, in case you were looking forward to a mimosa or bloody Mary.

Depending on when you go, there could be a very long line, as well. I've heard stories of people waiting for an hour to get in! But it's worth it! (We usually go in the off season so that we don't have to wait as long)
If you're looking for a more traditional brunch buffet, we've done the one at the Westin. You don't have to get really dressed up for it, but I don't think that swimwear or flip-flops are allowed. I believe there is also a brunch like this at the Marriott and perhaps the Crowne Plaza and/or Holiday Inn.
I have heard that the Sunday brunch at the Old Fort Pub is great. For Easter this year, we went to the Ocean Grille (right across the marina from the resort). I believe they are both table service (order from a menu) and require a more dressy attire.
I'm sure there are lots more options. Be sure to pick up a copy of the restaurant guide with the menus in it!