My opinion:
For men, flip-flops are fine during the day but are too casual for the dining room. The cost of the flip-flops is not an issue; you might have a very expensive bathing suit, but it's not appropriate for dinner either. For those of you who are into complete comfort, I'd suggest Topsiders or leather boat shoes (thinking casual nights here). My hubbie wears them all the time without socks. For formal nights, if you're wearing a suit, guys, you're pretty much stuck with the dress shoes.
For women, flip-flops run a wide range. Obviously, the plastic wear-to-the-pool type is out. But a pair of dressy leather flip-flops or a pair of colored flip-flops with beads might be just right with a sundress for casual night. Black with rhinestones might do for formal night. And, of course, ladies can get away with just about anything if they're wearing a long dress on formal night. Don't be jealous guys -- we ladies can get away with this one little thing, but our wardrobes are more complicated in general!