If your family is big into breakfast, check out Cape May Cafe, over at the Beach Club. Its just a short walk from your hotel, and they have one of the best breakfast buffets on property!
As far as second guessing hotel decisions.... I feel your pain. Too many choices leads to indecision! I love the Boardwalk area. If you DO decide to change hotels, check out the Beach Club/Yacht Club hotels. They have a huge sand bottomed pool, and they're in the same area as the Boardwalk. Ive never stayed on property, planning my first on-prop stay in April (yay!), but I do have several friends that have stayed at both the Poly and Beach Club. They would pick Beach Club every time, if only for the pool and the close proximity to Epcot and the Boardwalk area. I know Beach Club also does Movies on the Beach at night, which your kids might really enjoy.