I highly recommend going that day if it works for you - we went on the CM party day earlier this year and it was nice and quiet. Wait times weren't much more than the AH party we went to. The day went fast being so short, and it was kind of weird leaving at 4:30pm, but I would do it again in a heartbeat! We had splurged on a TP view room at the Poly since it was our honeymoon, so we just grabbed some dinner and then went to our room to watch the fireworks, but you could just as easily grab dinner and then watch the fireworks from the beach. Otherwise if you've got park hopper tickets you could head to EPCOT or DHS for the evening.