I think flight delays are more common later in the day, as opposed to earlier in the day. When making the decision, assume that the flight will leave late. Maybe you will find out about a delay while you are still relaxing at the hotel, but maybe you will find out about a delay when you are already at the airport. Will your group do well hanging out at the airport for a long time? Will they do well if they arrive home late at night?
If someone is picking you up at the airport, then a changing arrival time could be annoying. If you left your car at the airport, then this isn't an issue. If you will take Uber/Lyft/cab home from the airport, then you need to figure out how hard they would be to get if your flight arrives later than scheduled.
What are the flight alternatives? If the only options are 8:00 pm and 6:00 am, for example, you might find that the earlier flight would not work based on the personalities of the people in your group. But if there are other flights which aren't "crack of dawn" early, that's a different story.