Melatonin doesn't work for me. I have the same issues OP. Melatonin simply gives me a headache, but doesn't make me sleepy at ALL.
I have found that the only OTC sleep aid that works for me is Unisom sleep tabs with doxylamine succinate (same active ingredient in NyQuil...it's an antihistamine). Maybe try those. No grogginess, no side effects the next day.
Being overtired makes it worse. Our last trip, I didn't sleep well the 2 nights leading up to the trip or the first 2 nights at the hotel. By day 3 of the vacation, I developed a horrible migraine, drugged myself up with meds (a muscle relaxer and a double dose of Aleve) and finally had a good night's sleep. I slept great the rest of the vacation.
Don't drink any caffeine if you can help it. Many people metabolize it VERY slowly and can affect your ability to fall asleep at night. Alcohol can also make things worse if you have a tendency to have insomnia. Alcohol makes my husband pass right out. Me? It makes me lie in bed, wide awake with a pounding heart and anxiety, even several hours after drinking. I drink, but keep it moderate, no more than 3 drinks a day, and I space them out on vacation. If I drink 3 drinks back to back a couple hours before bedtime, forget about sleeping.
I have also moved away from "the earliest possible flight time" and prefer to leave late morning or early afternoon. Early flights stress me out and make sleep impossible. If I don't have to set an alarm for a flight the next day and know I can sleep as late as I want, I sleep fine the day before travel.