HP was my only reason for visiting Islands, it was the only reason we even bothered to leave Disney property, so I say, YES! YES, it is worth it!!! Be there about 45 minutes before the park opens, which means leave your Disney hotel by 7:45am. (If you need coffee, there is a Starbucks at Citywalk on the way. Send one member of your party to retrieve coffee, the other go get in line.) I cannot emphasize this statement enough: DO NOT BRING ANYTHING TO ISLANDS IN WITH YOU THAT CANNOT BE WRAPPED AROUND YOU! If you have a waistpack, bring it. If you have a backpack, leave it. Trust me. That tip alone will save you 20-30 minutes trying to get on the FJ.
We arrived at 8:15am on a Thursday, Sept. 30th. There were maybe 30 people in line in front of us. By 8:45 there were a couple hundred behind us. We walked quickly, but orderly, through Seussland and headed straight for the castle. With nothing to dump in a locker, (and they will make you dump you things into a locker so stowe that camera and your phone in your waist pack!), we walked straight through the castle and on to the ride. By the time we exited the castle the wait was at 20 minutes, all from the people that had previously been waiting for lockers. We walked straight to Ollivanders and got in line and the wait was already at 30 minutes. This was at 9:15 in the morning.
The wait for Dragon Challenge the 3 hours we spent just in Hogsmeade was never over 15 minutes and hovered around 10 minutes the majority of the time. Unless you have a small child, do not wait 30 minutes in a line for Flight of the Hippogriff. Seeing Buckbeak and Hagrid's Hut are great, but the ride only lasts like 30 seconds and isn't worth it.
By 11am, the wait at the Butterbeer cart was 20 minutes and the wait for Ollivanders was well over an hour and stretched back to the area where the choir preforms.
There was no wait for a table in the Three Broomsticks though, we walked right in, ordered, and ate at noon-ish. By that time, Hogsmeade was extremely crowded. Honeydukes and Zonkos were a nightmare.
Hope this info helps!
P.S. The cost of the cab from the Polynesian to Citywalk was $47 one way for the two of us.