The answer to your question is YES it's worth it, but that is merely my opinion and everyone around here has their own.
First, your correct the party starts in the evening, but they will let you in at 4:00.
It seems as though everyone has their own reason for going to the party as well. Some love the parade, some love just taking in all the special things during the party, but for us since we're coming to celebrate our anniversary without kids I'm using it as a time to ride-hop til I'm silly while stopping for only cookies and hot chocolate. lol You could use the time to get there at 4:00 (hopefully right after her nap) then ride a bunch of the fantasyland rides. Grab a quick dinner, catch the parade and then plan accordingly as the time reaches her bedtime.
There are two times for the parade, so with a young child I would do the first parade and then call it a day if they were too tired to continue. If I could get them to sleep in a stroller, I'd probably wait to see the fireworks, shop or take turns riding rides.
It's just magical. The snow, the atmosphere, the parade, the cookies...etc.

Every year when I hang up my Disney ornament or get out my mini Mickey Mouse tree, I remember how much fun we had.
Also, if you search the threads for
MVMCP you will find a ton of useful info.
Good luck!!