I've taken my son to DHS every trip, but only once for a full day. It actually has a lot for Toddlers. The park simply is just low on attractions period, but about half of them are toddler-appropriate.
2yo - we hopped there just to see the Frozen Sing Along (he loved the music) and just to take his picture on the Speeder Bike while my wife and I went on Star Tours.
3yo - Full day at DHS - Disney Junior Live! Toy Story Midway Mania (he could have rode that several times), Muppets 3D, Little Mermaid, Frozen Sing Along, Frozen parade (not there now), make a Lightsaber at Tatooine Traders, Honey I Shrunk the Playground (now gone), Disney Junior Character meal (and meet and greets). He enjoyed eating at Pizza Planet (he was big into Toy Story at the time). He opted out of Fantasmic because he thought it may be too loud and scary. His 4yo cousin even got a Elsa/Anna makeover since it was the "Frozen Summer Fun" summer.
4yo - Half day at DHS -- Star Wars focused. Star Tours (a little scary for him), Toy Story, Star Wars Launch Bay, see the Star Wars Galaxy Far Far Away show, watch the march of the Fist Order parade, make a Droid at Tatooine Traders. We left at lunch and hopped to MK.
Basically because "I" love DHS we go there. There are things we even wanted to do but we just didn't have time: eating at Sci-Fi Drive In, Indiana Jones and Beauty and the Beast. Like others have said. Because of all the timing of the shows, it really is hard to fit a lot in.
We've opted out of Great Movie Ride simply because we think the Alien scene may be too scary and the line is usually long and we use our Fast Pass on the superior Toy Story.
Personally I think it is worth some time there with a Toddler because of:
Disney Junior Live!
Toy Story Midway Mania
Little Mermaid show
And the wide variety of Meet and Greets (Disney Junior, Buzz/Woody, Olaf, Mickey, Minnie, all the star Wars characters). My son is not into M&Gs (too scary), but he enjoys looking at them from afar.
And because the park is PHYSICALLY small right now and there is no reason to walk down Sunset Avenue (Tower of Terror/Rock n Roller Coaster) .. the amount of walking you do is practically nothing.