We took our girls last sept at ages 4.5 and 2.5. They both LOVED it! The 2 year old sat on my lap, and I whispered things to her about the show while it was happening (to help her "get it"). SHe wore the glasses, kept them on just fine, and enjoyed trying to catch Jasmine's jewels (among other things) when they floated in front of her.
If they don't wear the glasses, I would think they would be pretty confused, as the movie will look rather goofy.