So far, we still believe we need the stroller for our youngest (4). I've seen posts about people considering not needing one by 4, but honestly, I prefer having the stroller for the same reason you've said here. We went in early and were wearing jackets and the kids, even had pants on some mornings. But within 2-3 hours or so, it was warm enough we had to peel off those layers. That's a lot of junk to carry around -- in addition to the backpack! With that being said, we do try to travel lightly into the parks. Juice boxes or water bottles, a few prepackaged snacks, say cheez-its, napkins, band-aids and aspirin. That's about all we have in our "Survival kit" along with camera, cell phone, tickets and maps/showtime guides.
But when we'd go in at night after our break, we found we didn't need to bring in as much stuff -- the kids weren't changing their clothes as the weather was at a constant.
I would say, go ahead and try a day without it but if it's too much of a pain, go ahead and use it. You don't want to be miserable b/c of all the stuff you feel you have to lug around. But if you try to go into the parks light, you just may find you like that too -- no need to waste time parking the stroller, navigating crowds with it and then backtracking and searching for it after each ride. After awhile, that's all kind of time consuming.