This happened to me years ago on a flight to NJ from Atlanta. We boarded, pushed away from the gate and got in line to taxi. Then stopped....ultimately turning engines off...and sitting for almost 8 hours because of weather delays in the Northeast.
Passengers were upset, but the problems really lie in the way airline and airport schedules are set. Had we left our spot on the taxiway to offload, we'd lose our landing slot in NJ. There literally was a 'line', on paper, of all the planes waiting to come in. Get out of line, lose our spot. Go ahead and take off, and do nothing more than circle around until the weather cleared, and hope we didn't have to divert because of low fuel. Me? I'd rather just wait on the ground until it was definitely safe to fly and we'd get on the ground without too much delay.
The jetway we were originally parked at was needed for the next incoming flight. Either our plane sits there empty, or the next plane landing has to sit outside and wait for us to move. There were no spots at the airport to move empty planes for 'parking' and waiting until they could be cleared to load and take off...and even if there was a spot for them to wait, what happens to all the planes already scheduled to be in those jetways when the delayed flight is now ready to board?
I will say that the airline did their best, at the time, to keep us comfortable while we waited. Drinks were free, and they served every last food item on board to anyone who asked.
Unfortunately, turning aircraft around and moving about on the ground to return to gates is not as easy as it seems.