I guess it would depend on how long dinner goes. For 4 people with appetizers, wine, dessert, etc it could take more than a couple hours.
When I stayed at AKL it seemed like a long trip to Downtown Disney. IIRC, it was probably a half hour or so. And thats if you get there right when the bus comes. A taxi might make things a lot smoother if it's in your budget.
But I think it's doable.