It's a buffet and there aren't any characters, so how long it will take is very much dependent on what type of diner you are.
If you ask for the check when you sit down, you'll be ale to just get up and leave whenever you're ready.
Now, depending on what kind of transportation you have, getting from Jambo House to the Polynesian can take quite some time. If you're driving you're ok. But if you're using the Disney bus service, you'll have to take the bus to a park or DTD first and then transfer to a bus for the Poly (or take the bus to a monorail). In that case, it won't be Boma, but the transportation that will be the real unknown. If you wind up waiting a while for a bus.....all bets are off.