I've stayed at both and loved them both. I think only you can answer the "is it worth it?" question because the answer highly depends on your own financial situation and the priority you put on certain aspects of your vacation.
The rooms at the Poly are larger than at the WL, but that didn't make much difference to us since it's just my DD and I traveling together. With 4 of you, you might appreciate the extra space. Although the WL does offer the bunk bed rooms which your 5yo might really enjoy.
Transportation options are the biggest plus to the Poly because it's a short walk to the TTC. You can take the monorail from there to Epcot. For the MK, you can take the resort monorail or the boat launch from the Poly, or walk to the TTC and take the express monorail to the MK or the ferryboat.
The boat from the WL to the MK is really nice, but you are reliant on buses for all the other parks.
I find both the WL and the Poly to be very relaxing. You really can't go wrong with either.