Location
------------
The deluxes tend to be located closer to a park. The Cont is a 5 minute walk to the MK, the YC/BC are a 5 minute walk to Epcot, etc.
Transportation
------------------
This ties in with the location. Even if a deluxe isn't located adjacent to a park it is often serviced by more 'Magical' transportation than a bus. The Poly is the best example - you can take the monorail from the resort to the MK, or a boat. You can walk (5 minutes) to the TTC and take the monorail to Epcot. If you have little ones and need to get back to the resort for a mid-day nap that convenience of location/transportation might save you an hour a day.
Room Differences
----------------------
The rooms at the 'true' deluxes are actually much larger than the moderates (there are large rooms at the AKL, but the AKL and WL in general have rooms about the size of the moderates - but they don't cost a 'deluxe' price either). You can easily fit 5 adults into a Poly or Cont, YC/BC room. Most importantly to me the rooms at the deluxes use real Queen Sized beds - the moderates use doubles. Another big difference is that the deluxes use central ac/htg which makes the rooms very quiet compared to the moderates - which use wall units.
Amenities
------------
Health clubs, Spas, Kids clubs, more shops, etc.
Now - is that worth paying extra for? Only you can determine that. We've stayed at deluxes and moderates and will stay at both again - depends on the budget for that trip.
The pool at POFQ is quite nice - but the Poly has a nifty new volcano pool and the pool at AKL is zero entry, has a slide, and is the largest on property.
If you were going to spend 2 days at a resort without going to a park - I'd stay at AKL. The multiple savanahs full of animals, nightly story-telling, great restaurants, fantastic CMs - it would be perfect!