In my opinion, the Waldorf (I stayed 8 nights last year in a 1 bedroom suite) is more luxurious than the Grand Floridian or the Poly. (and I have stayed in the GF, Poly, Boardwalk, Beach Club).
If you are really into the Disney theming, you may miss that...but the level of luxury is above. To me, as much as I loved Poly, it is in serious need of some TLC and I find it to be a little "grimy" now. We did also see children at the Waldorf and they were enjoying it. You can walk over the the Hilton to use their pool (kid heaven).
We are headed back to the Waldorf for our second stay. We've also stayed club level at the Polynesian and in the Grand Floridian main building (to compare to Disney deluxes), and I have definitely been accused of being a hotel snob.
Speaking as such, Waldorf is head and shoulders above either of those top-end Disney resorts in terms of actual luxury, for sometimes about half the price. We booked "deluxe" Disney-view suites both times, and used the standby upgrade to the Waldorf Disney-view suite for a modest up charge, which was totally worth it. (We went during slow times, so no guarantee that that's always available, but if it is, you will be offered the potential paid upgrade option after you make a reservation; at least you will if you're an Honors member.)
Hands down the best view of EVERY Disney park we've ever had. Epcot's Spaceship Earth was perfectly framed out the bedroom window, Magic Kingdom (distant) and Hollywood Studios (directly in front) off the balcony. Spectacular fireworks views.
I also found the service at the Waldorf to be far superior than what I've experienced at Disney resorts. They really went above and beyond to make you feel welcome and comfortable. Beds are comfortable. Rooms were super quiet. And my young son enjoyed himself there immensely.
The Hilton pool complex, accessible to Waldorf guests, is nicer than just about any pool on Disney property, save Beach and Yacht clubs, though the slide is pretty small. Lazy river is a lot of fun, though. We actually liked the Waldorf pool a lot. There's a big zero entry one and my son always found kids to play with so we only walked down to the Hilton once or twice.
Food was excellent. Ate at Bull and Bear, which is a seriously good steakhouse, and the Oscar's breakfast, free if you're a Hilton gold member, was a HUGE, fancy, multi-room buffet with lots of cook-to-order stations and just about anything you could think of. We didn't do it on early park mornings, but loved it on our lazy days.
Yes, transportation options are more limited to Magic Kingdom, but as someone who loathes Disney buses, that was our only loss. It's located right next to a Disney moderate (forget which one, maybe Caribbean beach?)--actually closer to the parks than it is--and around the corner from Hollywood Studios and Downtown Disney. We actually found it faster to drive to most of the parks from the Waldorf than the Grand.
Honestly, everything just seemed much more relaxing at the Waldorf than at any of the Disney resorts we've stayed in, or visited. And for room quality, service and price ... I don't think you can beat the Waldorf.
The Ritz, and we're normally big Ritz fans, is a serious hike from the parks. (It's closer to Sea World.) Nice, and part of the big JW campus, but I wouldn't want to stay there if I had any intention of visiting the Disney theme parks.
Non zoomed in view from our corner suite bedroom window. It was even more impressive in person ,especially when Disney calls seeing 2" of Space Mountain a "theme park view" and charges an extra $100/night for it.
Anyway, the grounds and pool were quite nice and guests can also use the Hilton's pool next door. 
