It’s definitely not silly to expect Disney to stick to its own schedule.Here's the thing. Cleaning a villa is much like cleaning a house (especially GV) So mousekeeping has 5 hours to clean all these villas. To expect one right at 4pm is just silly.
In addition, yes it is very different. Most rooms on any given day aren’t getting any attention at all from mousekeeping.
The 5 hour buffer between check out and check in is quite large. It’s meant to accommodate for unforeseen issues.