This is going to sound very Disney-apologist to some but frankly, when I think back on all the times I have gotten into my villa well before 4pm, it would be difficult for me to get too upset over a delay.
The latest we have ever gotten into a room is about 2:30pm. On MULTIPLE occasions we have received our room between 8-9am. And I am truly grateful for the times that we did get in early.
Personally, I would be totally understanding of anything before 5pm. From 5:30pm onward I'd be simmering a bit...particularly if we were sitting in the lobby waiting for our room. And at 6pm...I'm not entirely sure what I would do.

Fortunately it never has happened and I'm hopeful that it will not.
But given how accommodating the resorts have proven to be in the past, 4pm is certainly not a hard line drawn in the sand beyond which I would demand satisfaction/compensation.