Oh heck, we do this all the time. We are not morning people at all. There are still people arriving but most importantly, there are lots of people leaving. We get a lot done after 3:00 since most of the crowds are exiting because they got there so early.
We've done the safari in the afternoon and while the animals might be a little sleepy, we've had some really good rides. Plus, it you get a good afternoon shower, make sure you are near the hippo tank on the trail. Those hippos love rain and will be very active. Plus, the viewing area is enclosed.