The safari is just riding around in the back of an open air truck type vehicle. So, dress for the outside weather would be appropriate, with consideration for normal signature dining dress as you mention. It was warm when we did it, I wore shorts and a nice collard shirt. My wife wore a sun dress.
Same thing we would have worn if going out for a dinner date on property. Dressy/comfortable casual.
BTW, we really liked this experience. Easily in the top 3 of anything we’ve done at WDW.