I think you have allowed plenty of time. If you're not staying at a monorail resort (GF, Poly or Contemporary) take your resort bus to MK then catch a boat over to GF.
We landed around 1:30 and were probably in our room just after 3pm. You've got a 2hr head start.
Just dress or pack whatever you plan to wear to the tea in your carry on bag as your luggage may not be delivered before you leave for your tea.