Well that would be fine if you are just going for the party. If you make it earlier and you are using MYW tix you would need to use a day's park entry unless you have hoppers and are already doing so. If you have UK length of stay tix it makes no difference then.
ALthough you can get in at 4 pm, none of the party events or merchandise are out until 7. Hope that helps
If you're planning to enter the MK with only the party ticket I would make the BBB appointment for 4:30. You'll be finished in plenty of time for the party.
Last year we saw a very long line of party goers waiting to enter at 4. BBB is in the castle and you'll need to allow time to enter (unless you're at the very front of the line), time to walk to the castle and the probability that they recommend you arrive a little early for the appointment