Yes you can. Just make sure you have your ADR number with you, they will ask for it. And get there plenty early, that is a wonderful way to get great pictures in front of the castle without the hoardes of humanity in the background!
Have a great trip!
Now one thing I did just think of is if it is an EMH morning, that may be a problem. But I still think you can get in fine as long as you have that ADR. You won't get a bracelet for the rides but it shouldn't be a problem for getting in for a breakfast. And maybe its not even an EMH mornign you are talking about!