I understand your being torn, I was too. We brought my DD to the CRT breakfast back in 2007, when she was 3. We paid OOP. Like a PP said, breakfast is a much safer bet, because it is harder to screw up. I thought the food was on par to the other breakfasts we had attended, and I actually found that the service was quite good and the princess interaction was the same for us at the castle as it was at Akershus. My only problem with the castle experience, was Cinderella. I don't know how to say this without sounding mean, but she was overweight, unattractive and not friendly at all. I think WDW must have been short on Cinderellas that day or something, because I can't imagine why they would cast her in the role. I don't think it was her regular gig. This bothered me tremendously, because the selling point of the castle meal is the picture package! I got my DD all dressed up in her Cindy costume and was hoping for a nice picture to have framed. The picture left a lot to be desired!
So now fast forward to three years later. We are going back for the first time. My DD is 6 years old and she does not remember being in the castle. I missed the ADR date to reserve breakfast or lunch, so I reserved Akershus and 1900 Park Fare. Then it was announced that the princesses would be at the Castle dinner as well. I decided to book it and take my chances on the food, because when it came down to it, I just don't have the heart to take my DD into the Magic Kingdom and then tell her that we cannot go inside the Castle! (I know, she has me wrapped around her finger.) We are doing the DxDP so we have the credits to spare. I feel a little bit better about using the credits on dinner than I would for breakfast, because it is more money, but my husband and I are foodies, and we are gearing ourselves up for a major disappointment. But hey, I made the reservation for our first night, so we will be so happy to be there, I have a feeling we won't care what we are eating! (And then we have much better dinners to look forward to the rest of the week!)
I say, if you can afford it and your kids really want it, then book it. The next time we will probably get back to WDW, my DD will be at least 10 years old. We can choose to be more frugal then.