Do you think he would enjoy one over the other(breakfast vs dinner), not really like either one of them, and maybe try someplace else?!
We have:
Sci-fi
Tusker House breakfast
Chef Mickey's
Nine Dragons
Le Cellier
Les CHefs de France
Teppan Edo
Cape May buffet dinner
My son will be 6 in January and we went to 1900 PF breakfast in July. He loved it! In fact, he still tries to make the "Mad Hatter funny face"!!

The food was excellent there too, the best I have ever eaten at a WDW character breakfast. Dinner is funny with Cinderella's step-family. We took the boys there 2 years ago when they were 1 and 3. Food was just OK, not nearly as good as at breakfast IMO. I think dinner is OK for young boys though. I brought my sons to CRT dinner and Akershus lunch on our July trip and they both enjoyed them. DS5 was just shy around the princesses, but he said he liked those meals.
Sci-Fi is great for kids this age, IMO. We went to lunch there and we all loved it. The kids sat in the front of the car and DH and I sat behind them. They loved it there.
Chef Mickey's was great for the boys too. We had breakfast there in 2008, and it was wonderful. This past July, we had dinner there and we all loved the character interaction as well as the food and service.
Le Cellier is fine for children. DS5 loved it there because they had steak on the children's menu and steak is his favorite. That was one of the reasons we wanted to go there. DH and I found the food to be fine, just nothing extra special. We would like to try Yachtsman Steakhouse next time with the boys. They could still get steak there, and I have read the steak is better there than Le Cellier, so it is worth checking out for us. The boys will be 5 and 7 when we go back.
Chefs de France was excellent too, and our kids enjoyed it there. We got to see Remy (from the movie "Ratatouille"). We were there for lunch. I think Remy is there from 12-6pm, but he is never there on Sundays. I don't know how much longer he will be there though...we considered ourselves lucky that we got to see him. We didn't tell the kids he might be there, in case he wasn't so there was no disappointment. We surprised them and they were very happy!
We had an ADR for Cape May dinner buffet, but never got to eat there. I knew they served shellfish at the buffet, but I didn't know they actually steam the crab legs right up at the buffet, out in the open. DS5 is allergic to shellfish. I knew he was allergic if he ingested it (the only time he had a reaction was from eating it), as he was tested, but we have eaten shellfish in front of him and he was fine, so I thought Cape May would be OK. As soon as the hostess walked us into the restaurant, he started to get a stuffy nose, his cheeks turned red and his eyes got swollen. I took him right out of there into the lobby of the Beach Club. The manager came out to make sure he was OK. His reaction cleared up within minutes and he was fine. The manager went above and beyond the call of duty and arranged for us to eat at a different restaurant. We went to Kouzzina (our choice). Sorry to hijack this thread for a sec, just wanted to let you know that if you have anyone in your party allergic to shellfish, they steam crab legs out in the open at Cape May.