I am sorry, but after a while I get bothered by the comments like "why would anyone with a 2 yr old want to do fine dining", " as long as they are not fussy, loud..." and the list goes on.
Well, I took my 2 year old to fine dining because I enjoy that type of food and he was going to experience it a lot in his life. He was fine, I find it a lot more troublesome sitting by adult snobs and obnoxious individuals who feel that they are better than others in the restaurant and can dictate when you should dine as not to bother them. Over my years, I have more trouble with arrogant and ignorant "adults" than small children and infants in all restaurants.
I have found that most parents know their children better than most of the experts out here who never met them or lived with them. Most parents will try to make sure their children will behave and those who do not are just more to add to the list of "adults" who are the problem.
I found taking my son out to nice places since he was more or less born (Narcoosee's and Artist Pt at six months) and other before that at home, he was only a problem to us once. And then we did the responsible thing and took him out of the dining area until we could calm him down. More times than not we were complimented by other guests and staff on how well he ws behaved.
Age is not a reason not to take anyone to fine dinng. I would feel fine taking my son to V&A and sure than most people would be horrified to hear that. But, now approaching 10 and he enjoys trying new foods and finre dining why would I deprive him of such a grand experience?
For the original poster, please do what you feel is correct for you and your family and do not worry about the others. You seem like a considerate person who would understand that some do not like or prefer to have children around them, so please be patient with them and if you child gets fussy try to to make them behave better

.
Have a great trip.