Remy is similar to Victoria & Albert's in that you will be pampered as you are served several small (consistently delicious) courses and that, with advance notice, the menu can be customized to your dietary requirements/preferences. Remy would be my choice for an anniversary dinner, especially since you clearly enjoyed Victoria & Albert's enough for a return visit. (My husband and I, who avoid meat and shellfish, have dined at both and can't decide which we liked better, since both were amazing.)
Palo has several vegetarian options on the standard menu, and other dishes that can be adapted to remove meat, so it would be a fine choice as well. It just isn't quite the "wow" level of Remy.