If you are looking at dinner, I'd suggest Yachtsman. Mainly because dinner at Concourse will "feature" the next door antics of Chef Mickey's. It will be a little, ahem, loud.
Yachtsman is my favorite steak place (well, LeCellier is up there too). It is a little on the pricy side but very adult and I couldn't think of a better place to have a Father's Day dinner with your husband.