DH and I are in a similar boat- we're refinancing out of an FHA loan to remove PMI (about 350 per month)
What our lender told me is that that they will take the middle score of the lowest person on the loan. So unfortunately, they will be using your husband's score. Also unfortunately, they use an older scoring model that will typically show your score as 20-30 points lower than what you are used to seeing. When he told me mine I freaked out. So be prepared for that. However, there is still a lot that you can do to get a decent rate by paying more in points, etc.
I think the biggest thing will be determined by what interest rate you can get, and what costs you will have rolled into your loan. For us, the biggest benefit was not the lowest rate offered to us, because it had more costs and fees associated with it, and so the overall payment was actually higher. We are going from 3.5 to 3.625, and it will eliminate nearly all of our PMI payment. Because you are at an even lower rate than we are- the difference in interest may be too high to really offset any savings, especially over the course of only 5 years. (we have 9 more... ouch!)