If you decide to go without your child, just make sure you can follow it through.
Thirty years ago my DH's company was having a "Partner's Meeting" which was a weekend at a location about 6 hours from our home. At the time we had 2 children, a daughter age 2 and a son age 6 months. I was thrilled to go. Got a gown for the Formal Dinner, all new clothes (he had just made partner) and it would be the first time since our daughter was born that we would be by ourselves. We made arrangements for the children with family and went our merry way.
My DH lasted one night. On Saturday morning he said, "I can't do this. We have to go home." I couldn't believe it. He never fed the babies, changed diapers, never was alone with them (he was really nervous that he would do something wrong but pretended it was a macho thing) so I didn't think he would have a problem leaving them with capable family members.
We went home and never went away without them until our youngest was 21. So if you can go and not be anxious/worried the whole time then go with your wife and have a great time. If not, then take the baby and I'm sure you'll still have a wonderful time, maybe even a better time.
I never did get to wear that gown.