On the one hand, no, I don't expect DH to stick around in exactly the same room when I'm talking to a guest.
On the other hand, if the guest was a family member of DH's and he took off, leaving me to entertain his family alone for an extended period of time, I wouldn't be happy about it.
That actually happened one day... DH's family members (who live nearby) were over for a visit. DH went down to the basement to toodle with his stuff without saying anything, basically abandoning me to deal with his family on my own. Turns out his family was driving DH nuts. I gave him what for later.