I've sailed concierge a number of times and can't recall anyone wearing PJs in the lounge. Personally, the lounge feels a bit upscale to me and it's definitely a public area, so, personally, I would feel uncomfortable wearing anything that might resemble PJs/nightgowns/robes, etc. in there. I never allowed my DS to go in there in PJs either - it just feels more like a restaurant to me and it's a public space - maybe I'm just really old-fashioned?!
I think sweatpants or lounge pants and a t-shirt are fine in the AM, but as others have noted, wearing full-on PJ sets or anything resembling sleepwear might look a little out of place in there. I'm usually the one who runs to the lounge in the morning just to pick up coffees and pastries to snack on in the room before everyone is dressed and ready to go to the lounge/walk around the ship, etc. I typically just throw on a sundress and some flip flops to go to the lounge to pick everything up.