We have been camping more years as empty nesters than we did with our three children!! In fact, dare I say, we enjoy our camping times now more than ever - more relaxation, rest, spur of the moment decisions, etc. etc. etc.
We take many trips a year now - no work schedules - so we can take off most times when we're ready. Having the children was great, but does not in any way diminish what we enjoy on our trips now.
By no means will you be the only ones - in fact most of the times we enjoy going, we see mostly couples, with a few families thrown in. Go, and have fun!!