We don't live anywhere near downtown.

We live near Wrightsville Beach, but we are in the Wilmington city limits. We live in a nice neighborhood with lots of kids (and also the "older crowd" - not sure if you count 40-somethings in that or not, so I'm not sure if I qualify). I rarely go downtown, but that is because everything I need is located where I live. There are lots of bars downtown, but I know very few people who live there. Most live in the suburbs like any other city of Wilmington's size.
I wouldn't expect the OP would want to live downtown if the beach is important. They are nowhere near each other, unless you are referring to the banks of the Cape Fear River - not exactly a beach. Quiet, middle class Wilmington includes many different areas.
FWIW, UNCW is not downtown either. We appreciate the amenities of the college (sports, summer camps, lectures, concerts, etc) without actually attending any keg parties.
And Wilmington is much smaller and quieter than Raleigh, and certainly has fewer bars and fewer college students.
We are less than two hours from Myrtle Beach - we just went there last weekend.