Back in 1996/97 I did let my then 4.5 use the bathroom by himself in WDW. You did not hear about attacks and such as often as you do now.
Fast forward, no way is my current 4 year old ready to use the restroom by himself other than at preschool or home. When DH has the kids out by himself, we let DD 5.5 use the ladies room by herself while DH waits outside the door. Nicest way to put this, Gentlemen you need to clean up your act!
I have no problem with boys using the restroom with their mother's no matter what their age. As long as they are not looking under the stall, peeking through the cracks, or tall enough to see over the top, who cares.