Do what you need to do... I would probably look at you twice if you left them outside alone.
FWIW... my son (he is now 10) was assaulted in a Target bathroom the very first time that I EVER let him go in a public restroom alone (he was 8 at the time.
I had been thinking that maybe it was time to give him a little independence, he had started arguing about going into the ladies room with me and it was a Tuesday afternoon at 2:00... store was slow and I was right outside of the bathroom.
I heard a bit of a commotion and then 4 teens ran out. He came out of the bathroom holding his rapidly bruising eye... he was walking to the sink and one of the boys punched him in the face.
Needless to say, he went in the ladies room with me for the rest of the year. Then, we made sure that we knew locations with single bathrooms (Chickfilet, the book store, etc). Now he is going to the mens alone. But I am always right outside and if I hear ANYTHING, I have no problem barging in.
You can never be too careful. Not meaning to scare you, but I would take them in...