Funny you should ask! Our boys are ages 5, 12 and 13 and I often take them to SAB on my own while my husband takes a break. It really depends on the variety of factors others have mentioned. I love it, and I don't find it either dangerous or overwhelming. But here's why:
1) I have no problem walking around with them and getting in the pool. But I do that at most pools.
2) Our boys are all strong swimmers. They are all on swim team in the summer and our youngest is a year-round swimmer.
3) I don't expect that it will be entirely relaxing. Yes, it's easier to sit in one place and watch your kids at other resort pools. I was able to do that easily at CR, for example. With kids, we've stayed and swam at Poly, WL, CR, BWI, AKL, BC, YC, Dolphin, CBR, CSR and POFQ. But it's a trade-off - the more fun pools require more monitoring.
4) There are a few places where I can sit down and relax. Our boys all enjoy spending some time in the shallow sandy area of SAB. When they all decide to locate there, I grab my Sunken Treasure and sit by the side of the pool. Same when they decide to swim over by the small slide / kiddie pool area at the base of the pirate ship near the lake. But most of the time I'm in the water or at the side of the pool taking photos anyway.
5) Occasionally, our older boys go to another area and then meet us at a designated spot. At ages 12 and 13, with strong swimming skills, I'm very comfortable with that. But they usually choose to stick together and we move to different areas of the pool together.
I know you've already seen lots of photos, but here's my SAB photo thread, in case it's helpful:
http://www.disboards.com/showthread.php?t=2790323
We have APs so we've been at WDW 2-4 times per year recently, and we usually stay at BC/YC for at least part of our trip. For our upcoming trip, we're splitting between Poly and BWI because my husband loves BWI CL and wants to mix it up a bit. But the Luna Park pool will be closed during our stay and we'll have access to SAB instead. We are THRILLED!!! Since my husband often takes a break during our pool time, I'll be the mom at SAB on my own with three boys at the end of January. And I definitely prefer it over the BWI pool.