I know you will. This is not a VB issue, it is an Ocean issue. Different world, hidden critters. Still, you are in much more danger during the drive over to VB. Hard to quantify that, but it is very true.
I've been been going to VB regularly since 1999. Many times I am the only one out in the surf (I tolerate cold better than most). Sometimes I get this uneasy feeling when I'm alone and that far out. If my fear gets the best of me I move in closer.
Fear is a funny thing. Sometimes, it is good and keeps you from putting yourself in a bad situation. Sometimes, it consumes you and prevents you from experiencing wonderful things. You have to try to put it into the proper perspective and you can only do that with knowledge.
Imagine if every entrance to a highway had a sign that said said "Warning... In the last 3 years X number of people have died and X number of people have been injured on this road... Enter at your own risk".
Enjoy yourself at VB and test the waters. Don't go beyond what your fear says not to, but don't let your fear get away from you either.
Have a great time my friend.