Our 4yo disappeared from our sight for awhile, though not because we intended it! It was for about 10 minutes and my wife was nearly in tears. I told her that aside from getting eaten by an alligator, what was the the worst that could happen on this isolated piece of land? Probably not the best effort at humor. We eventually found him, running through the fort at top speed and grinning like mad.
All that said, I would imagine that on our next trip, when he'll be 6 3/4, we'll probably let him have a fair amount of freedom.