I honestly like having my bigger sized wallet on me. If you get too thin or too light, you could very easily lose it and/or send yourself into panic mode. Cargo shorts do offer deep pockets, but if you have a sleek, thin wallet, it's easier to remove. Sure, you don't want to have the unnecessary stuff with you, but if you can cut 4 down to 2, and always have it where your hand is swinging, you'll have that "security" of knowing it's there.
Otherwise, if I'm wearing athletic shorts in the park (I try to find ones with pockets) and they don't have pockets, I'll give my stuff to my DGF who wears her crossbody bag which she wears on the side that I walk on so no one can get in from the outside.