I have used a small backpack and also what I call a backsack (those bags with pull strings to close) Both worked really well, and didn't weigh me down.
The only thing I DID notice, is it made me hotter - atleast my back. Other than that, whenever I got on a ride, I would wrap the straps around my legs and then put my foot on the floor. Never lost it!!
However, I tried that with our mister bottle, and it broke - really upset me, but I guess next time I will be a bit smarter!