Last July, I wore sundresses with skimmies underneath and was very comfortable!
To answer all the questions -
If your thighs touch, most women's shorts will chafe or ride up, especially when it's super hot and sweaty out. Bike or compression shorts will not, as they are tight. They are also usually made of thin fabric and many are designed for wearing while exercising, so they handle sweat well. However, I would not be caught dead wearing bike shorts, or anything similar, out in public.
So, by wearing the bike shorts with a skirt over it, you are as cool as you would be in regular shorts, don't have to worry about shorts riding up/chafing, and don't need to worry about people seeing things they shouldn't when you go on certain rides. Plus there will be no photos of you in bike shorts at the park!