We had a pool party for DD7 when she was 6 at our neighborhood pool. We had to pay for each child and each adult ($3 each). I wrote the invitations to indicate drop-off and pick-up times so the parents would know it was a drop off party, but I paid for any adult who wanted to stay with their child (since it was a pool party, some did). When parents called to RSVP and invariably asked if it was a drop-off party, I said it was but they could stay if they wanted to. I never mentioned the cost to them, but I also didn't invite their whole family.
With a place like a children's museum that is probably going to be open during the party, you could either tell the parents when they RSVP or put a note in that sibling admission would be $ if they wanted to stay and play with their child separate from the party. Our children's museum has a limit of adults and kids in the party room and you can't go over that number - we would not be able to accommodate siblings or extra adults if we had a party there. You can also state that due to the size of the party room, you will only have space for one adult and the invited child, but other family members might want to enjoy the museum during the party for the price of admission.
I agree you shouldn't have to pay for everyone, but I would think some parents might stay for a 5 year old party, especially if the museum is a fair distance from where they live.