Why not just spend the coins little by little. Give the cashier some coins each time you shop instead of having to get back yet more coins in change.
I will almost swear that even Coinstar machines are not extremely accurate. A few months ago I made up a dozen carefully hand counted batches of coins and took them to a Coinstar machine. I made just one transaction but put in the batches one at a time with a pause after each batch. The machine shortchanged me a few coins, not all from the same batch.
"... a dime that was worth over 4 dollars ..."
U.S. dimes, quarters, and halves made before 1965 are worth a lot more than face value. In a stack, one of those will usually stand out by not having a gold or bronze tinged edge.