Good tasting wine can be found in a box, and good tasting wine can be found with a twist off cap.
Most of my favorite wines are in a bottle, and have a cork and cost under $20 a bottle....a couple were $5 a bottle.
And paying a lot for a bottle of wine can get you a wine you don't like.
Never apologize for the price, container, or method of containment of a wine. If you like it, that's all that matters.