As someone in your age group, who has been in a similar situation, I would say it is not too late to find someone else to spend your life with. You need to take care of yourself! You deserve better than to be with someone you can't trust. Even being alone is better than being in a stressful relationship. I know how scary it is to feel alone, too. There is someone out there for you, someone you can trust and feel comfortable with.
You also sound like you could use some outside support. Have you thought about counseling? It can be a real help when you find yourself in this type of situation.
It is probably going to be easier on you to start distancing yourself from this person now, rather than later when you are even more deeply involved. It doesn't mean you can't remain friends and see each other sometimes. I would try to avoid becoming too emotionally involved or dependent on his company. Keep your eyes open.....and play "Someday My Prince Will Come." That's what I did and it worked!