I think you're hesitating because you might feel that in a way you're apologizing for feeling the way you do about all the other aspects about him that you don't like, when in fact you are only apologizing for this one thing. You probably want it still to be known that there are factors you don't like, even though that really has no bearing on this particular thing you are apologizing for.
If you really liked this guy and there weren't any other things that bothered you about him, I don't think you'd have any problem apologizing. I know it's probably hard, but it's the right thing to do in my opinion. Just keep it short and to the point, but genuine. Something like "The other day I was out of line when I wrongly accused you of ____. I was angry, but I shouldn't have taken it out on you and I just wanted to apolgize for that" would probably work.
Good luck!