It's not a matter of whether or not he deserves you. It's a matter of his deserving to walk the path with a partner who shares the same common bond of love. You deserve that too. It is the most basic bond of marriage, unworkable and worthless without it. If you cannot offer that to him, you have an obligation, to both of you, to walk away. Because you know it has no chance to be a whole marriage, built on what you both promised.
Sure, you could just coast through and try really hard to make it work, but when things get rough (and they will, many times), you will go back to the same thing. "I didn't love him from the beginning, I felt obligated to stay. I am trapped with someone I don't love". And he will remember this message too, "She doesn't love me, never did. Nothing I do is good enough. I am trapped with someone who does not love me" It will eat away at both of you and crush both of your spirits. The strength of a marriage is tested time and time again. A marriage that is not built on mutual love does not have the strength needed to withstand challenge.
You have basically told us that there is nothing really wrong with him or the marriage other than the fact that you are not sure if you love him and you are not sure if you ever did. Where is he supposed to go with that? What is he working on? Becoming more lovable? Becoming someone you will love? You can't change things to make someone love you if they do not have those feelings and especially if they never did. It is not fair to make someone work on something that is not real for the other person.
If you really, truly feel that you don't think you ever loved him, you have an obligation to cut him free. Because you are basically saying that your vows were not real for you. Even if you learned this later, it still boils down to the same thing. People do not say what you said out of no where. Especially if it was not said in the heat of passion, to get the other persons attention. Not that doing so is right, but sometimes people say horrible things to people they love, for attention. Doesn't sound like that to me. Unless you are a flighty person, which you don't seem to come off as, you have had this feeling for a long time and it is the truth for you.
I don't think that makes you bad, but I think you have an obligation to both him and yourself to walk away. because I don't think the feelings are there for you. I don't think you would have said the words if they were.
JMHO