I have never met my father. I know who he is, where he lives, but he chose not to be a part of my life when I was born, and my mom was ok with that.
It was weird for me as a kid. People ask who your dad is, what your dad does, do you live with your mommy and daddy? I had no daddy to take to the father/daughter stuff they had at school.
But, my mom was always open and honest with me, and I really think that helped it be less of an issue. She (to this day) has never said anything bad about my father (to me, anyway

). She said he was a great guy but wasn't ready to be a parent, and she was, and that she had more than enough love for me for two parents; she would tell me that families are all different shapes, sizes, and colors, but what matters most is that families, whoever they are, whatever form they take- love each other.
I would let your son talk about whatever is bothering him (and maybe it's not even bothering him so much as he's just trying to figure it all out). A lot of times kids will make up the answers to questions they don't know how to ask. Ask if it is bothering him. Ask him if there is anything specific he wants to know. Maybe he is just really curious and he's trying to come up with answers (which of course sound outlandish, because 4 yr olds are by definition, outlandish

).
Growing up I felt resentful that my dad didn't want to be around me. I thought maybe it was me, maybe he wouldn't like me. Now, I know that he was young and scared and didn't realize what he had gotten himself into. I know it has nothing to do with me, and everything to do with him. Your DS may need reassurance of that, too.
I think of my father every so often- I would love to meet him just to talk. See if we share any of the same quirks or interests, kwim? But, I am not angry, or upset, because it is what it is. If anything, I feel bad for him- he missed all of my important stuff: I graduated with honors from high school and got to go to U of I. I got married to a wonderful guy (and walked
myself down the aisle). I gave birth to a beautiful, smart, happy little girl, and he doesn't get to be her grandpa. He missed out, and continues to miss out.