I read through all the replies and I can honestly tell you there is no solution. Your child will outgrow the night waking when he's ready and there's not much you can do till then. It is not a result of your parenting, the fact that you have a husband, any flaws in your bedtime routine, nursing, etc., so don't feel guilty or blame yourself. It is just your child. I speak from experience. This will be long, but hopefully it will help you, and all the other sleepless moms, in some way.
My daughter, who is now almost 4yo and sleeps 11-12 hours every night, was the world's worst sleeper until she was 2.5yo. Every night from birth to 3.5 months, I put her down to sleep in her own bed at the same time. She was awake when she went into her bed, even as a newborn, and would fall asleep on her own. Around 6 or 7 weeks old she started sleeping longer chunks from say 8pm-11pm, feeding, back to sleep at 11:15pm-3am, woke for a feeding, back to sleep by 3:15 and slept till 7:30am. This lasted till 3.5 months and then for absolutely no reason whatsoever, she started waking up every 45 minutes - 2 hours, screaming hysterically. She had no reflux issues, I was nursing and hadn't changed my diet in any way, nothing we could think of. The ped went over everything with us and couldn't find an explanation either. My husband is also a physician and was at a loss, as were his collegues.
At 5 months things were so bad that I spent many a night sleeping with her on my chest in our rocking chair, which I had never previously rocked her in, and if I stopped moving the chair she'd wake up and start screaming. I was exhausted, and even though I'm married, my husband does not get up overnight with our kids, so it was all on me.
At 8 months old, after trying everything else (Ferber, No-Cry Sleep Solution, Happiest Baby on the Block, etc.) we did CIO. For 3 weeks straight I didn't respond to her nighttime wakings, but, just like your son, she was cry for literally hours on end until she vomited. I'd go in, change her sheets, put her back in the bed, and the cry till she vomited resumed. This went on literally all night for 3 weeks, with her never falling asleep the entire night. Oh, and she also stopped falling asleep on her own at that point, probably because she now associated the crib with crying and vomiting.
After 3 weeks we were at the ped's office for her 9 month check up and told her our progress and her advice at that time was to give up because Madison had more willpower than us and she was going to outlast us. We were all exhausted and I knew she was right, I was on the verge of caving in and bagging the CIO anyway, so we stopped. I brought her into our bed and we all got some sleep.
She still didn't sleep through the night, but at least when she woke up, there was no crying because mommy was right there. She'd move over so she was touching me and go back to sleep. She also started night-nursing again, which I didn't initially want to go back to, but I was desparate for sleep and I could sleep while she nursed in bed so all was well.
Oh, and I want to add, she never went to sleep nursing at night, even as a newborn. I always waited around 30 minutes before putting her to sleep as I read that if a baby had time to digest first, they'd sleep better. Clearly it wasn't true for my daughter, but the result was that she was going to sleep without nursing, so that was not the problem for her.
We tried CIO again, along with all the other methods, at 11 months, and again no luck so we gave up again.
At 14 months I was done with the night-nursing and ready to try again to get her to sleep on her own. We bought her a toddler bed as she hated the crib and liked our bed so we thought we'd give it a try. She did like the bed more, but still woke up between every 45 minutes and 2 hours. I did break her of the night nursing around that time though, but I had to do it on my own. My husband tried getting up with her a few times, but night wakings just aren't his thing so that was that.
Her ped suggested bringing her to the sleep clinic at Children's National Medical Center so at 16 months that's what we did. They had no advice we hadn't heard/tried before, but they did say some kids just wake up at night for seemingly no reason and they grow out of it. Great, just what we wanted to hear.
The one thing they did suggest was to move further away from her each night waking. At that point she would wake up overnight, come to our bed, we'd walk her back to her bed and sit there while she fell asleep or just pull her into our bed. So they said to walk her back to bed and sit with her as long as it took for her to fall asleep each waking for a week, preferably with a hand on her back or touching her in some way and time it. After a week we should remove the hand from her back, or whatever, a few minutes before she would normally be asleep. Then make it longer and longer every few days until I was just sitting next to her bed as she went back to sleep. Then move further away from the bed gradually following the same plan until I was at the doorway to her room. The idea was that if she got used to going back to sleep with me just standing in the doorway, then she'd be able to do it on her own.
Well, I tried that for 2 months and I was purely exhausted. I never made it as far as the doorway cause she'd break out in screams, but I was able to get several feet away from her bed. When I realized that after 2 months I still hadn't made it to the door, I gave up. She came back in our bed when she woke up at night.
At 19 months I bought an inflatable toddler bed and put it on the floor next to my bed. She slept happily on it all night from 19 months till we moved at 26 months. She still woke up periodically, but she'd just feel around for my hand and hold onto a finger till she fell back to sleep. And she was finally out of my bed so that was a big plus too!
We put her in her own room when we moved (she was just over 2yo) and it was a mistake cause the night waking was back to me going in and out of her room, only now I was pregnant and her room was on a different floor than ours. Well we decided to do CIO again, though she wasn't really crying this time around, but she ended up pushing down the baby gate in her room, climbing the stairs and falling down the whole flight in the middle of the night while trying to get in our bed. That was it. We said it wasn't worth her possibly getting hurt next time (luckily she was completely fine), so we brought her bed upstairs in our room and all was well. She woke up at night, but we told her that if she wanted to stay in our room then she needed to stay in her own bed and that worked. She was 2yo, 5 months. Soon after that she was sleeping all night!!!
Now, at nearly 4yo, she sleeps in her own bed all night long, though she does optionally share a room with her brother. Fine with me as they both coexist in there without complaint. Oh, and just to add, she never had a night light (neither of my kids did actually) and my husband is a light freak so all of the bedrooms get black at night, so not even a crack of light can get in.
Now, the reason I can tell you without a doubt that it is the child's personality, not parenting, is that my son was born when my daughter was 2yo,5 months. I did everything exactly the same with him as my daughter as a newborn. I was terrified he would be the same way, but didn't know how I could do anything differently with a newborn. In fact, he was born preterm and had severe jaundice, so I actually had to wake him up to feed him more frequently at night, plus he always fell asleep nursing. Always.
At 6 weeks old he was sleeping 6pm-6am with 1 feeding at 1am. By 3 months old he was sleeping 6-6 straight. And he was the one who used to fall asleep nursing, got night nursed like crazy, but he slept just fine. Plus, being preterm, 6 weeks was really young to sleep all night. Oh, and he did not like sleeping with us at all. He wanted his own bed from the start. The only time he woke up overnight was if he was sick or teething. And one other thing, this kid is addicted to nursing.
He's almost 17 months and I can't get him to stop no matter what. He still nurses before bed and naps. My daughter though, was down to once a day by this point (when she woke up in the morning, so nothing to do with sleep) and completely weaned soon after. My son shows no signs of being on his way to weaned.
So you see, 2 siblings, raised exactly the same way, for the most part, totally different. And what the above poster mentioned about her poor sleeper being a challenging child in all ways, that is my daughter as well. My son is a much happier, free-spirited child, but my daughter, well, she can be a handful and tempermental when she wants to be. She's outspoken and knows how to get what she wants, which is what she did with the sleeping, and years from now I think I'll be glad she has those qualities.
So you'll probably just have to wait it out or take him in your bed. He will not be there forever, I promise!
I can't even imagine dealing with it as long as you have been. 
I hope you are able to find a solution soon.
They grow so quickly.
. Been there, done that, unfortunately. I hate to say it but some kids are just that way. My DS was an issue for a few months but after that was a great sleeper, my DD on the other had was not so good. She was a preemie so she had to be woken up every 3 hours to eat and I just think it screwed the poor girl up for life. She is now almost 7 and she still has nights when she won't sleep through. On the bright side when she does there isn't much that will wake her, but when she doesn't it is not fun. I am not a happy Mommy if I don't get my sleep 
. All I can say is give him as little attention as possible when he wakes up, so it's not a satisfying social time for him, and switch off with your DH if you can. My DH and I would do that so one night I had to deal with her, the next he did. That way at least I wasn't woken up every night. If you wake up anyway and have a hard time not getting up or getting back to sleep (which is my problem) invest in a good pair of ear plugs, they work wonders 
She woke up and tried to climb on me and give mer her cup and get me to hold her and I just layed there like I was asleep. I wouldn't have ignored her if she was crying/screaming, but she was just unsure what to do I think. She rolled around for a while and went back to sleep till right before 8. We'll try that again tonight I guess. Evan didn't sleep through till 18 mos and then he just did it one night on his own and slept all night since then. He did end up losing a # from sleeping through the night though and we had to up his daytime calories.
