I hear you. Although I am fortunate enough to be able to work from home, I would love to be a 100% stay at home mom, rather than a work-at-home and stay-at-home mom! Because I work for a real employer with real hours, I needed to have in-home care of the girls when they were younger, and now they go to preschool and elementary school during my work hours. It's hard not being the sole caregiver of my girls, but it's been a necessity. How old is your little one?
We are constantly re-assessing the chance for me to just be a full-time homemaker, but right now, this is where we are. I don't know if you've ever heard of it, but we took Dave Ramsey's Financial Peace University course and it was life-changing (his 'Baby Steps' are so easy to follow:
http://www.daveramsey.com/new/baby-steps). On days when I feel down about working, it helps me press on, knowing we are working towards long-term goals that will go beyond just keeping us ahead of our bills, but will also build a legacy for our children.
Hang in there. No matter what decision you make, as long as it's in the best interest of your family, it will be the right one!