I went back to work this year, when my youngest son started first grade, after 12 yrs as a SAHM. I'm teaching preschool, so I don't need to leave for work until after my kids leave and I usually get home before my oldest kids do. That's part of the reason I chose not to go back to teaching elementary school - I would not have felt good about my oldest two being home alone every day after school and they're too old for sitters. My kids were initially very bothered that I was going back to work (partly b/c I let them know that they were going to be more responsible for helping around the house now) and generally made me feel very sad about it, but they have adjusted. They still don't like it when I have meetings in the evening, or Open Houses, etc though.

It also takes time on weekends, etc for me to get my planning done, although that will be better in years to come.
I'll be honest, I do feel more rushed to get things done and I found holiday shopping especially tough. However, I also know that I am much more organized when I am busy, so it just means that I waste a lot less time. I find myself feeling very jealous that many of my friends have kids even older than mine, yet they don't work, so they have plenty of time for working out, lunches together, etc. They seem to regard my working as more of a hobby and seem annoyed when it's difficult for me to get together with them.
BUT, there are many, many positives. While I loved staying home, I LOVE being a teacher and feeling somewhat professional every day. I love that I have stories to tell my family at dinner that aren't just neighborhood gossip, and that I have
a life of my own.

I totally underestimated how nice that feeling would be! I love my students

and I love having coworkers. And in six short years, when my older kids go to college, hopefully, having a life outside of my kids will ease that transition a bit. I will also have the satisfaction of knowing that I helped pay for their college. We have never been able to save anything for their college, but now, we are putting most of what I earn (which isn't much!) into college funds.
So, yes, I totally understand why you would feel a little panicky about going back to work. But, I think if you give it a little time, you may find it very rewarding and your family will adjust easily. Good Luck!