I would not feel guilty at all about not leaving him. You only have one child and he cannot be replaced (no matter what, having the one doesn't really come into play). If you don't feel he is safe and there seems to be more then one incident then I wouldn't think twice about not leaving him ever.
I also would skip the preschool at that age, especially since you'll be home anyways. Kids are only little once, enjoy your time with them. They'll do just fine when the time comes.