I don't think you're being selfish, but I think others are too quick to judge your friend, who wants to help in her child's classroom and is trying to figure out a way to make it work. Perhaps she can't afford childcare, or she's had difficulty finding someone whom she trusts to care for her baby.Maybe she would like to return the favor but hasn't thought of a way she could be helpful to you.
I suggest offering a trade. Maybe she picks your children up from school once a week (assuming your children go to the same school as hers and that she has a vehicle that can fit three children and a baby). Or she could watch them on a school holiday, or some other task that would prove helpful to you.
When I ws a FT SAHM, I had numerous people ask me to watch their children for various reasons. I often agreed, for the first time anyway, but if there was no offer of reciprocity, I had no trouble saying no the next time. Some people brought me gift cards as thanks, some watched my children in return, and some brought gifts for my children. Yes, there were some who didn't seem to appreciate it at all, and I was always too busy the next time they asked.
Don't agree to help if you're not comfortable with the person asking for your help or if you think you might end up feeling resentful and take advantage of, but also don't write her off as a bad friend just for asking.