I'm sure your mom doesn't mean to be mean, and I'm really glad she's open to talking about it.
Here's my perspective: I was your kid. My grandmother used to criticize me all the time when I was little: what I wore, how my hair looked, and so on. I hated it. When she would visit, I would be so worried that I would get stomach aches. In general, I was a pretty resilient kid and didn't let things get to me, but I felt entirely powerless--I was trying to be good and respectful so I never stood up to her, but would stand there and take it.
I'm sure from the outside it didn't seem like a big deal, and any one of the comments on their own wouldn't have sounded many alarm bells: "wearing your hair down would be so much more flattering to your round face" or "I don't think that shirt works with your figure" (for a nine year old!) or "you could be so pretty if you just wore more blue". But it was constant, and it was always the same theme--I'm not pretty enough, I'm not good enough, and I need to improve myself. It was also relentless, and I ended up being really scared to walk into the kitchen in the morning because I knew there was a huge chance she would find something to criticize me about. The worst part was that she could be quite kind in other instances, which always threw me off; it kept me from growing a thick skin with her, and it also made me wary around other people who were kind (because I worried that she was saying what everyone else was secretly thinking).
The hardest part was, I didn't think my mom was on my side. I was pretty stoic about it, as much as I could be, because I was sure that my grandmother was right and I did need to try to make myself prettier. But I remember feeling so alone and so hurt that I had to take her critiques over and over again without any support for my mom. The breaking point was at ten when she swapped my after-school cookie for an apple because she thought I was getting, as she told me, chubby. (I was a perfectly normal weight, even a little skinny for my height.) I ran down to the basement sobbing, and I heard my mom yell at my grandmother, and it was the first time that I thought that maybe my mom was on my side and maybe she didn't secretly agree with my grandma.
I brought it up with my mom a couple years ago, and she said that she had actually spoken privately to her mother many times before that--I just hadn't known. I still remember the feeling of absolute relief that my mom stood up for me when I couldn't, no matter how belated. (I should say that my mom never made fun of me or criticized me like this, and actually told me I was wonderful and beautiful many times a day. I think she was just trying to figure out how to stop her unstoppable mom, until she realized that the behind-the-scenes chats weren't having the desired effect. My mom, I should emphasize, was and is fantastic.)
I'm now over 30, and I still don't like to be alone with my grandmother. Seeing her still makes me feel panicky and prickly, and I avoid her as much as I can. I still try to be a good granddaughter--I've visited her in the hospital, etc.--but I don't like spending time with her. It's made even sadder because I'm quite close to my other grandparents.
Phew--therapy session over. All of this is to say, if it's gotten to the point where your daughter is actually scared of her grandmother and has been brave enough to tell you, it's gone way too far. Your mother may not realize it, but what she's saying is deeply hurtful to your kids and is making at least one of them think less of herself. You need to set some firm boundaries and stick to them: for instance, if she brings it up in front of the kids again, you will say, "Mom, I think Jane looks great. I'm really glad she's picked something she likes"--and if she says it again, you leave. Sticking up for your kids, respectfully but firmly, is fundamental.
I'm not saying the situations are totally analogous, but I thought something from the kids' perspective might be useful. I'm sure your mom is actually a lovely person whose concern for her grandkids is just manifesting in a less than productive way. I'm really glad you're talking honestly and openly with her, and I hope the situation gets better.