My youngest is neurodivergent and does not like the interactions at all and the cast members get the hint and leave her alone. A couple of times (we're local so we go a lot) I've had to specifically-and kindly- tell them that she is scared and doesn't want to play- and they always leave her alone. I have had, more than once, a CM break character to tell her that it's ok and they are nice.
Even as a parent of neurodivergent kids I have honestly never heard of this sunflower thing so I'm guessing most cast members would have absolutely no idea what it is. They are in character, but they're also human beings. Leaving you alone is not breaking character and you can ask them to do so, or at least back off.
That said, I have never heard them call anyone stupid. That could have been a CM taking it too far.