I agree that it depends on the tone of the red or green. A deep red or deep green that looks a shade or more darker than the pants would do. A pastel pink or mint green a couple of shades lighter would also work. The primary color shades of red and green would not work as well, in my opinion. I also think you need to match whether the gray is a warm gray or a cool gray. If it is a cool gray, you don’t want a shirt with a yellow undertone to it, which some reds and greens have. You want something with a touch of a blue undertone. If it’s a warm gray, then a shade with a yellow undertone works much better. I wouldn’t wear cream with a cool gray, but I would wear it with a warm gray.
I have the same issues when wearing a beige or khaki colored pants, especially when it comes to warm and cool undertones clashing. Just because it’s a neutral doesn’t always mean it’s easy to wear. . .