It sounds like the OP has a gut feeling something is wrong. If the grandparents truly wanted to be grandparents they would see that child every chance they got, even if they "felt like great aunt/uncle" when the rest of the extended family is around. And they would have discussed what role they wanted to have in the grandchild's life with the adoptive parents when the adoption happened. It's not like the baby was born and adopted the next week. The child was 1.
Their behavior is suspicious and if I were the OP there is no way the "alone" visits would continue. If they want to spend time with my daughter they can do it with me in the room. Period.
This. Soooo well said.
OP - I am usually shaking my head with the majority of threads posted on the DIS that complain about grandparents doing too much or not enough. Based on what you have posted I'd be very, very suspicious of their motives and intent!