Actually, the setting of it is part of the debate. Its not actually clear when the movie is set. And, Slavery was still force in the late 1800s. The war didn't end til 1865. So AT MOST that move is set 10-20 years after the end of slavery, at a very tough time for African-Americans in the US. Yes, absolutely the portrayal of the Happy Sharecropper who was a slave just a decade ago is problematic. And yea I'd say its racist - even by the standards of the day in which it was made.
And its not like its all that far from the civil war now, either. 200 years isn't that long ago. I knew my great-grandmother. She died when I was 18. Her father was born during the time of slavery. He died only a few years before I was born. And I'm a relatively young adult.
Personally, if I am watching a movie involving marginalized groups in the US, I go by what that group thinks of the portrayal as I am not in a position judge. The were protests before the movie came out for exactly the same reasons that people it find racist today. So, yes. I'd say it is, since the people it's about say it is. There were protests at the time, in fact.
Movies don't happen in a cultural vacuum. There is a long history of Black characters being portrayed as simple, magical and/or happy to be servant/savior of white people to justify treating them unfairly. Uncle Remus is all three. The stories he tells are well-known to be racist in origin, a fact that is purposely ignored in the movie. Tar-baby, for instance is a racist slur against Black people.
That doesn't make it bad move, or make Walt a bad person, or make people who enjoy it bad people. It means that the portrayal of slavery and its impact is white-washed, and problematic, and we need to acknowledge that. I also find it fascinating that people get way more upset about being called racist or having something they like called racist than they are about it actually BEING racist.