I just erased an overlong post I was writing about how I'd never seen Avatar and loved Pandora and both rides and how I'm not a Star Wars fan, although I am familiar with a lot of it, yet I loved Batuu and RotR.
RotR is the most innovative "ride" (really, it's an immersive experience, not just a "ride") that Disney has ever done. There's a reason people are willing to get up early, stand in line, and play the BG game--and do it more than once a trip. Yes, there are things you won't get or appreciate as much as a Star Wars fan will, but even if you'd never heard of Star Wars, I don't really see how you wouldn't be incredibly impressed by it and probably blown away, as we were.
You've got a lot of people in your group. Do all of you have to stay together? Maybe just some of you could do this on a designated morning?
I know there are posters here who are saying they weren't all that impressed or that their DD loved ToT more (perhaps she prefers thrill rides), but if you love Pandora and love Disney storytelling, I'd have to say that RotR is not to be missed. Do yourself a favor and do NOT watch any videos of the ride and stay far away from spoilers.
So, yes, I think it's worth it. Just cut out the worrying part--it won't change anything!