We love Sci Fi, and we have lunch there every trip to WDW. We don't have any interest in eating at Prime Time, it just doesn't appeal to our family. Having said that, I think Prime Time would be a better choice for your situation.
Given the ages of the children and the size of your party, I'm not sure you'd enjoy Sci Fi. The cars are so fun, but they aren't set up for larger parties and will make conversation and caring for young children difficult. There are some tables at the back where you could sit instead, but it seems that that would take away from the atmosphere, which is the whole reason to eat at Sci Fi, IMO.
Because it is set up like a drive-in theatre, Sci Fi is also pretty dark, which might not work well with very young children.