No, it is an objective fact that FastPass (and any other line-skip paradigm) works better the less people have it. There is no "seeming" involved here. When lots of people have and use FastPass, the expected wait times in the FastPass queue go up, reducing the value proposition for everyone involved. When few people have FastPass, the wait times in the FastPass queue decrease, and the value proposition goes up accordingly.
Remember, everyone gets on the same ride in the end; Disney cannot simply add more mine trains to boost ride capacity and therefore there is an absolute limit of how many people can ride. If there are 100 people in the FastPass queue, then depending on what percentage of the train is reserved for FastPass, the 100th person in that queue is looking at a wait of 5 trains at minimum and possibly as many as 10. By contrast, if there are only 10 people in the FastPass queue, all ten can get on the next train, no problem.
I'm not a super huge fan of thrill rides, so I don't really have a horse in this race. But it's important to establish the fact that "FastPass is better when less people use FastPass and gets worse when more people do" is a fact.