You don't have to be under the height requirement to use Rider Switch. This is from the Disney website:
What Is Rider Switch and How Does It Work?
If Guests in your party don’t want to board an attraction, you don’t have to miss out. Read More
If Guests in your party don’t want to board an attraction, you don’t have to miss out. One adult can wait with the non-rider while the rest of the party experiences the attraction. The waiting adult can then board the attraction without having to wait in the regular line again when another adult in the party returns to supervise the child or children who do not ride—either because they do not meet the height requirement or simply do not wish to ride.
You will be fine!