Staying on, or close to the ship, is often fun on port days. The ship is much less crowded. If there is nothing that appeals to you and you like the idea of a quiet ship, don't worry about it. We have cruised several times and have only gone on 1 excursion (we will do the same one in April).
Your choice...certainly don't feel obligated. Sometimes, I think people do.