I really enjoy Seaworld. I am vegetarian and fairly animal rights-ey, but to me, the whales have been bred in captivity and aren't aware that there is any more world out there than they experience. I believe they are well treated, and so don't feel sad for the animals at Seaworld, but grateful for what they are doing to raise awareness of animals as sentient beings. I think whether concern for the animals will ruin it for you is a fairly personal thing though--if you are offended by the idea of animals being treated as performers even if happy and not mistreated, then Seaworld is probably not for you.
Animal Kingdom made me feel a bit uncomfortable when it first opened, but I think it has improved slightly since then.
To the OP: if you think you can fit in everything you want to do at Disney in 4 days, then Seaworld is well worth it. If not, worrying about missing old favourites may overshadow the fun of the new experience and mean it isnt a good idea this time, particularly if your 5 day passes are already paid for. Maybe skip it this time and build it into the schedule for your next trip.