TTW is NOT science fiction. It may have science fiction elements in it, but that doesn't make it a sci-fi book. Even I can tell that from reading a couple of quotes.
That's like saying Star Trek: 2009 is a romance movie because Spock and Uhura kissed. The theme was science fiction (wars between the planets in space) with romantic elements (Spock and Uhura, Kirk making out with that chick). The Time Traveler's Wife is the complete opposite. It's theme is romance (two lovers who know each other their whole life, even if they didn't know it) with sci-fi elements (time travel).