The copyright law explicitly allows libraries. The Fair Use Doctrine only permits partial use, and only for review and scholarly purposes, not copying of an entire work, without placing it in the context of a larger work of your own, which you then publish.
You are correct about the special exemptions that are made for libraries.
But your statement on fair use is a bit simplistic. The law permits "fair use", then goes on to mention several things that should be considered when determining what a "fair use" is. It does not explicitly define what is legal and what is illegal. That has been left up to the courts to decide and there have been a number of differing court opinions about what constitutes "fair use".
In Sony vs Universal Studios, the courts ruled that recording an entire broadcast TV show for personal use and viewing at a later time did qualify as "fair use". This is essentially, the "copying of an entire work, without placing it in the context of a larger work of your own". And it's not for any scholarly purposes.
One of the factors that is to be considered when determining whether a use is fair or not, is the effect that the use has on the work's value. Copying a VHS tape or DVD for personal use, has basically no effect on a work's value.
For anyone interested, there is some good information here:
http://en.wikipedia.org/wiki/Fair_use
"The fourth factor measures the effect that the allegedly infringing use has had on the copyright owner's ability to exploit his or her original work. The court not only investigates whether the defendant's specific use of the work has significantly harmed the copyright owner's market, but also whether such uses in general, if widespread, would harm the potential market of the original. The burden of proof here rests on the defendant for commercial uses, but on the copyright owner for noncommercial uses. See Sony Corp. v. Universal City Studios,[14] where the copyright owner, Universal, failed to provide any empirical evidence that the use of Betamax had either reduced their viewership or negatively impacted their business. In the aforementioned Nation case regarding President Ford's memoirs, the Supreme Court labeled this factor "the single most important element of fair use" and it has indeed enjoyed some level of primacy in fair use analyses ever since. Yet the Supreme Court's more recent announcement in Campbell v. Acuff-Rose Music, Inc.[15] that "all [four factors] are to be explored, and the results weighed together, in light of the purposes of copyright" has helped modulate this emphasis in interpretation."