Obviously links to valid news sources are just that, a collection of links.
If that was all he had nobody would have a problem with his site. It's the "exclusive" content and links to questionable sources that draws in the target demogrphic and keeps the rest from considering him anything but a tabloid.
That's basically the gist of it, but let me amplify this.
I have no problem with Drudge in the case of most linked sources. Acting as the de-facto editor, Drudge (or whoever operates the site on a day-to-day basis) chooses which stories are linked to the site. The stories themselves may be legitimate, but one way that an editor (in this case Drudge) can influence opinion by choosing which links will be run and which will not see the light of day. Drudge frequently chooses stories that favor the viewpoint of one side over the other.
Bottom line on this specific point; In this day and age, hopefully no one is relying on merely one source for their news gathering for that specific reason.
The second point; It's up to the discerning reader to understand what the difference is between a "linked" story in Drudge and one which gives no verifiable cite for its source. Very often, Drudge will run an unlinked story which is nothing more than a planned political attack.
As a rule of thumb when reading any internet media source, if there's no link and it's not an established media player, take the contents with a rather large grain of salt. Drudge's "scoops" are far more often than not nothing more than someone using the site (with Drudge's consent) as a means of targeting a political opponent. For every "scoop" that Drudge gets right (such as Lewinsky), there are many that turn out to be untrue. (A recent example is the fictituous 'heckling' of McCain by a CNN reporter that turned out to be blatantly untrue when video of the incident was revealed)
Unfortunately, many readers are not media savvy to understand the difference between the use of sources, and Drudge deliberately uses that fact to his advantage to give those stories the same appearance of legitimacy on his site as legitimate linked ones.
Caveat Emptor.