Letterman has a different style of interviewing. He doesn't fawn all over his guests, and he doesn't suffer fools gladly. If he thinks a guest is not worth his time, he doesn't really try to hide his disdain. Pair this with his dry sense of humor, and he can seem pretty curmudgeonly.
Julia Roberts was on the show recently, and she told Dave she was nervous and scared of him before her first appearance. Dave asked why she would ever be afraid, and she told him several fellow young actresses had told her horror stories about how mean he was and how he made their interviews hard. She then said that she was fine after her first appearance, and she figured out Dave just didn't like stupid people.