i *think* i know the story... here's what i've heard.
he played a concert in the 70's at MSG and was booed b/c they didn't like his "new look" and the fact that he was trying to change his image. he wrote "garden party" and quit touring. if you listen to "garden party" it sort of bashes the fans and talks about him not looking the same. then, rather ironically, the song became one of his biggest hits.