With a college team, I say "we" because its a team representing a school I paid good money to attend. I was a part of campus life, attended games, etc. In my case that means I can, but do not always, refer to the UNC Tarheels, Duke Blue Devils, and UNC Charlotte 49ers as "we".
A bit trickier on the pro level. I do sometimes say "we" when referring to the Carolina Panthers. While I am some what financially invested in them, I supsect it has more to do with emotional regional attachment similar to what another poster said about the Patriots and BoSox. FWIW, I tend to use the "we" only on sports talk boards and such and not so much IRL.