We almost always stay off-site (we drive to WDW), and it is much cheaper to do so. If you know the immediate area and exercise due care, you can use Priceline or Travelocity or fill-in-the-blank and pay much less per night than even the least-expensive on-site hotel. Some of the least-expensive hotels around are among the closest to WDW. If we had a lot more money, we might stay on-property more often -- I dream of staying at the Contemporary or the Polynesian -- but we don't. At worst, we can afford to stay an additional night or two by staying off-site, and the only thing we really miss is EMH. The Disney "atmosphere" is nice, and the service is great, but we really tend to use our hotel room to sleep and shower -- it's just a staging area. Whether others see it as worth it would depend on what you expect/demand from a hotel. We'd rather spend our money on having fun in the parks and sometimes on food on-property -- and we enjoy visiting the resorts at which we usually cannot afford to stay (or choose not to spend the money).
There's no doubt that staying on-site has its perks, and it has been a pleasant experience every time we've done it -- and once or twice we've removed an off-site place from our "stay here again" list because we've had a less-than-stellar experience, but those have been few and far between; none of those came anywhere close to "ruining" a trip. We roll with the punches and have found those punches to have little force behind them, in the grand scheme of things. We remember the good things from our trips much more than the bad!
Scott