If you are paying yourself back with interest I see no problem in borrowing the portion that is invested in conservative funds. The borrowing I think we need to see tightened is the Federal Gov's borrowing of Social Security.
The problem with this idea, as far as I know, is that most plans don't allow you to drain one of your investment options for a loan (though thinking about it, I guess you could rebalance after the loan was funded).
I've come back around somewhat about 401K loans: I used to be somewhat for them - having good luck myself, but when you look at it historically, they are a bad idea from a cost/benefit risk/return basis.
I got lucky, that the 401K loan I did to help furnish our 2nd house was money pulled out of aggressive growth funds a few months before the dot com bust. In the long run, those funds (which were about 20% of my total 401K at the time) did better over the next 3 years than the money in the market did.
