I did that in a July, many years ago. I don't know what my problem is, not being prepared for vacations (see my response to hiking the Grand Canyon), but it was hot as hell in Durango and then when we got into the mountains (duh!) it was so bitterly cold I nearly died. We had an open car so I froze my rear end off. But it would have been lovely and the scenery a lot nicer had I been better prepared. Silverton is such a cute town. And they got most of my money because I bought a parka for the way home. Also, the heights are quite frightening!
OP, we are planning a trip to CO to visit family with our kids, who are just a little older than yours. So I am interested in hearing responses, too. We are going to Denver area and were thinking of doing a day trip to the Royal Gorge this time. ETA: I will bundle up.