All of the fudge recipes I've ever seen call for sweetened condensed milk, not evaporated.
That said, give it to me! I keep evaporated skim milk in my pantry at all times as I use it in any savory recipe that calls for cream. It gives you the same thick texture, but isn't nearly as rich or high-calorie. Miracle stuff, evaporated milk. Great cream sauces for pasta.