It so isn't the fence owner's responsibility to maintain the yard on the other side of a fence if it's not their property. If there was no fence, would you be weed-whacking a couple of inches into their yard "just because". Of course not.
I have had fenced-in yards all my life, with the fence right on the property line. We maintained our side, the neighbor maintained theirs. Where the heck else are you supposed to put a fence other than on the property line? The point of a fence, generally, is to delineate your property.
Bottom line...neighbors are lazy, don't feel like properly maintaining their yard...their problem, not the OP's..
I've always read that if you put up a fence, you should put it a few feet within your property. That way, if there's any error in surveying, measuring etc, you have a little wiggle room. If it's right on the line, and there's an error, your neighbor has the right to tell you to move the fence off their property. Talk about expense and hassle to have to take down the fence.