I may get flamed, but I am a stickler for this too. I want to be asked just once, it is about respect for me.
I lived next door to a family that put a basketball court in thier back yard, right next to my bedroom window. That would be okay if they were not up at 6:30 AM playing basketball every single day and until 10 PM at night, when I had an INFANT!!! They were within the city ordinance, but they didn't have a shred of common decency. So when they came over the fence (without permission, of course), painted my grass white, left things on my property and boxed on thier front lawn, I was so sleep deprived and upset that I took pictures of everything.
All it would have taken to get on the right foot would have been for them to come to my door once and ask to get thier ball with a respectful tone in their voices. I was worse than your neighbor, I used to scream at the kids. We moved, or I would be in jail for assault I am sure.
ETA on the fence thing, I don't think that he can force you to put a fence up, he can do it on his side of the property line. What I considered doing because I didn't have the money for a fence was to put up a tall hedge. Though I sympathize with the old man, he is responsible for putting up a fence.