You neighbor is the big problem, more than the birds. Yes, birds are going to land on your fence and childrens' playthings regardless of where they feed. Even if they're not feeding, birds fly around and perch wherever they want. Its not a case of where the feeders are located - front yard, back yard, side yard, or down the street. They can pick up seeds from one yard and enjoy the munch in yours. Yes, we have the feeders and birdbaths, too, that even the squirrels, skunks, raccoons, and woodchucks enjoy, and we live in suburbia. Fences and swingsets and other toys can be hosed off for sanitary purposes. Your problem is the neighbor. You asked him in a nice way to relocate his feeders and through no fault of your own war ensued. He's doing this to bully, intimidate, and harrass you, and you can take legal action if he's destroying your property by drilling feeders, etc. into your stockade fence.
Wildlife is going to wander through your yard regardless of whether or not you, your neighbors, or the guy down the street feeds them. They're all in search of food and they migrate. Putting up a fence isn't going to stop the animals (winged and 4-legged) but the numbnuts behind you doesn't appear to be letting up anytime soon. Call your lawyer and have him/her send a letter.