I think a lot of this goes back to teaching our kids good manners. Kids always used to knock on your door if they hit a ball into your yard (at least if it was enclosed). They were supposed to get permission first to retrieve their ball. It was just polite.
Now, if you're okay with your kids playing anywhere in the neighborhood, perhaps they should ask the neighbors if it is okay first. If the neighbor isn't bothered, then fine. If the neighbor would rather you didn't at all or at certain times, that should be respected. It doesn't mean they are trying to be mean. Some people worry more than others about accidents and some people are just quieter or like to keep to themselves more.
I'm not an assertive person, and I would feel awkward going out into my own yard or driveway with my two young dds and saying, Uh, excuse me, neighbor kid, can we play with our own swingset/toys - or having to make sure they didn't play too rough with my little kids or break things we have bought.
Personally, I wouldn't want to feel like I had to watch someone else's kids either. Now if it were a friend of my kid and their parent knew they were there, that's different.
I don't think it's fair to assume the entire neighborhood should be your child's playground. If you are one of the neighbors who is okay with sharing your yard, I think that's very nice too. It's just not what everyone wants. Different strokes for different folks.