I am working for the Census this time around and can tell you that
I have an entire street of people who say they sent theirs in. Know what? I still have to ask every single household the questions. A lot of things could have happened to account for the mix-ups. The common thread I have found is that they all had PO boxes so I'm thinking they put that address in address column INSTEAD of their physical one.

Luckily, they have all been very nice about answering the questions. And we do all carry ID and there is a phone number on it that you can call to verify the census taker's identity. As for giving our your neighbor's info, that is usually only asked as a last resort kind of thing. We are only allowed to visit a limited number of times in person to try to get the info, so any we can gather on that last trip is good. Also neighbors can be used if the house is vacant. We can't just look at a place and record it as vacant based on our own observations. It has to be verified by a proxy(the neighbor or apt. manager, etc.).