Contact the police/ local FBI agency (should the PD blow you off), file a restraining order, get a concealed weapons permit and carry your gun on your person should the incident turn worse case scenario. By no means, am I trying to shock value this but with all the rampages going on in the media lately, your ex co-worker may start to get ideas to be a copy cat. This is a very serious matter and something tells me this guy is doing more stalking and harassing than you may suspect. I don't think it's a coincidence that he shows up, when everyone is out of the office but a few ladies. I seriously hope this works out for you and this guy goes away. Please be careful.
We were all at a conference that he knew the dates of.