I agree with others you need to report him to the church elders or deacons. What if he tries to do this to teens in the church. I also wouldn't leave the church you did nothing wrong at all. It's really hard to find a good church home and it sounds like you have really liked this church just not this man.