Louisa...these day's you cannot be too careful...EVER. You did right in calling your husband...and I would not drop this guy a note...I would not give this guy the time of day. When you're alone like that...if you have to drive around the block and let your husband come out to the car to get you.
I had a man get mad at me once (road rage...because I would not pass a guy on a bicycle while on a small bridge. I waited until it was safe and then passed him. The German man was so upset because I slowed down that after I passed the guy on the bicycle he pull up alonside me and began to tell me how I should drive). He asked me why the (BLEP)didn't I pass the man? (Okay...I am normally a calm person until someone upsets me behind the wheel)...I looked at the man as if he'd lost his mind and said, "Because it was (Bleeping) unsafe!" He then pulled in front of me and slammed on his brakes. I went around him and turned to go towards my village. Then he gunned his car and sped past me again and tried to stop me again...when I would not stop he followed me through the village. I decided to head towards the police station on the other side of the village but halfway through town the guy turned off onto another street. That night I parked my car in the garage in case the man tried to see if he could find me.
I'm tell you ladies, you should take no chances with any man...stranger or known neighbor who acts weird on you.

Mustang Gal...if it were me and I had a neighbor like you do and he tried to force me to do anything (referring here to him forcing you down on the bench), I'd tell him if he touched me again I was calling the police. You don't have to talk to anyone you don't want to, especially a man like that.