You'd have to concede that men should never have acted in a way that instilled that fear though. We all know no man ever HAD to be a brute, a lout or a rapist. You cannot seriously argue that the feat women feel around men I'd unjustified.
Why not work to get rid of the valid causes, simply because that's what decent people would naturally choose to do?
Why do you fear the idea of trying to address this?