Do you seriously think people do not double check minority men ALL THE TIME? Surveys upon surveys have shown active discrimination (intentionally discriminating), let alone unconscious bias (double checking to make sure they aren’t sexist). I’m talking about unconscious bias here, bias I have said I fight.
Also, I have only ever been harassed/assaulted by white men who look a certain way (suited professionals in D.C. with a certain haircut, look, and attitude). So whether or not you think that’s justifiable, it’s certainly understandable why I’d be more afraid of them.