Okay, I'm female, but I haven't done as much street shooting as I'd like. Small town and all that. Kinda sucks.
Anyway, I find that, probably because I *do* live in a small town, people know me and question why I'm taking pictures more often. Half the cops in this town know who I am, half the citizens know me, and the rest have, I'm pretty sure, at least seen mme around. They usually just ask the harmless questions: what are you shooting, why are you shooting, what are you doing with the photos, etc. I have had just a couple people give me a dirty look.
When I go out of town and take the camera, I don't really have that same problem with recognition. I can go out and take photos anonymously. I do see that most people leave me alone in this instance. People see my face in certain places a lot, but they don't often question what I'm doing. There's no police coming up to me and asking what I'm doing or why I'm doing it...I think that's because it's Iowa and none of us here are scared s!@#less about terrorism even though it could happen here as well.
I think it depends more on the area you're shooting in than your sex as to whether or not you get trouble. Now, if I was shooting in a park, maybe...but I hardly ever do that.