I have been a street shooter for many, many years. Fortunately, I have only a handful of unpleasant experiences, with most of those being the past couple of years. Usually, when confronted by someone about taking their picture, I smile, walk up to them, offer them my card and explain that I am a street shooter. Most of the time this works.
However, one of the more unusual confrontations I had came at a biker rally. I was shooting random people in a crowd when a tough looking biker approached me and asked if I had just shot him and a group of his friends. He pointed to his friends. I told him that I had shot them and he replied in the nicest way, "Hey, I hate to be an asshole about this, but do you think you could delete that photo? My friend over there is very, very paranoid about having his picture taken and he has a good reason for that."
I promptly deleted the picture. He thanked me and walked away. I saw him nod to his friends.
So, sometimes, people may really have a legitimate concern about having their picture taken in public. This example is the only time, when confronted by someone, that I felt as though the subject really had a reason to be concerned about me shooting his picture and where and how that picture would be shown.
Other than that, I have found that for the most part, your body language, your demeanor, and your verbal communication skills can go a long, long way in making people at ease while you are street shooting.