Typically the thread that will last for dozens of pages without any definite answer or sensible advice about the behaviour to have in such casual circumstances.
This happened to me as well, and to many people I know too. As soon as you want to shoot photos in the streets or at any public place, displaying strangers as those people being one important part of your composition, this is going to happen to you at least once.
Either you accept that risk, or you shoot your own family in front of a bricks wall, or flowers in a wild forest, or whatever, away from any human figure.
Talking to those other people (one would notice that the people you shoot seldom react themselves, the overreaction is almost always coming from third characters witnessing) trying to have you stop taking pictures is worthless believe me.
So, yes the best thing is to move along or at least telling those people to get concerned with their own business not yours.
Worse case, you will get hurt or your camera will end its journey in the sea if you get accross particularly crazy characters. But this is only at this point that you'll need to speak about your rights as an amateur photographer taking candid shots in a public place and having no intention to publish the pictures on the Internet or in a newspaper.
