Ethics and legality are often not the same matters.
I agree that "motive" is a critical component of the answer/question.
I completely disagree that 'anything in public is fair game' from an ethical perspective. People who say such things seem not to be able to imagine a situation where a human being can be exploited. And, that's where i draw my personal lines — on 'exploitation.'
I don't find it 'okay' to take photographs of people 'in public' when the objective is to make them look stupid/ugly/worthless/etc. That's just not how i 'get off.' I had a client once. An otherwise sophisticated, worldly man. And, when he took a picture, he only seemed to enjoy it if he 'caught someone looking foolish.' I considered that to be quite puerile. To each his own, but in that matter he lost esteem in my eyes.
There's a 'landmark' case currently, concerning a self-professed 'art photographer' who captured very intrusive images with a telephoto lens, peering into apartment windows across from his own building. While the subjects did not take enough precautions to ensure they were not visible from spying eyes/lenses, i still feel that to be exploitation. Others will differ and point to 'the letter of the law.' My sense is that ethics should be on a higher plane that the law provides. Otherwise, aren't you just doing what you can 'get away with?' That's not ethical.
This also reminds me of the Charlie Hebdo thing.... While the retribution was reprehensible, i also find that the French publishers were arrogant and fully instigated the dispute. There's a matter of RESPECT. And, suggesting that it is in their "rights" to publish such imagery is sorta beside the point. Frankly, i feel that anyone who believes those 'cartoons' had some sort of value is a bit of an idiot.... So, there's that.... And, while i'm virtually 'anti-religion,' i am 187% 'pro-respect.' It's one simple rule: don't publish images of that dude. You couldn't comply with that? Respect. It's simple. Back to this question. If you're photographing someone in a swimsuit, without their knowledge, and the image doesn't illustrate a respectfulness of that person, i kinda feel it's wrong. But, this brings into question more questions. What is respect? Objectifying a woman can be considered either vulgar ogling, or worshipping of beauty. Depends on the shooter, the subject, the resulting image, the motivation, and the dissemination of the image. All of those points have to be addressed, in every situation. Isn't it overly simplistic to think there's just one answer for all situations?