It has nothing to do with size, Brassaï used a plate camera, I don't think its the size rather the ubiquity of cameras. Some early photographers took pictures of kids acting up for the camera; those kids probably rarely had pictures taken of them-they wanted to be in the frame.
Taking images was unusual people didn't own camera's having their image recorded seemed unusual to them—interesting even...
Come forward 70 years and everyone has a camera they always carry, so now rather than unusual it becomes intrusive so attitudes have changed.
If it was size related then tiny cameras in phones would matter less than Leicas, its about the change in attitude of the subjects not the smaller cameras.
I'd like to back this sentiment up from my own experiences of seeing others photographing 'street.'
I've seen photographers working quickly and discreetly (not stealthily, simply not getting in the way) with the top range Nikon and Canons in a manner no different from those sainted for their use of small "discreet & stealthy" cameras. I've also seen the "stick it in their face and rely on AF" brigade at work too - which can as easily include those using zone focusing and a rangefinder. Its more about, suddenly remembering to put in the obligatory "IMO," the attitude of the photographer (not the more recent kind of 'showing attitude,' but your position regarding your subject/topic.)
That being said, in terms of answering in some vague way the question asked, I'd suggest a large part of the 'problem' is simply numbers. The sheer number of people out there doing it. I've never known a photography fad, if I can get away with calling it that, so popular and totally global. I've just been working in Lisbon and then Seville and felt surrounded by street photographers every day (to the point I wondered if the RFF EuroMeet was early this year,) London is always full of them...
us 😱 and I end up nodding, smiling or chatting to others far more often than I'd actually like. So perhaps people just get peeved when it feels like
everyone is out to fit you into their creative little frame. Add to that the fact that the more people doing it will inevitably mean more people will do it badly; insofar as their bad practice, lack of basic care/manners or even simply the fear that they're doing something wrong will lead to a poor experience for the subject.
Perhaps like a landscape photographer won't litter the Natural Wonder they've just photographed a street photographer should respect their chosen subject in a similar way - though in such vast numbers its like replacing one landscape photographer with a coach full of tourists at a viewpoint and I imagine we've all had a coach pull up and shatter that all too brief moment alone with a beautiful view.
There will be a myriad of answers shedding some light onto this on following pages I'm sure. Which will be interesting as this is a very good question and something I think everyone who considers themselves this type of photographer or who enjoys street photography must think about.