I think we all find our own balance when it comes to size vs performance.
Back in the late 2000s, I carried a 5D Mark II + 35L everywhere. This came after I only carried small sensor cameras like the Canon G7 or G10, and I wanted to take better quality images of my girlfriend of the time, so I resolved to carry the best quality image maker possible. This later gave way to the M9, and my back thanked me.
For me, finding the right combination of size and performance has been a matter of budget and opportunity. I love tiny pocket cameras, and my EDC is the Sony RX0. It goes with me everywhere. Next up from that is the miniscule Panasonic GM1, which is smaller than a deck of cards and takes equally small m43 primes. For everyday shooting, this is a perfectly acceptable combination. When I want something more substantial, I use my M9 + a smallish prime, or the Panasonic G9. I'd love to try a Leica Q or Sony RX1r, but my budget hasn't allowed for that for a while.
My recent purchase of the Panasonic S5 for work and personal use has lead me to experiment with small RF lenses and larger manual focus SLR lenses, as well as Canon and Sigma EF lenses on the same body. The Summicron M 50 gives me awesome image quality in a eensy weensy lens, albeit with a MFD of 70cm. If I want more reach, there's the Summarit M 75. I love the feeling of Minolta MC and MD primes, but along with the adapter, the package gets bigger and bigger. The Tele 135mm f2.8 gains an inch in length with the adapter. The camera itself is about as large as what I would want for everyday carry.
Some time in the future, I will probably invest in the Sigma 24/3.5, 35/2 and 65/2. This will give me a matching set of compact primes which can be used for paid and personal work. I already use the Sigma 24-105 adapted to the S5 for work, and it's just too much for a walkaround lens.