I've carried most types of digital cameras for everyday purposes, and as much as I like the handling of larger cameras, the smaller the better. Over the years, I went from 5D Mark II to the Sony RX0, which is the size of the Canon's battery. The tradeoff in image quality and versatility is worth it as I really enjoy the small size and unobtrusiveness.
In the past, the Ricoh GRD III was my everyday camera, and I even dreamed about using it on multiple occasions. The aps-c GR succeeded the GRD III, but now both of them are unfortunately malfunctioning. Then it was the Panasonic LX7, the LX10, then finally the RX0. I still carry the LX10 if I think I'll want zoom, but the RX0 does just about everything I want in a pocket sized everyday camera. Way better image quality than a smartphone in the same size as a DSLR battery.
As for whether you 'need' an everyday camera that goes in a pocket, why not? Find a camera which is the best compromise of image quality, operational speed, carryability and stealth. Buy it.