Late to the party but I love my CLE. I have pretty huge hands and I didn't mind the size of it. But I recently got the grip that was made for it and makes the handling even better.
In my opinion, if you are looking for a camera to shoot on the street, this is probably the best rangefinder you can get. It is small and discrete, 40mm works great, usually you stop down so short rangefinder base length doesn't matter as much, accurate metering.
I have an M2 and I can't get over its size. If I'm shooting 35mm I want a small camera! The Hexar RF is the same, I really wanted to like it but it's even bigger than the M2. I probably have an odd obsession with having small rangefinders.
I think the CLE kinda gets a bad reputation because the CL wasn't that reliable. While the CLE is electronic, I really think people overestimate the reliability of the camera.
My complaints about the CLE? No metering during manual exposure (why!?). I prefer the shutter speed dial on the CL. Shutter speed dial locks when changing from aperture priority to manual, especially annoying since you'll have to switch between it if you want to meter. No shutter speed read out in the viewfinder (why?!)
I actually really like the CL but the one I have has a pretty bad viewfinder relative to the CL, the meter on mine doesn't work, and the slow shutter speeds are off. But honestly, it's probably just as reliable as the M5. People just don't want to spend $200 on a CLA of a $200 camera.