Darthfeeble
But you can call me Steve
As a die hard Fuji fan and a past Leica user... I would never be happy using Leica lenses on a Fuji. The RF of a Leica is my favorite manual focus aid and I feel the digital counterparts in the Fujis doesn't even come close. I'd honestly figure out how to keep both.
I'll give this a second. I have the Fuji adapter and all it does is make it quick to get in the menu so if you need to go to M lenses, just get a decent third party adapter. As jsrocket notes, the focus experience isn't the same. I'd love to use my M lenses on the Fuji but it just doesn't seem to work for me. Focus peaking isn't that good, it doesn't snap into focus and over all isn't an easy way to get the focus. As noted, an XE1 can be had pretty cheaply and the 35/2 is a reasonably priced lens. All this said, still keep the M4. You'll hate yourself for selling it.