I've used the Voigtlander 40/1.4 and the 28/1.9 with both the X-Pro1 and X-E1. Image quality is excellent with any of these combinations.. no complaints at all.
Now that Fuji has implemented their focus peaking, I had zero problems focusing the lenses in any kind of light. I also used the Hawk's helicoid adapter that allowed for very close focusing. The Hawk's adapter isn't any bigger or heavier than Fuji's M-to-X adapter.
I've also been using Nikkor SLR lenses in combination with the Metabones Speed Booster combination. Same comments as above... excellent image quality.
The only issue I have with use of legacy lenses is, as you know, that you are then forced to use the electronic viewfinder or LCD. The optical viewfinder of the X-Pro1 is much less useful, although for zone focusing the wide-angle lenses its pretty good. I found the X-Pro1 framelines somewhat difficult to get a perfect match to the lens focal length. It takes a bit of trial-and-error to get them close. For example, when using the CV 28mm lens, I'd have to set the X-Pro1 framelines somewhere around 23mm for them to be a pretty good match.
For walk-around lenses, I've gone back to the Fuji AF lenses. But for slow, considered photos I use the Nikkor with the Metabones adapter. I'd still have the CV lenses as well, but I can't afford all these different sets of lenses. And I can buy two or three nice Nikkor NAi or Ai lenses for the price of one CV lens. 🙂