I can't speak for the xe2. I had an xe1 and xpro1. I got them originally for manual focus of legacy lenses. I found the xe1 especially bad for reasons others have given. But the xpro1 was also a struggle and quite frustrating - boils down to the fact that you can't focus legacy lenses with the OVF and the other options are horrible (EVF, LCD - I don't see the point of buying a camera like this and zone focusing).
I ended up buying 18 and 27 Fuji lenses, and at that point the xpro1 really shines using AF with the OVF . I didn't have an issue with focus speed or accuracy, with the latest firmware.
I think the main problem with the xpro1 is likely to be finding one that isn't well worn at this point.
If I was mainly into manual focus legacy lenses, I'd go for an xt of some sort (much bigger/better viewfinder) or an a7 (which is what I did).
If I wanted a cheap RF-like experience in digital, I'd go xpro1 if I could find a decent one, or xpro2. There's also the X100 series of course, but they don't scream rangefinder as much as the xpro1 does.