I've found the 35mm to be completely useable wide open, and only better as it is stopped down. The Fujinon also seems less prone to CA and terrible flaring when shot into harsh backlighting. As for IQ, it's easily on par with the best I've used from Nikon, Canon, and even Leica in the 50mm range, and packs a nice signature in its bokeh that helps give the image a nice sense of depth without being distracting.
Yes, the 35mm isn't an AF speed demon but then again, it certainly "feels" subjectively faster than my Canon 50 1.2L on a 1D mk iv, and yes it seems to hit proper focus quite accurately/consistently, which can't be said for the 50L (just like gavinlg pointed out)...so again not sure what others are comparing the 35mm's speed to but it isn't THAT bad in my opinion. Sure there are faster focusing AF lenses and camera systems but I don't think that the Fuji system is super slow as it's online reputation seems to imply...the jittering and focising inconsistencies of the 35mm disappeared like two firmware updates ago for me.
I took this shot at f/2, 10 second exposure for the blurred water, some insanely high ISO (3200 I think). See the light on the horizon? That was from the MOON, there were a few VERY distant arc sodium street lights directly behind me about 300 yards. And the X Pro's AF managed a lock on the subject on the third try using just the available light, which was so dim it barely registered at all in the long exposure. As a longtime Canon user I can tell you, it wouldn't have happened without a flashlight or a Canon Speedlite with AF assist.
Flare control, and ability to lock AF on a moving subject in low-light? If you get used to working with the camera, its possible. And at least it doesn't "hunt" the way the 50L does:
And for situations where you need AF accuracy on something moving quickly...well I just prefocused this shot using the yellow railing as a reference point. I'd do the same with Canon's latest, greatest, and fastest too if I wanted to be sure I'd nail this shot. Techique > Technology all day, everyday:
I've used the X Pro (supposedly one of the slowest-focusing X-cameras) to shoot fast, up close action by modifying my technique and prefocusing or whatever...which is also what I have to do with the 1D in situations where the 1D's AF falls short, like shooting through a fence or tracking a moving subject in front of the ocean on a sunny day. In other words, like every other camera system ever, of course there are caveats; the question isn't what are the drawbacks, it's which drawbacks you can live/shoot with, and which you cannot.
Hope this helps!