I'm not terribly fast with RF focus in relatively close (say, 35mm in a standard sized room), even the X100's slow AF is faster most of the time, so I sympathize. But I imagine that's a practice, practice, practice issue.
My problem with SLRs, aside from size, is AF inaccuracy. Seems that every contemporary lens (particularly consumer-grade but also pro) has to be calibrated to each body. I've only had two exceptions - my original 5D and 35L and then the D700/24-70G. Both were great, great combinations, if you didn't mind carrying a whole lotta weight everywhere. Sold the 24-70G, got a 50/1.4G that never seemed right AF-wise, even at f/2.8 and was too inaccurate to use at f/1.4 (which made the lens itself pointless).
Granted, I've heard that the M9 has issues with minute RF misalignment as well, to a degree that may not have been noticeable with film.