You didn't specify which X100 model you have. The older X100 is noted to have pretty poor AF performance compared to later models.
I have the wide and tele conversion lenses that I use with X100S. The lenses do add bulk to the camera but the wide model does this less than the tele. That's something that may be important to the user. You don't lose much, if anything, optically compared to the built-in 23/2 lens alone. Like the bare 23mm, one or two stops down from maximum aperture has best performance. AF in the X100S is not the world's fastest but it's not terrible either. I can't see any difference in the speed with the conversion lenses. I don't think the X-E1 is noted for its AF either. The X-E2, updated with current firmware, has essentially the same performance as the X-E2S which is very good overall. While the 18mm is an older design and catches a lot of criticism, it's actually pretty good. It loses sharpness along the edges but very sharp in the center, even at maximum aperture, and AF is also pretty fast and accurate.
Personally, I would pick the X-E2 with 18/2 for the types of photography indicated. I love my X100S and carry it a lot but I only use the conversion lenses for those times when I must limit the amount of gear I'm carrying.