Aperture priority, entirely manual, entirely automatic -- if the camera's turned on and one of the Fuji X mount lenses is on the camera, it happens. The only way to make it not happen so far as I can tell is to turn the power save feature on, use the optical viewfinder only and turn the LCD off (if powersave & the LCD aren't off it'll make the skittering sound even if you are using the optical viewfinder).
Other unfortunate reality related to this -- when you're using the electronic viewfinder the aperture you're looking through is not necessessarily the aperture you've set it to.
I do not get this feature at all -- I see no real reason for it. I've spent a lot of time, now, with Leica M lenses on the Fuji and the EVF & LCD really adjust quite quickly to brightness changes and things look just fine with an M lens wide open in full sun. It is irksome, it makes manual focus unreliable (fortunately, so far as I can tell, the 35 doesn't show any evidence of focus shift) and I'm just not sure it does anything useful.
I really enjoy this camera and am not really a critic, but I would really like Fuji to give people an option to turn this feature off.