Interesting--trouble focusing if the M-3 VF adapter is removed? The RF linkage on Leicas is internal + mechanical. The viewfinder and bright-lines are optical. The focus is a relation between the RF cam within the camera and the RF coupling within the lens. The 35mm VF adapter for M-3's doesn't affect the focusing mechanism the way it does on a 50mm DR Summicron.
If the lens is focusing properly with one camera, it should do so on another-with or without the optical VF adapter. I have an M-3 and a CL. I use one mid-sixties E.Leitz Canada (chrome) 35mm f/2.0 Summicron on both.
I removed M-3 adapter ("eyes") to use lens on my CL. I use auxilliary 35mm VF clipped in the shoe. The results: Focus is spot-on with or without adapter on M-3. I verified focus with GG on film plane w/ 8X Agfa loupe.
Results on CL indicate no issue with accuracy of rangefinder-50mm frame pops up, which I ignore, and I don't think CL has any paralax correction. I use auxilliary VF anyway.
Accuracy of focus is fine, although distance scales reads out differently on the two cameras. CL indicates in focus at 15 feet, M-3 indicates in focus for same point is 10 feet. However, checked on GG at film plane, when RF indicates it's sharp, it is.
Perhaps there's a calibration issue in the RF mechanism? I'm also pretty sure the magnification of the M-4 VF is different then the M-3 VF, so angle of view will be off with the M-3 adapter used on M-4.