I have observed the difference with several of my Nikkor lenses and my M3. On cameras with a shorter EBL it matters much less.
Obviously you know your gear best, however this is still anectodal evidence in a sample of size 1 with a lot of variables.
Everybody including me is aware that there are quality differences between adapter brands, and I'm not going to argue that no-name $3 Chinese adapters made from recycled Coke cans are as consistent as Leitz adapters.
But at the high end of the scale, which is what we're talking about here, I don't think you can draw the conclusion that CV adapters are
generally worse than Leitz adapters. It may be the case with your particular body/lens/adapter combination. I've just done a number of quick searches here on RFF, and from that I can't see either that Voigtländer adapters would have a
consistent bad rap. Maybe you've just got a bad adapter - that can happen, but that's what warranties are for!
As my equipment list might demonstrate, I am quite un-emotional about equipment manufacturer
But then I'm not saying that. All I'm saying is that one shouldn't overestimate differences between high-end manufacturers of precision goods (and CV certainly is one in comparison with other adapter makers) with what is essentially a very primitive product.
You look at the back of a used Leitz adapter, and you typically see brassing, which might cause the difference.
I hope it doesn't! If it does, it means the adapter was too thick when it left the factory.
Maybe Leitz used to make them too thick and then ground them down individually to proper thickness, it sounds like the sort of thing they might have done with their "adjust-to-fit" attitude, before the 1970's, anyway. But I somehow don't think so.
Wear on the contact surfaces
is one of the theoretical problems with most bayonet and screw mounts (and there are mounts designed to avoid this, such as Canon FD or Pentacon Six), but I think you would have to change lenses pretty often to cause enough substantial wear that the focusing difference will actually be significant in comparison to plain old body and lens adjustment problems - and in this case you can just put a layer of paint on your adapter.
Philipp