I'm with ZeissFan: keep the M4 (seriously...I don't want you coming back here, a year or two from now, whining "What the (expletive deleted) was I thinking!?" (BTW, the expletives get more explicit where the sale of a Leica is concerned...just thought you'd want to know.)
So, here's my idea
1) Go buy that XA. Shouldn't have to sell anything to snag one.
2) Buy a few rolls of Kodak Ektar 100
3) Buy a few rolls of (a) Ilford FP4, (b) Kodak Plus-X, (c) Ilford XP-2 Super.
4) Pretend the cameras you're holding are big, ponderous medium-format numbers. Handle accordingly. Don't rush your way through your shots.
5) Have them developed carefully. Get decent scans from the lab, or scan 'em yourself.
6) Evaluate.
As I see it, the problem people have with 35mm is in the way the format is approached: its principal forté (at least until digital capture came along) has long been speed. Quality can be had, but you had to (and still need to) pay attention to detail, right down to how you handle the thing. Big MF cameras pretty much force you to slow down and pay more attention to minutiae, whether you're in the mood for it or not. 35mm encourages serendipitous, happy-go-lucky behavior on the photographer's part (not that I'd have it any other way, but hopefully you grok my point).
Up to a certain point, format counts, but not always as much as one might think.
- Barrett