I believe every Leica shooter should have one fast lens to take advantage of the available light ability of the M rangefinder. For me that is the 50mm Summilux ASPH. I used the 35mm Summilux ASPH for a year or so and it is optically a great lens. Very versatile too. I goot rid of it simply because I'm a 50mm guy. Part of me regrets the loss, but I'm a bit of a minimalist when it comes to Leica equipment -- 2 M bodies and 3 lenses (35mm Summicron ASPH; 50mm Summilux ASPH; 90mm Elmarit).
Ask yourself what type of shooting do you do? If it's all outside on a sunny day, then you don't need the extra speed.
But getting a fast lens opens up more shooting opportunities. You may not shoot inside or at night much now, but if ypou had something like the 35mm Summilux ASPH, then maybe you would. It is a near perfect one camera one lens outfit where you can take it and almost shoot anywhere.