You could do that--I have a goggled Summaron with my M3, and it's not that big a kludge, although certainly not as nice as the unadulterated viewfinder.
There are a couple of other options, though. One is to buy an external 35mm viewfinder, by Leica or another manufacturer. The Leica is most expensive, but is brilliant indeed--very bright and clear; the Zeiss would be next most costly, and of course Voigtlander has (or at least had) some available which are certainly nice, and reasonably priced. And there are no doubt loads of old finders by various other makers, but with few exceptions not as nice as those mentioned above (well, except maybe for the old Canon external finder).
Another possibility is to cover the frameline illumination window with a bit of black tape and then use the whole viewfinder, which is roughly the same field of view as 35mm. There won't be any framelines, but it works reasonably well in most cases.
(One problem with the goggles is that the round window is actually a prism, and it's secured in place by three screws that can be adjusted to change its orientation. If it comes out of alignment, meaning your rangefinder image will be out of vertical alignment and/or may be slightly off as far as distance, it can be a real dickens to get back in alignment--and to keep it there. Mine did that, and I sent it off to a repairer who got it back in alignment three times--and three times, within a short period, it came back out. I finally sent it to Malcolm Taylor, the elder god of Leica repairers, who used extra long screws and a bit of lacquer to keep it where it belonged. But it still seems to drift a bit with temperature changes, on occasion. Even Malcolm said he couldn't swear it would stay in alignment forever, certainly. So that is one aggravation with the goggles to keep in mind.)