The goggles don't only adapt the view through the viewfinder, the goggled lenses have a different cam build. The rangefinder cam is also adapted to the different view through the goggles.
So a goggled lens can't be properly used without goggles. At infinity everything is sharp, so the closer the focus, the bigger the difference.
With the DR, things are a little different. The DR has a Dual Range cam. The rangefinder cam in most lenses is a tube. In the DR it is also a tube, but one half of the diameter sticks deeper into the body. The DR glasses are only mounted when the close focus bit of the cam is used.
Longer focus lenses can be used on the R-D1, but focussing and framing becomes increasingly difficult. I can't wait on getting mine back from Steve and try the 200mm Komura...