OK let's talk. My use and researches have led me to these convictions/conclusions:
Ai-S Nikkor 20mm f/3.5 -- I love this lens and love love love this lens. It's got corner issues as do most (except Zeiss) 20/21 mm lenses but they never bother me. Wonderful, small light. Many people swear by the f/2.8 but I doubt it is significantly better, it will cost you more, and it's bigger.
Ai or Ai-S Nikkor 24mm f/2.8 By all reports -- and I do mean ALL -- the f/2.0 version should be avoided. I have had no few than five copies of the preferred f/2.8 lens. All were unimpressive, soft, low contrast, unintersting flat rendering. Has a greater tendency toward haze and other problems than any other Nikkor I've seen and bought and sold and worked with. If you want this focal length, go with the older (cheaper) Zeiss Distagon 25/2.8. I had this lens in C/Y mount. It's a kiiller. Will cost you more than 2x what the Nikon costs.
Ai-S Nikkor 28mm f/2.8 As all say, a beautiful lens. As a few will admit, it's much more beautiful close than at infinity. But it's not bad at infinity -- just not superb. The 28mm f/2 (Ai or Ai-S) which I have is better but on those closer shots, just not as dazzling. In any case both are superb lenses. VERY IMPORTANT: do not buy the Ai version of the 28/2.8 it's a totally different design and not well thought of by anyone knowledgeable that I've ever read. If you want to see what a 28mm looks like in street photography check out Garry Winogrand's stuff online. He used that focal length quite a bit.
It's like this: with the 40mm you'll want either 20mm -- 28mm -- 40mm or you'll want 24(25)mm -- 40mm. If you can afford the Zeiss ($600-700 used) then the latter is simpler certainly. If not, you'll get far far better image quality with the 20/3.5 and 28mm combination.
Do do do pick up an Ai or Ai-S 105/2.5 at some point. It's simply the most beautiful rendering lens I've ever used. It improves the world. It is sharp but not THE SHARPEST. It's just beautiful, and that's enough.
Other greats:
Ai 50mm f/2. Nikon's greatest 50mm. World class. there is no Ai-S version. Second best; Ai-S Nikkor 50mm f/1.2. Since you have a 40mm this is not really a necessary focal length for you. HOWEVER you'll not believe the sharpness and beauty of the 55mm macros, either older f/3.5 or newer f/2.8. they've gone up in price in recent years. That is a focal length of more use to you, when 40 feels a little wide.
85/1.4 ($$$) and 85/2
135/2 and 2.8
180/2.8 ED versions only
300/f/4.5 ED versions only
Zooms:
My first fave is an AF-D (auto focus) lens, which can feel a little strange on mf film cameras, though they're perfectly usable. The AF 28-105mm f/3.5-4.5D Really sharp, and very good macro function from 50mm and up.
Ai Zoom Nikkor 80-200mm f/4.5. Just beautiful.
The most reliable and interesting web site reviewing the older Nikkors is Bjorn Rorslett's, to be found here:
http://www.naturfotograf.com/lens_surv.html
to check serial numbers, versions, design, dates, this is an irreplaceable site:
http://www.photosynthesis.co.nz/nikon/lenses.html
Enjoy that FM3A. It's the culmination of Nikon's smaller SLR design and a truly great camera. If you find you don't like, send it to me. I sold mine recently and regret it.