Kat,
Firstly - if there is a shot that you cannot miss in the bit that was exposed at ISO100, then develop the film at ISO100 and be damned to the first 20 shots.
Other than that - it would almost have been better to keep going at ISO400 till the end of the roll. What you would have done is effectively underexpose every shot by two stops. Then you could go to the lab and ask them to overdevelopthe roll by two stops (normally called 'pushing' by two stops). This would get all the negs out at the correct density, though perhaps with some loss of contrast or sharpness.
However, you now have some shots at 100 and some at 400. You could have the roll developed normally, and the shots at 400 would be too dark, or have it push-processed to 400, and then the shots at 100 would be too light. You're lucky in that traditional B&W is pretty tolerant to over- or under-exposure; I think I'd suggest developing at ISO400 and the shots at ISO100 will probably be fine.
Or you could split the difference, and develop at ISO200 (ie pushing by one stop). What do others think?
Jamie
edit - oops, stepping on Clarence's toes there! Kat - have you exposed any of the film at ISO100? If not, do just as Clarence says.