bgtc's comment is worth following up on. As things settle in SLRs, the mirror tends to get into dropping a bit farther through time (maybe from those thousands of little slaps at the mirror stop, year after year?) This causes back-focus as the dropped mirror forms the sharp image not quite up to the ground glass, which causes you to bump the focus back a hair to get the focused image that extra distance up to the GG. If you are consistently focusing a couple of inches behind where you intended, this is probably the problem.
A quick check is to take one of your wider lenses--maybe a 28mm--and try to focus on something that's genuinely at infinity, like a mile or more away. Does the microprism grid clear just as you touch the infinity stop? If so, OK there. If it wants you to keep focusing past infinity, probably the mirror needs to be looked at. On most cameras (I don't remember how my Olys were) there's a stop under/behind the mirror on the right side, and that is adjustable. Often the stop is an eccentric cam which needs to be turned up a bit, or sometimes (cough, cough: Hasselblad!) it's just a finger of metal that needs the tiniest bend upwards until infinity and infinity coincide. On the really junky nikons, a bit of tape (shim) on the back of the mirror where the stop touches seems to be the only cure.
For years I had one OM1 that back focused and I always blamed myself. It's only after I switched to Nikon that I found out how to fix that.