Like you, if I'm wearing contacts I then need reading glasses for close to vision, or I need to remove my perscription glasses if I'm wearing those. The trouble I have with wearing glasses when photographing, is I hate not being able to see the whole VF, so I end up with my glasses pushed up against my face then I am constantly cleaning them.
I've found a solution that works for me at any rate, give it a try I promise you will get used to it quite quickly. Just wear the one contact lens in your shooting eye, in my case my right, the left eye then does all the close reading work. Your normal non photography vision is compromised, but not as badly as you might think, it's not something I do if I was just out and about with a camera, but if I'm working in a studio or shooting a wedding, even for a portrait session I would find it difficult to work any other way now.