I've had it happen a few times. The most memorable was several years ago to the day. It was Halloween afternoon, and I was traveling through the small South Carolina city of Chester. I decided to try a street parallel to the main road through town. I spotted an old house in a sad state of repair, and stopped to photograph it.
I was on the sidewalk, just snapping away when a woman drove up in a white Oldsmobile. She got out of her car and demanded, "What are you doing?" Like the OP I was of two minds. The smartarsed side of my brain was coming up with all sorts of witty and clever things. I didn't listen to this imp of the perverse, and gave what I normally do in this sort of situation (the simplest of answers) and said, "I'm taking pictures of this house." Well, this answer didn't satisfy her, and she next wanted to know "why are you taking pictures of this house?" To which I replied that it seemed interesting given that this was Halloween day.
Next, Oldsmobile lady told me that the house was her son's. She went on, and on, about this being her son's house, and by the way she sounded I really began to wonder if her son was still amongst the living. I wondered if I was going to find evidence of the paranormal in the images. I assured the woman that I simply found the house visually interesting, and further resisted the imp. She then left.
No less than five minutes later another car pulls up. I man emerges and asks "why are you taking photos of my house?" I answer him, "Oh, hello, I just met your mom" while ignoring the strong desire to say "hey man, glad see that you're alive." House owner asked me next if I was from the city. I told him, "no, I don't live in your town." The fellow repeated the question three more times as we talked.
It turned out that the guy was afraid that his city was going to declare his house a public danger and have it torn down. House owner told me he was going to restore the old structure to its former glory. Not being an expert in such matters it seemed that restoration might cost more than the place was worth, and the impish side of my brain kept running the chorus of "Tobacco Road" through my ears -- 'bring dynamite and a crane, tear it down and start again'. However, the fellow was still wound a bit tight so I just wished him good luck on the restoration and said, "Dude, you do know it's Halloween and you might want to get some plywood for the porch. You don't want any kids disappearing though those holes."