Yes, you are right. Filters freely pass their own colors, while rejecting the complementary color. Yellow and blue are complementary, so the filter will hold back the blue sky, making the negative lighter, and therefore the print will be darker. But if the sky is gray, there is nothing to hold back, so the filter will just reduce the exposure somewhat. You said it--a neutral density filter as far as the sky goes. Now take the example Robin used, above. Say the building walls have a yellowy color--maybe a warm, sandy color. The filter will do little to change that, since it is yellow. But anything blue in the shot will get held back--i.e., darkened. If that is what you wanted, fine. If not, you would have been better off without a filter.
what about colors other than blue and yellow? What should you do if you want to lighten a green building against a blue sky? Answer: use a green filter. It will freely pass the green color. And since blue and green are split complements, it will partially hold back the sky. Good enough. what if you wanted to darken the green building? A blue filter will darken green--split complements again--and lighten the sky.
The best rule is, if you don't know why you are using a filter in a given shot, don't use it. If not sure, take a backup shot without the filter. I don't use filters on overcast days. In fact, with modern films, I find that I am getting adequately dark sky with no filter, on clear days. That's because modern films no longer have the excessive blue sensitivity of older films.
Looking through a filter is only a rough guide, since film does not respond to colors the same way the eye does. But it's better than nothing.