Photography is an abstraction of the real world. It compresses 3 dimensions into 2 (let's forget about 3D for the moment). B&W is an abstraction of the visible spectrum into monochrome tonality. Painting and sculpture are similarly abstract.
If color media speaks to you more than B&W, that's OK. There are many who feel the same. There are people who wouldn't watch a B&W film. Some will not watch a foreign language film with subtitles. Some people don't like opera or musical theatre. Some people are excited about 3D video. (I'm not one of them.)
All art forms, or media, have some sort of constraints. None of them are intended to be, or capable of being a substitute for "experience." They are all stylized or abstracted representations. Some can be very powerful. Some may even be more powerful than primary "experience." But they are not the same as "experience." A person either accepts the conceit of an art form, or he doesn't, and it will be impossible to appreciate the art if you don't.
There's nothing wrong with a preference for color photography. But I think you're missing something.