A is for 35mm film
B for 120mm, holds 2 6x6 frames with the holder
C is for 6x9, the largest size you can scan with the V500 using a holder, though I do not have that holder as I have no 6x9 camera at the moment
There are different holders for each tab, each format.
On the scanning menu, if you click the preview tab at the bottom you will get a preview scan of all your individual frames, which you can then click through and change individual scan settings for each file if you wish If you unclick the preview tab, the scanner will simply scan the entire area within the holder and you can manually select the portion to scan.
There are grooves on the 6x6 holder to fit your film into, and a hinged door that swings down and locks the negative into place. Maybe you had your holder flipped over on the wrong side?
I am pretty happy with my V500, and it does especially well with medium format.