There are all kinds of sizes. The size of the hips, the size of the waist, the size of the head, the size of the nose... All of them matter. The set of sizes creates the whole effect.
Now, I can only speak for me. As far as girls are concerned, I've liked big-breasted ones and small-breasted ones. I like the girl, and (bonus) the girl has breasts, not the other way around.
But, considering I have a lot of issues with my 4 inch dick (like considering myself crippled and a prime target for being humiliated if I let my pants drop, or regretting that some positions will be plainly impossible for me), I'd say in my (broken) mind, size matters.