It's usually the idiosyncratic details.
The way someone sits, the way they move their hands.
The way they smile, or give a knowing look.
Their flaws, attitude, beliefs.
If I can use Glenn Gould as an example:
Look at his passion, his glee. It's his true self.
People are passionate when they are doing or talking about what they love.
It's an attractive trait.
I like his hands. He has those expressive, long-fingered pianist's hands.
I wouldn't say he's remarkably handsome, but he is physically attractive.
He also has blue eyes, which I like on most people, but I also like seagreen and amber eyes.
He has decent cheek bones, he's lean.
So yes, I too like a broad range of features on an individual.
Glenn is not someone I would look at without knowing anything about him, and say that he is attractive.
But the attraction comes from observing his characteristics, if that makes sense.