I am the same way, and I hate feeling like a victim too, haha. I don't fear people or social interactions, I fear situations where I'm typically the
center of attention and/or being judged and/or where I have no choice but to talk, like public speaking (which I'm also not good at), interviews because I need to sell myself, oral exams, etc.
I also hate speaking on the phone with strangers, which people may think is a fear of social interaction, but I'm actually bothered because I can't hear on the phone very well and it's not a good look to ask "what? sorry? can you repeat?" every five seconds. It's like an oral performance in itself.
But going out with friends? Speaking to strangers at concerts? Going to SF meetups? I don't mind, I'm usually pretty excited by them.