I know how you feel sorta,
Me and my ex have been 3 years on and off, she cheated on me twice, the third time be brokeup was because I simply couldn’t trust her like I use to and i would have horrible anxiety when I was with her. We haven’t talked since we broke up nearly 3 months ago. Even though we no longer talk I also have an extremely hard time dealing with how close we were, she was one of the only people I felt comfortable around and who knew truly everything about me as i knew...almost everything about her
3 months latter and I’m still lonely as hell I regret almost every day not trying to Just be friends with her, this quote sorta explains how I was feeling “You wanna know what the truth is? I still love you and I probably will love you for a very long time. But I can't just be your buddy, because as much as i enjoy the concept of being "just friends" in reality it's a bizarre form of torture and i'm just not willing to participate in it. so right now what i wanna do is just move on and get over you and the only way for me to do that is to not be around you anymore.”
Anyways....i guess what im trying to say is...before you completely shut him out of your life maybe just slowly let him go, see how it feels, just go with what feels right. Someone else will come along onday, it’s a slow process and they will never be able to “replace” him and what he was for you at that point in time, however he will be all you will need. its not fair to you to have to be around this person and have it cause you pain