Honestly, I also see a massive difference between having thoughts and acting on them. Morally, one is still fine and the other can be wrong.
You're not hurting anyone by having thoughts, and pretty much everyone has bad thoughts at times. Like a wish to kill a person you hate, it might even be accompanied by a lot of angry feelings, but as long as you never plan for it and actually try (most people won't), you're very far from a murderer.
You're being very hard on yourself, and I think you shouldn't feel so guilty. It sounds like you're single so it's not even like you're betraying anyone. So many people find random people attractive and have sexual thoughts about them! It's biological.
That being said, it causes you a lot of distress right now and I'm sorry about that. You might feel better if you could avoid that person (?) and/or perhaps get into dating yourself so you can fill your brain with thoughts about someone available.
