Does it matter if it did happen? And at that age you wouldn't have really understood it anyway - so stop bothering yourself about it now. You were not hurt and it really doesn't matter if you maybe saw a penis one time, very briefly when you were three or four - it doesn't affect your life now in any way, so focus on more important things
I'm really sorry that you're dealing with this; I know from personal experience just how incredibly difficult it can be when things come up that you're not sure happened, and/or that you wish had never occurred. I too have memories that I'm not certain about (sexual abuse related as well), and I know how intrusive and difficult to manage they can be, and what an intense impact they can have. My best advice to you would be to practice an awful lot of self-compassion, and rather than focus on the content of the memories (which can be really dysregulating), pay attention to and do your best to deal with all the feelings that come up. Regardless of what exactly did or didn't happen, the feelings you're experiencing are very much real and deserve respect and the proper attention. Please take good care of yourself, and know that there are others out there who understand.
@Rockclimbinggirl, I have had moments like that too and it is very unsettling. Something like that happened to me and it was the first time I was able to remember about my abuser. I kept thinking, "was that a dream? Am I making it up?" I'm glad that nothing happened further besides him flashing you (which is still awful). Maybe you can bring it up in therapy and see what they say? That way you can give it the attention it deserves instead of trying to ignore or forget about it. I understand what it's like to have memories resurface. Take care!