how am I supposed to take that other than saying I'm not nor will I ever be good enough?
The penis size obsession is something that has been aggressively promoted in recent years. Honestly, I think it's a form of psychological warfare.
I think many women and men have been indoctrinated with the idea that penis size is the be-all and end-all of romantic relationships, and frankly it's mostly young, sexually inexperienced people who end up believing this. If you talk to women who have had a number of different partners, the one with the biggest dick is not disproportionately the one they say they liked the most.
Some women may have a psychological fascination with dick size, but as LadyWolfshead pointed out, it's not the case that a larger dick has a greater capacity to produce pleasure. Beyond that, in any really loving relationship, the whole person is way more important than any fascination with genitalia.
Similarly, there are some men who have a fascination with breast size, but breasts are pretty much functionally the same regardless of size. Even a man who has a preference for large breasts isn't going to base his relationship choices solely on breast size, unless he's an idiot.
It can be hard to counteract negative messages sometimes, even if you know they don't have substance. Your coworkers are idiots though, and I hope you can learn to ignore their opinions.