I hope things work out between you and her, but always remember that you are a dad as well, and your daughter deserves to have you in her life. I think you should write asking for increased visitation, keeping copies of all communications. I hope you aren't forced to go to court, but if it does end up that way, you can show you have been reasonable in trying to reach an agreement which is in your daughter's best interests.
From the way you describe your ex, I'd be concerned about any other men she brings into your daughter's life, which is even more reason for you to ask for increased contact. It wouldn't do any harm for you to talk with a family attorney just to get a better idea of your parental rights and responsibilities - you dont have to start any legal action just yet.