My thoughts, You are wrong.
My wife has a girl friend I don't particularly like, but I would not dream of 'speaking out' about them having lunch together now and then.
(I'd rather that than she came here)
Her friends are Her friends, not mine.
I have an Ex I still like very much, as in Best Friend, Our relationship ended amicably 30 years ago. We contact each other several times a year, and even vacation together. But in that case, my wife also thinks my Ex and her husband are nice people.
Last year my wife went away, with her daughter, to be with her dying Ex, I've never met the guy, I understand (from my wife) during their break-up he was a real jerk, but if She want's to see him, I would not dream of stopping her, or even questioning her discussion.
I fear you my have a frail ego.
What do you fear?