Unless you can ask a straight man how many guys he’s slept with, it isn’t gender neutral, no matter how resistant to this fact you are.
E: the fact that neither of you give a shit about the people telling you the term isn’t gender neutral, doesn’t apply to us, and that we don’t feel comfortable with you using it to speak to or about us says it all. No matter how much mental gymnastics you do to convince yourself otherwise you are the ones choosing to be the problem instead of actually listening to others and showing some basic respect. It’s an easy fix, too - all you have to do is give a minimal fuck about others.
You’ve already had some great replies, but in case you want to watch something rather than read, in addition to the Leah Remini documentary series, Louis Theroux also made a documentary (I’m pretty sure there were earlier ones, but the current one is dominating the search results and I can’t find them on wiki or imdb) that is very good.