It's not "just a joke" if it's an actual attitude held by people who have the ability to actually enforce policy based on that prejudice. It's also not "a joke" if you're unwilling to even say that it was a joke, and instead double down on your garbage behavior and cry about people being mean to you for your bigoted ideas because they just didn't get your humor.

Honestly I feel like the number of good members you could lose if they feel like none of the leadership has their backs when somebody makes them feel unwelcome over race/gender/orientation/whatever definitely outweighs the odd prejudiced asshole here and there who throws a fit and leaves because nobody likes their bad sexist jokes.

If you can't be funny without mocking a marginalized community, you should probably just accept that you are not funny and try to pursue other talents.