Might help also to describe what you think feminism is, since it’s one of those terms that is overloaded.
I once had a physical therapist tell me she wasn’t a feminist because she thought women couldn’t be as physically capable as men when serving as soldiers, and seemed to believe feminism requires treating women exactly like men.
I told her I was a feminist because I believe in equal rights for men and women, an idea she did not seem so opposed to.
I haven’t excluded him as being vegan, I’m just not going out of my way to avoid calling him an antisemitic eugenicist. Just because the label sounds mean doesn’t make the label false.
Sexism is expressed in a lot of different ways, and some are more harmful than others. It doesn’t make the label inaccurate simply because it hurts your feelings. You can be a lot of other things in addition to being sexist, too.