Detecting Gender Stereotypes: Lexicon vs. Supervised Learning Methods

Jenna Cryan*
Shiliang Tang*
Xinyi Zhang
Miriam Metzger
Haitao Zheng
Ben Y. Zhao

* denotes co-first authors with equal contribution.

Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2020)

Best Paper Honorable Mention Award

[Full Text in PDF Format, 1.3MB]

Paper Abstract

Biases in language influence how we interact with each other and society at large. Language affirming gender stereotypes is often observed in various contexts today, from recommendation letters and Wikipedia entries to fiction novels and movie dialogue. Yet to date, there is little agreement on the methodology to quantify gender stereotypes in natural language (specifically the English language). Common methodology (including those adopted by companies tasked with detecting gender bias) rely on a lexicon approach largely based on the original BSRI study from 1974.

In this paper, we reexamine the role of gender stereotype detection in the context of modern tools, by comparatively analyzing efficacy of lexicon-based approaches and end-to-end, ML-based approaches prevalent in state-of-the-art natural language processing systems. Our efforts using a large dataset show that even compared to an updated lexicon-based approach, end-to-end classification approaches are significantly more robust and accurate, even when trained by moderately sized corpora.