A Story of Discrimination and Unfairness

There is no Alice and Bob in this talk. This talk is intended for an audience that genuinely cares for humanity and believes in equality while supporting fairness and acts against discrimination. This talk might not be interesting for folks who promote exclusion while discouraging diversity. Many of us have felt excluded in certain situations because of our gender, race, nationality, sexual orientation, disabilities, or physical appearance. This talk aims to communicate how big data driven machine learning is pushing the society towards discrimination, unfairness, and prejudices that harm billions of people every single day. This year, I will not talk about de-anonymizing programmers, re-identifying underground forum members, or anonymous writing. I will be talking about a human right, namely equality, and the issue of unfairness which happens to be embedded in machines that make decisions about our future, what we see and read, or whether we go to prison or not. Machine learning models are widely used for various applications that end up affecting billions of people and Internet users every day. Random forest classifiers guide the U.S. drone program to predict couriers that can lead to terrorists in Pakistan. Employers use algorithms, which might be racist, to aid in employment decisions. Insurance companies determine health care or car insurance rates based on machine learning outcomes. Internet search results are personalized according to machine learning models, which are known to discriminate against women by showing advertisements with lower salaries, while showing higher paying job advertisements for men. On the other hand, natural language processing models are being used for generating text and speech, machine translation, sentiment analysis, and sentence completion, which collectively influence search engine results, page ranks, and the information presented to all Internet users within filter bubbles. Given the enormous and unavoidable effect of machine learning algorithms on individuals and society, we attempt to uncover implicit bias embedded in machine learning models, focusing particularly on word embeddings. We show empirically that natural language necessarily contains human biases, and the paradigm of training machine learning on language corpora means that AI will inevitably imbibe these biases as well. We look at “word embeddings”, a state-of-the-art language representation used in machine learning. Each word is mapped to a point in a 300-dimensional vector space so that semantically similar words map to nearby points. We show that a wide variety of results from psychology on human bias can be replicated using nothing but these word embeddings. We primarily look at the Implicit Association Test (IAT), a widely used and accepted test of implicit bias. The IAT asks subjects to pair concepts together (e.g., white/black-sounding names with pleasant or unpleasant words) and measures reaction times as an indicator of bias. In place of reaction times, we use the semantic closeness between pairs of words. In short, we were able to replicate every single implicit bias result that we tested, with high effect sizes and low p-values. These include innocuous, universal associations (flowers are associated with pleasantness and insects with unpleasantness), racial prejudice (European-American names are associated with pleasantness and African-American names with unpleasantness), and a variety of gender stereotypes (for example, career words are associated with male names and family words with female names). We look at nationalism, mental health stigma, and prejudice towards the elderly. We also look at word embeddings generated from German text to investigate prejudice based on German data. We do not cherry pick any of these IATs, they have been extensively performed by millions of people from various countries and they are also available for German speakers (https://implicit.harvard.edu/implicit/germany/). We go further. We show that information about the real world is recoverable from word embeddings to a striking degree. We can accurately predict the percentage of U.S. workers in an occupation who are women using nothing but the semantic closeness of the occupation word to feminine words! These results simultaneously show that the biases in question are embedded in human language, and that word embeddings are picking up the biases. Our finding of pervasive, human-like bias in AI may be surprising, but we consider it inevitable. We mean “bias” in a morally neutral sense. Some biases are prejudices, which society deems unacceptable. Others are facts about the real world (such as gender gaps in occupations), even if they reflect historical injustices that we wish to mitigate. Yet others are perfectly innocuous. Algorithms don’t have a good way of telling these apart. If AI learns language sufficiently well, it will also learn cultural associations that are offensive, objectionable, or harmful. At a high level, bias is meaning. “Debiasing” these machine models, while intriguing and technically interesting, necessarily harms meaning. Instead, we suggest that mitigating prejudice should be a separate component of an AI system. Rather than altering AI’s representation of language, we should alter how or whether it acts on that knowledge, just as humans are able to learn not to act on our implicit biases. This requires a long-term research program that includes ethicists and domain experts, rather than formulating ethics as just another technical constraint in a learning system. Finally, our results have implications for human prejudice. Given how deeply bias is embedded in language, to what extent does the influence of language explain prejudiced behavior? And could transmission of language explain transmission of prejudices? These explanations are simplistic, but that is precisely our point: in the future, we should treat these as “null hypotheses’’ to be eliminated before we turn to more complex accounts of bias in humans.