Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings

Man is to Computer Programmer as W oman is to Homemak er? Debiasing W ord Em b eddings T olga Bolukbasi 1 , Kai-W ei Chang 2 , James Zou 2 , V enk atesh Saligrama 1,2 , A dam Kalai 2 1 Boston Univ ersity , 8 Saint Mary’s Street, Boston, MA 2 Microsoft Researc h New England, 1 Memorial Drive, Cam bridge, MA tolgab@bu.edu, kw@kw chang.net, jamesyzou@gmail.com, srv@bu.edu, adam.k alai@microsoft.com Abstract The blind application of machine learning runs the risk of amplifying biases presen t in data. Such a danger is facing us with wor d emb e dding , a p opular framework to represent text data as vectors which has b een used in many machine learning and natural language pro cessing tasks. W e show that even w ord em b eddings trained on Go ogle News articles exhibit female/male gender stereotypes to a disturbing exten t. This raises concerns b ecause their widespread use, as we describ e, often tends to amplify these biases. Geometrically , gender bias is ﬁrst shown to b e captured by a direction in the word em b edding. Second, gender neutral words are shown to b e linearly separable from gender deﬁnition words in the word em b edding. Using these prop erties, we pro vide a metho dology for mo difying an embedding to remov e gender stereotypes, such as the asso ciation b et ween b etw een the words r e c eptionist and female , while main taining desired asso ciations suc h as b et ween the words queen and female . W e deﬁne metrics to quan tify b oth direct and indirect gender biases in embeddings, and develop algorithms to “debias” the em b edding. Using crowd-w ork er ev aluation as well as standard b enc hmarks, we empirically demonstrate that our algorithms signiﬁcan tly reduce gender bias in embeddings while preserving the its useful prop erties suc h as the ability to cluster related concepts and to solve analogy tasks. The resulting embeddings can b e used in applications without amplifying gender bias. 1 In tro duction There hav e b een hundreds or thousands of pap ers written ab out word embeddings and their applications, from W eb searc h [ 27 ] to parsing Curriculum Vitae [ 16 ]. How ever, none of these pap ers hav e recognized how blatan tly sexist the em b eddings are and hence risk introducing biases of v arious types into real-world systems. A word embedding that represent each word (or common phrase) w as a d -dimensional wor d ve ctor ~ w ∈ R d . W ord embeddings, trained only on w ord co-o ccurrence in text corp ora, serve as a dictionary of sorts for computer programs that would lik e to use word meaning. First, words with similar semantic meanings tend to ha ve vectors that are close together. Second, the vector diﬀerences b etw een words in embeddings ha ve b een shown to represent relationships b etw een words [ 32 , 26 ]. F or example given an analogy puzzle, “man is to king as woman is to x ” (denoted as man : king :: woman : x ), simple arithmetic of the embedding v ectors ﬁnds that x = que en is the b est answer b ecause: − − → man − − − − − → w oman ≈ − − → king − − − − → queen Similarly , x = Jap an is returned for Paris : F r anc e :: T okyo : x . It is surprising that a simple vector arithmetic can simultaneously capture a v ariet y of relationships. It has also excited practitioners b ecause such a to ol could b e useful across applications inv olving natural language. Indeed, they are b eing studied and used in a v ariety of downstream applications (e.g., do cumen t ranking [ 27 ], sentimen t analysis [ 18 ], and question retriev al [22]). Ho wev er, the embeddings also pinp oin t sexism implicit in text. F or instance, it is also the case that: − − → man − − − − − → w oman ≈ − − − − − − − − − − − − − − − − → computer programmer − − − − − − − − − → homemak er . 1 Extreme she o ccupations 1. homemaker 2. nurse 3. receptionist 4. librarian 5. so cialite 6. hairdresser 7. nanny 8. b o okk eep er 9. stylist 10. housekeeper 11. interior designer 12. guidance counselor Extreme he o ccupations 1. maestro 2. skipp er 3. protege 4. philosopher 5. captain 6. architect 7. ﬁnancier 8. warrior 9. broadcaster 10. magician 11. ﬁgher pilot 12. b oss Figure 1: The most extreme o ccupations as pro jected on to the she − he gender direction on g2vNEWS. Occupations suc h as businesswoman , where gender is suggested by the orthography , were excluded. Gender stereotype she - he analogies. sewing-carp en try register-n urse-physician housewife-shopk eep er n urse-surgeon in terior designer-architect softball-baseball blond-burly feminism-conserv atism cosmetics-pharmaceuticals giggle-c huc kle v ocalist-guitarist p etite-lanky sassy-snapp y div a-superstar c harming-aﬀable v olleyball-fo otball cupcakes-pizzas hairdresser-barb er Gender appropriate she - he analogies. queen-king sister-brother mother-father w aitress-waiter o v arian cancer-prostate cancer con ven t-monastery Figure 2: Analogy examples . Examples of automatically generated analogies for the pair she-he using the pro cedure describ ed in text. F or example, the ﬁrst analogy is interpreted as she : sewing :: he : c arp entry in the original w2vNEWS embedding. Each automatically generated analogy is ev aluated by 10 crowd-w ork ers are to whether or not it reﬂects gender stereotype. T op: illustrative gender stereotypic analogies automatically generated from w2vNEWS, as rated by at least 5 of the 10 crowd-w ork ers. Bottom: illustrative generated gender-appropriate analogies. softb al l extreme gender p ortion after debiasing 1. pitcher -1% 1. pitcher 2. b o okk eep er 20% 2. inﬁelder 3. receptionist 67% 3. ma jor leaguer 4. registered nurse 29% 4. b o okk eep er 5. waitress 35% 5. inv estigator fo otb al l extreme gender p ortion after debiasing 1. fo otballer 2% 1. fo otballer 2. businessman 31% 2. cleric 3. pundit 10% 3. vice chancellor 4. maestro 42% 4. lecturer 5. cleric 2% 5. midﬁelder Figure 3: Example of indirect bias . The ﬁve most extreme o ccupations on the softb al l-fo otb al l axis, whic h indirectly captures gender bias. F or each o ccupation, the degree to which the asso ciation represen ts a gender bias is sho wn, as describ ed in Section 5.3. 2 In other w ords, the same system that solved the ab o ve reasonable analogies will oﬀensively answer “man is to computer programmer as woman is to x ” with x = homemaker . Similarly , it outputs that a father is to a do ctor as a mother is to a nurse . The primary embedding studied in this pap er is the p opular publicly-av ailable w ord2vec [ 24 , 25 ] embedding trained on a corpus of Go ogle News texts consisting of 3 million English w ords and terms into 300 dimensions, whic h w e refer to here as the w2vNEWS. One might hav e hop ed that the Go ogle News embedding would exhibit little gender bias b ecause many of its authors are professional journalists. W e also analyze other publicly av ailable embeddings trained via other algorithms and ﬁnd similar biases. In this pap er, we will quantitativ ely demonstrate that word-em b eddings contain biases in their geometry that reﬂect gender stereotypes present in broader so ciet y . Due to their wide-spread usage as basic features, w ord embeddings not only reﬂect such stereotypes but can also amplify them. This p oses a signiﬁcant risk and c hallenge for machine learning and its applications. T o illustrate bias ampliﬁcation, consider bias presen t in the task of retrieving relev an t w eb pages for a given query . In web searc h, one recent pro ject has shown that, when carefully combined with existing approac hes, word vectors ha ve the p otential to improv e web page relev ance results [ 27 ]. As an example, supp ose the search query is cmu c omputer scienc e phd student for a computer science Ph.D. student at Carnegie Mellon Univ ersity . Now, the directory 1 oﬀers 127 nearly iden tical web pages for students — these pages diﬀer only in the names of the students. A word em b edding’s semantic kno wledge can improv e relev ance b y identifying, for examples, that the terms gr aduate r ese ar ch assistant and phd student are related. How ever, w ord embeddings also rank terms related to computer science closer to male names than female names (e.g., the embeddings give John : c omputer pr o gr ammer :: Mary : homemaker ). The consequence is that, b et ween t wo pages that diﬀer only in the names Mary and John , the word embedding would inﬂuence the search engine to rank John’s web page higher than Mary . In this h yp othetical example, the usage of word embedding makes it ev en harder for women to b e recognized as computer scientists and would contribute to widening the existing gender gap in computer science. While we fo cus on gender bias, sp eciﬁcally F emale-Male (F-M) bias, the approac h may b e applied to other types of bias. Unco vering gender stereotypes from text may seem like a trivial matter of counting pairs of words that o ccur together. How ev er, such counts are often misleading [ 14 ]. F or instance, the term male nurse is sev eral times more frequent than female nurse (similarly female quarterb ack is many times more frequent than male quarterb ack ). Hence, extracting associations from text, F-M or otherwise, is not simple, and “ﬁrst-order” approac hes would predict that the word nurse is more male than quarterb ack . More generally , Gordon and V an Durme show how r ep orting bias [ 14 ], including the fact that common assumptions are often left unsaid, p oses a challenge to extracting knowledge from raw text. Nonetheless, − − − → n urse is closer to − − − − → female than − − → male , suggesting that word embeddings may b e capable of circum ven ting rep orting bias in some cases. This happ ens b ecause w ord embeddings are traine d using second-order metho ds which require large amounts of data to extract asso ciations and relationships ab out words. The analogies generated from these embeddings sp ell out the bias implicit in the data on which they were trained. Hence, w ord embeddings may serve as a means to extract implicit gender asso ciations from a large text corpus similar to how Implicit Asso ciation T ests [ 15 ] detect automatic gender asso ciations p ossessed by p eople, which often do not align with self rep orts. T o quantify bias, we compare a word embedding to the embeddings of a pair of gender-sp eciﬁc words. F or instance, the fact that − − − → n urse is close to − − − − → w oman is not in itself necessarily biased (it is also somewhat close to − − → man – all are humans), but the fact that these distances are unequal suggests bias. T o make this rigorous, consider the distinction b et ween gender sp e ciﬁc words that are asso ciated with a gender by deﬁnition, and the remaining gender neutr al words. Standard examples of gender sp eciﬁc words include br other , sister , businessman and businesswoman . The fact that − − − − − → brother is closer to − − → man than to − − − − → w oman is exp ected since they share the deﬁnitive feature of relating to males. W e will use the gender sp eciﬁc w ords to learn a gender subspace in the embedding, and our debiasing algorithm remov es the bias only from the gender neutral words while resp ecting the deﬁnitions of these gender sp eciﬁc words. W e refer to this type of bias, where there is an asso ciation b et w een a gender neutral word and a clear 1 Graduate Research Assistants listed at http://cs.cmu.edu/directory/csd . 3 gender pair as dir e ct bias . W e also consider a notion of indir e ct bias , 2 whic h manifests as asso ciations b etw een gender neutral words that are clearly arising from gender. F or instance, the fact that the word r e c eptionist is m uch closer to softb al l than fo otb al l may arise from female asso ciations with b oth r e c eptionist and softb al l . Note that many pairs of male-biased (or female-biased) w ords hav e legitimate as sociations having nothing to do with gender. F or instance, while the words mathematician and ge ometry b oth hav e a strong male bias, their similarit y is justiﬁed b y factors other than gender. More often than not, asso ciations are combinations of gender and other factors that can b e diﬃcult to disen tangle. Nonetheless, we can use the geometry of the w ord embedding to determine the degree to which those asso ciations are based on gender. Aligning biases with stereot yp es. Stereotypes are biases that are widely held among a group of p eople. W e sho w that the biases in the word embedding are in fact closely aligned with so cial conception of gender stereot yp e, as ev aluated b y U.S.-based crowd work ers on Amazon’s Mechanical T urk. 3 The crowd agreed that the biases reﬂected b oth in the lo cation of v ectors (e.g. − − − − → do ctor closer to − − → man than to − − − − → w oman ) as well as in analogies (e.g., he : c owar d :: she : whor e ) exhibit common gender stereotypes. Debiasing. Our goal is to reduce gender biases in the w ord embedding while preserving the useful prop erties of the embedding. Surprisingly , not only do es the embedding capture bias, but it also contains suﬃcient information to reduce this bias, as illustrated in 7. W e will leverage the fact that there exists a lo w dimensional subspace in the em b edding that empirically captures muc h of the gender bias. The goals of debiasing are: 1. Reduce bias: (a) Ensure that gender neutral words such as nurse are equidistant b et w een gender pairs such as he and she . (b) Reduce gender asso ciations that p erv ade the embedding even among gender neutral words. 2. Main tain embedding utility: (a) Main tain meaningful non-gender-related asso ciations b et ween gender neutral w ords, including asso ciations within stereotypical categories of words such as fashion-related words or w ords asso ciated with fo otball. (b) Correctly main tain deﬁnitional gender asso ciations suc h as b etw een man and father . P ap er outline. After discussing related literature, we give preliminaries necessary for understanding the pap er in Section 3. Next we prop ose metho ds to identify the gender bias of an embedding and show that w2vNEWS exhibits bias which is aligned with common gender stereotypes (Section 4). In Section 5, we deﬁne sev eral simple geometric prop erties asso ciated with bias, and in particular discuss ho w to iden tify the gender subspace. Using these geometric prop erties, we introduce debiasing algorithms (Section 6) and demonstrate their p erformance (Section 8). Finally we conclude with additional discussions of related literature, other t yp es of biases in the em b edding and future works. 2 Related w ork Related w ork can b e divided in to relev ant literature on bias in language and bias in algorithms. 2 The terminology indirect bias follows P edreshi et al. [ 29 ] who distinguish dir e ct versus indir e ct discrimination in rules of fair classiﬁers . Direct discrimination inv olves directly using sensitive features such as gender or race, whereas indirect discrimination inv olves using correlates that are not inherently based on sensitiv e features but that, inten tionally or uninten tionally , lead to disproportionate treatment nonetheless. 3 http://mturk.com 4 2.1 Gender bias and stereotype in English It is imp ortant to quantify and understand bias in languages as such biases can reinforce the psychological status of diﬀerent groups [ 33 ]. Gender bias in language has b een studied o ver a num b er of decades in a v ariet y of contexts (see, e.g., [ 17 ]) and w e only highlight some of the ﬁndings here. Biases diﬀer across p eople though commonalities can b e detected. Implicit Asso ciation T ests [ 15 ] hav e uncov ered gender-word biases that p eople do not self-rep ort and may not even b e aw are of. Common biases link female terms with lib eral arts and family and male terms with science and careers [ 28 ]. Bias is seen in w ord morphology , i.e., the fact that words such as actor are, by default, asso ciated with the dominant class [ 19 ], and female versions of these w ords, e.g., actr ess , are marked. There is also an imbalance in the num b er of words with F-M with v arious asso ciations. F or instance, while there are more words referring to males, there are many more words that sexualize females than males [35]. Glic k and Fiske [ 13 ] introduce the notion of b enevolent sexism in which women are p erceiv ed with p ositiv e traits such as helpful or intimacy-seeking. Despite its seemingly p ositiv e nature, b enev olent sexism can b e harmful, insulting, and discriminatory . In terms of words, female gender asso ciations with any word, ev en a sub jectiv ely p ositiv e word such as attr active , can cause discrimination against women if it reduces their asso ciation with other words, such as pr ofessional . Stereot yp es, as mentioned, are biases that are widely held within a group. While gender bias of any kind is concerning, stereotypes are often easier to study due to their consistent nature. Stereotypes hav e commonalities across cultures, though there is some v ariation b et w een cultures [ 5 ]. Complimentary ster e otyp es are common b et ween females and males, in whic h each gender is asso ciated with strengths that are p erceiv ed to oﬀset its own weaknesses and complimen t the strengths of the other gender [ 20 ]. These and comp ensatory stereot yp es are used b y p eople to justify the status quo. Consisten t biases hav e b een studied within online contexts and sp eciﬁcally related to the contexts we study such as online news (e.g., [ 31 ]), W eb searc h (e.g., [ 21 ]), and Wikip edia (e.g., [ 39 ]). In Wikip edia, W ager et al. [ 39 ] found that, as suggested by prior work on gender bias in language [ 2 ], articles ab out women more often emphasize their gender, their husbands and their husbands’ jobs, and other topics discussed consistently less often than in articles ab out men. Regarding individual words, they ﬁnd that certain words are predictive of gender, e.g., husb and app ears signiﬁcantly more often in articles ab out women while b aseb al l o ccurs more often in articles ab out men. 2.2 Bias within algorithms A num b er of online systems hav e b een sho wn to exhibit v arious biases, such as racial discrimination and gender bias in the ads presented to users [ 36 , 6 ]. A recent study found that algorithms used to predict rep eat oﬀenders exhibit indirect racial biases [ 1 ]. Diﬀerent demographic and geographic groups also use diﬀerent dialects and word-c hoices in so cial media [ 8 ]. An implication of this eﬀect is that language used by minority group migh t not b e able to b e pro cessed by natural language to ols that are trained on “standard” data-sets. Biases in the curation of mac hine learning data-sets hav e explored in [37, 4]. Indep enden t from our work, Schmidt [ 34 ] identiﬁed the bias present in word embeddings and prop osed debiasing by entirely removing multiple gender dimensions, one for each gender pair. His goal and approac h, similar but simpler than ours, was to en tirely remov e gender from the embedding. There is also an intense researc h agenda focused on impro ving the quality of w ord embeddings from diﬀeren t angles (e.g., [ 23 , 30 , 40 , 9 ]), and the diﬃculty of ev aluating embedding quality (as compared to sup ervised learning) parallels the diﬃculty of deﬁning bias in an em b edding. Within machine learning, a b ody of notable work has fo cused on “fair” binary classiﬁcation in particular. A deﬁnition of fairness based on legal traditions is presen ted by Baro cas and Selbst [ 3 ]. Approaches to mo dify classiﬁcation algorithms to deﬁne and ac hieve v arious notions of fairness ha ve b een describ ed in a num ber of w orks, see, e.g., [3, 7, 10] and a recent survey [41]. F eldman et al. [ 10 ] distinguish classiﬁcation algorithms that achiev e fairness by m odifying the underlying data from those that achiev e fairness by mo difying the classiﬁcation algorithm. Our approac h is more similar to the former. How ever, it is unclear how to apply any of these previous approaches without a clear 5 classiﬁcation task in hand, and the problem is exacerbated by indirect bias. This prior w ork on algorithmic fairness is largely for sup ervised learning. F air classiﬁcation is deﬁned based on the fact that algorithms were classifying a set of individuals using a set of features with a distinguished sensitiv e feature. In word embeddings, there are no clear individuals and no a priori deﬁned classiﬁcation problem. How ever, similar issues arise, such as direct and indirect bias [29]. 3 Preliminaries W e ﬁrst very brieﬂy deﬁne an embedding and some terminology . An embedding consists of a unit vector ~ w ∈ R d , with k ~ w k = 1 , for each word (or term) w ∈ W . W e assume there is a set of gender neutral words N ⊂ W , such as ﬂight att endant or sho es , which, b y deﬁnition, are not sp eciﬁc to any gender. W e denote the size of a set S b y | S | . W e also assume we are given a set of F-M gender pairs P ⊂ W × W , such as she-he or mother-father whose deﬁnitions diﬀer mainly in gender. Section 7 discusses how N and P can b e found within the em b edding itself, but un til then we take them as given. As is common, similarity b et w een words w 1 and w 2 is measured b y their inner pro duct, ~ w 1 · ~ w 2 . Finally , w e will abuse terminology and refer to the embedding of a word and the word interc hangeably . F or example, the statement c at is more similar to do g than to c ow means − → cat · − → dog ≥ − → cat · − − → co w . F or arbitrary vectors u and v , deﬁne: cos( u, v ) = u · v k u kk v k . This normalized similarity b et ween vectors u and v is written as cos b ecause it is the cosine of the angle b et w een the tw o vectors. Since words are normalized cos( ~ w 1 , ~ w 2 ) = ~ w 1 · ~ w 2 . Em b edding. Unless otherwise stated, the em b edding we refer to in this pap er is the aforementioned w2vNEWS em b edding, a d = 300 -dimensional word2v ec [ 24 , 25 ] em b edding, whic h has pro v en to be immensely useful since it is high quality , publicly av ailable, and easy to incorp orate into any application. In particular, w e downloaded the pre-trained embedding on the Go ogle News corpus, 4 and normalized each w ord to unit length as is common. Starting with the 50,000 most frequen t words, w e selected only low er-case w ords and phrases consisting of fewer than 20 low er-case characters (words with upp er-case letters, digits, or punctuation were discarded). After this ﬁltering, 26,377 words remained. While we fo cus on w2vNEWS, we sho w later that gender stereotypes are also present in other embedding data-sets. Cro wd exp erimen ts. All human exp erimen ts were p erformed on the Amazon Mechanical T urk crowdsourc- ing platform. W e selected for U.S.-based work ers to maintain homogeneity and repro ducibility to the extent p ossible with crowdsourcing. T wo types of exp erimen ts were p erformed: ones where w e solicited w ords from the crowd (to see if the embedding biases contain those of the crowd) and ones where we solicited ratings on words or analogies generated from our embedding (to see if the crowd’s biases contain those from the em b edding). These tw o types of exp erimen ts are analogous to exp erimen ts p erformed in rating results in information retriev al to ev aluate precision and recall. When we sp eak of the ma jorit y of 10 crowd judgments, w e mean those annotations made by 5 or more indep enden t work ers. Since gender asso ciations v ary by culture and p erson, w e ask for ratings of stereotypes rather than bias. In addition to possessing greater consistency than biases, p eople ma y feel more comfortable rating the stereot yp es of their culture than discussing their own gender biases. The App endix contains the questionnaires that were giv en to the crowd-w orkers to p erform these tasks. 4 Gender stereot yp es in w ord em b eddings Our ﬁrst task is to understand the biases present in the w ord-embedding (i.e. which words are closer to she than to he , etc.) and the extent to which these geometric biases agree with human notion of gender stereot yp es. W e use t wo simple metho ds to approach this problem: 1) ev aluate whether the em b edding has 4 https://code.google.com/archive/p/word2vec/ 6 Figure 4: Comparing the bias of tw o diﬀeren t em b eddings–the w2vNEWS and the GloV e w eb-crawl em b edding. In each embedding, the o ccupation words are pro jected onto the she - he direction. Eac h dot corresp onds to one o ccupation word; the gender bias of o ccupations is highly consistent across em b eddings (Sp earman ρ = 0 . 81 ). stereot yp es on o ccupation words and 2) ev aluate whether the embedding pro duces analogies that are judged to reﬂect stereotypes by humans. The exploratory analysis of this section will motiv ate the more rigorous metrics used in the next t wo sections. Occupational stereot yp es. Figure 1 lists the occupations that are closest to she and to he in the w2vNEWS embeddings. W e asked the crowdw orkers to ev aluate whether an o ccupation is considered fem ale- stereot ypic, male-stereotypic, or neutral. Each o ccupation w ord was ev aluated by ten cro wd-work ers as to whether or not it reﬂects gender stereotype. Hence, for each word we had a integer rating, on a scale of 0-10, of stereotypicalit y . The pro jection of the o ccupation words onto the she - he axis is strongly correlated with the stereotypicalit y estimates of these words (Sp earman ρ = 0 . 51 ), suggesting that the geometric biases of em b edding vectors is aligned with crowd judgment of gender stereotypes. W e used o ccupation words here b ecause they are easily interpretable by humans and often capture common gender stereotypes. Other word sets could b e used for this task. Also note that we could ha ve used other words, e.g. woman and man , as the gender-pair in the task. W e chose she and he b ecause they are frequent and do not hav e fewer alternativ e w ord senses (e.g., man can also refer to mankind ). W e pro jected each of the o ccupations onto the she-he direction in the w2vNEWS em b edding as well as a diﬀeren t embedding generated by the GloV e algorithm on a web-cra wl corpus [ 30 ]. The results are highly consisten t (Figure 4), suggesting that gender stereot yp es is prev alen t across diﬀerent embeddings and is not an artifact of the particular training corpus or metho dology of word2v ec. Analogies exhibiting stereotypes. Analogies are a useful wa y to b oth ev aluate the quality of a word em b edding and also its stereotypes. W e ﬁrst brieﬂy describ e how the em b edding generate analogies and then discuss how we use analogies to quantify gender stereotype in the embedding. A more detailed discussion of our algorithm and prior analogy solv ers is given in App endix A. In the standard analogy tasks, we are given three words, for example he, she, king , and lo ok for the 4th w ord to solve he to king is as she to x . Here we mo dify the analogy task so that given tw o words, e.g. he, she , w e wan t to generate a pair of words, x and y , suc h that he to x as she to y is a go od analogy . This mo diﬁcation allows us to systematically generate pairs of words that the embedding b elieves it analogous to he, she (or any other pair of seed words). The input into our analogy generator is a seed pair of words ( a, b ) determining a se e d dir e ction ~ a − ~ b corresp onding to the normalized diﬀerence b et ween the tw o seed words. In the task b elo w, we use ( a, b ) = 7 ( she , he ) . W e then score all pairs of words x, y b y the following metric: S ( a,b ) ( x, y ) = ( cos  ~ a − ~ b, ~ x − ~ y  if k ~ x − ~ y k ≤ δ 0 otherwise (1) where δ is a threshold for similarity . The intuition of the scoring m etric is that we w ant a goo d analogy pair to b e close to parallel to the seed direction while the tw o words are not to o far apart in order to b e seman tically coherent. The parameter δ sets the threshold for semantic similarity . In all the exp erimen ts, w e take δ = 1 as we ﬁnd that this choice often works well in practice. Since all em b eddings are normalized, this threshold corresp onds to an angle ≤ π / 3 , indicating that the tw o words are closer to each other than they are to the origin. In practice, it means that the tw o words forming the analogy are signiﬁcantly closer together than tw o random embedding vectors. Given the embedding and seed words, we output the top analogous pairs with the largest p ositiv e S ( a,b ) scores. T o reduce redundancy , we do not output multiple analogies sharing the same w ord x . Since analogies, stereotypes, and biases are heavily inﬂuenced by culture, we employ ed U.S. based crowd- w orkers to ev aluate the analogies output by the analogy generating algorithm describ ed ab ov e. F or each analogy , we asked the work ers t wo yes/no questions: (a) whether the pairing makes sense as an analogy , and (b) whether it reﬂects a gender stereotype. Every analogy is judged by 10 work ers, and we used the num b er of work ers that rated this pair as stereotyped to quantify the degree of bias of this analogy . Overall, 72 out of 150 analogies were rated as gender-appropriate by ﬁve or more crowd-w orkers, and 29 analogies were rated as exhibiting gender stereotype by ﬁve or more crowd-w ork ers (Figure 8). Examples of analogies generated from w2vNEWS that were rated as stereotypical are shown at the top of Figure 2, and examples of analogies that make sense and are rated as gender-appropriate are shown at the b ottom of Figure 2. The full list of analogies and cro wd ratings are in App endix G. Indirect gender bias. The direct bias analyzed ab o ve manifests in the relative similarities b et ween gender- sp eciﬁc words and gender neutral w ords. Gender bias could also aﬀect the relative geometry b et w een gender neutral words themselves. T o test this indir e ct gender bias, we take pairs of words that are gender-neutral, for example softb al l and fo otb al l . W e pro ject all the o ccupation words onto the − − − − → softball − − − − − − → fo otball direction and lo ok ed at the extremes words, which are listed in Figure 3. F or instance, the fact that the words b o okke ep er and r e c eptionist are muc h closer to softb al l than fo otb al l ma y result indirectly from female asso ciations with b o okke ep er , r e c eptionist and softb al l . It’s imp ortan t to p oin t out that that many pairs of male-biased (or female-biased) w ords hav e legitimate asso ciations having nothing to do with gender. F or example, while b oth fo otb al ler and fo otb al l hav e strong male biases, their similarit y is justiﬁed by factors other than gender. In Section 5, w e deﬁne a metric to more rigorously quantify these indirect eﬀects of gender bias. 5 Geometry of Gender and Bias In this section, we study the bias present in the embedding geometrically , identifying the gender direction and quantifying the bias indep enden t of the exten t to which it is aligned with the crowd bias. W e develop metrics of direct and indirect bias that more rigorously quantify the observ ations of the previous section. 5.1 Iden tifying the gender subspace Language use is “messy” and therefore individual word pairs do not alwa ys b eha ve as exp ected. F or instance, the w ord man has several diﬀerent usages: it may b e used as an exclamation as in oh man! or to refer to p eople of either gender or as a verb, e.g., man the station . T o more robustly estimate bias, we shall aggregate across multiple paired comparisons. By combining several directions, such as − → she − − → he and − − − − → w oman − − − → man , w e iden tify a gender direction g ∈ R d that largely captures gender in the embedding. This direction helps us to quan tify direct and indirect biases in words and asso ciations. 8 def. stereo. def. stereo. − → she − − → he 92% 89% − − − − − − → daugh ter − − → son 93% 91% − → her − − → his 84% 87% − − − − → mother − − − − → father 91% 85% − − − − → w oman − − − → man 90% 83% − → gal − − − → guy 85% 85% − − − → Mary − − − − → John 75% 87% − → girl − − − → b o y 90% 86% − − − − → herself − − − − − → himself 93% 89% − − − − → female − − − → male 84% 75% Figure 5: T en p ossible word pairs to deﬁne gender, ordered b y word frequency , along with agreement with t wo sets of 100 words solicited from the crowd, one with deﬁnitional and and one with stereotypical gender asso ciations. F or each set of words, comprised of the most frequent 50 female and 50 male cro wd suggestions, the accuracy is shown for the corresp onding gender classiﬁer based on which word is closer to a target word, e.g., the she-he classiﬁer predicts a word is female if it is closer to she than he . With roughly 80-90% accuracy , the gender pairs predict the gender of b oth stereotypes and deﬁnitionally gendered words solicited from the cro wd. In English as in many languages, there are numerous gender pair terms, and for each we can consider the diﬀerence b et ween their em b eddings. Before lo oking at the data, one might imagine that they all had roughly the same v ector diﬀerences, as in the following caricature: − − − − − − − − − → grandmother = − − → wise + − → gal − − − − − − − − → grandfather = − − → wise + − − → guy − − − − − − − − − → grandmother − − − − − − − − − → grandfather = − → gal − − − → guy = g Ho wev er, gender pair diﬀerences are not parallel in practice, for m ultiple reasons. First, there are diﬀerent biases asso ciated with with diﬀerent gender pairs. Second is p olysemy , as mentioned, whic h in this case o ccurs due to the other use of gr andfather as in to gr andfather a r e gulation . Finally , randomness in the word counts in an y ﬁnite sample will also lead to diﬀerences. Figure 5 illustrates ten p ossible gender pairs,  ( x i , y i )  10 i =1 . W e exp erimen tally veriﬁed that the pairs of vectors corresp onding to these words do agree with the cro wd concept of gender. On Amazon Mechanical T urk, we asked crowdw orkers to generate tw o lists of w ords: one list corresp onding to words that they think are gendered by deﬁnition ( waitr ess , menswe ar ) and a separate list corresp onding to words that they b eliev e captures gender stereotypes (e.g., sewing , fo otb al l ). F rom this we generated the most frequently suggested 50 male and 50 female words for each list to b e used for a classiﬁcation task. F or each candidate pair, for example − → she , − → he , we say that it accurately classiﬁes a cro wd suggested female deﬁnition (or stereotype) word if that word vector is closer to − → she than to − → he . T able 5 rep orts the classiﬁcation accuracy for deﬁnition and stereotype words for each gender pair. The accuracies are high, indicating that these pairs capture the intuitiv e notion of gender. T o identify the gender subspace, we to ok the ten gender pair diﬀerence vectors and computed its principal comp onen ts (PCs). As Figure 6 shows, there is a single direction that explains the ma jority of v ariance in these vectors. The ﬁrst eigenv alue is signiﬁcantly larger than the rest. Note that, from the randomness in a ﬁnite sample of ten noisy vectors, one exp ects a decrease in eigenv alues. Ho wev er, as also illustrated in 6, the decrease one observes due to random sampling is muc h more gradual and uniform. Therefore we h yp othesize that the top PC, denoted by the unit v ector g , captures the gender subspace. In general, the gender subspace could b e higher dimensional and all of our analysis and algorithms (describ ed b elow) work with general subspaces. 5.2 Direct bias T o measure direct bias, we ﬁrst identify words that should b e gender-neutral for the application in question. Ho w to generate this set of gender-neutral words is describ ed in Section 7. Giv en the gender neutral words, denoted by N , and the gender direction learned from ab o ve, g , we deﬁne the direct gender bias of an 9 Figure 6: Left: the p ercen tage of v ariance explained in the PCA of these vector diﬀerences (each diﬀerence normalized to b e a unit v ector). The top comp onen t explains signiﬁcantly more v ariance than any other. Righ t: for comparison, the corresp onding p ercen tages for random unit vectors (ﬁgure created b y av eraging o ver 1,000 draws of ten random unit vectors in 300 dimensions). em b edding to b e DirectBias c = 1 | N | X w ∈ N | cos( ~ w , g ) | c where c is a parameter that determines how strict do w e wan t to in measuring bias. If c is 0, then | cos( ~ w − g ) | c = 0 only if ~ w has no ov erlap with g and otherwise it is 1. Such strict measurement of bias migh t b e desirable in settings such as the college admissions example from the In tro duction, where it would b e unacceptable for the embedding to in tro duce a slight preference for one candidate ov er another by gender. A more gradual bias would b e setting c = 1 . The presentation we hav e chosen fav ors simplicit y – it would b e natural to extend our deﬁnitions to weigh t words by frequency . F or example, in w2vNEWS, if w e take N to b e the set of 327 o ccupations, then DirectBias 1 = 0 . 08 , which conﬁrms that many o ccupation w ords hav e substan tial comp onen t along the gender direction. 5.3 Indirect bias Unfortunately , the ab o ve deﬁnitions still do not capture indirect bias. T o see this, imagine completely remo ving from the embedding b oth w ords in gender pairs (as well as words such as b e ar d or uterus that are arguably gender-sp eciﬁc but whic h cannot b e paired). There would still b e indirect gender asso ciation in that a word that should b e gender neutral, such as r e c eptionist , is closer to softb al l than fo otb al l (see Figure 3). As discussed in the Introduction, it can b e subtle to obtain the ground truth of the extent to which such similarities is due to gender. The gender subspace g that we ha ve iden tiﬁed allo ws us to quan tify the contribution of g to the similarities b et w een any pair of words. W e can decomp ose a given w ord v ector w ∈ R d as w = w g + w ⊥ , where w g = ( w · g ) g is the con tribution from gender and w ⊥ = w − w g . Note that all the word v ectors are normalized to hav e unit length. W e deﬁne the gender comp onen t to the similarity b et ween tw o word vectors w and v as β ( w , v ) =  w · v − w ⊥ · v ⊥ k w ⊥ k 2 k v ⊥ k 2   w · v. The in tuition b ehind this metric is as follow: w ⊥ · v ⊥ k w ⊥ k 2 k v ⊥ k 2 is the inner pro duct b et ween the tw o vectors if w e pro ject out the gender subspace and renormalize the vectors to b e of unit length. The metric quan tiﬁes ho w muc h this inner pro duct changes (as a fraction of the original inner pro duct v alue) due to this op eration of removing the gender subspace. Because of noise in the data, every vector has some non-zero comp onen t w ⊥ and β is well-deﬁned. Note that β ( w , w ) = 0 , which is reasonable since the similarity of a word to itself should not dep end on gender contribution. If w g = 0 = v g , then β ( w , v ) = 0 ; and if w ⊥ = 0 = v ⊥ , then β ( w , v ) = 1 . In Figure 3, as a case study , w e examine the most extreme words on the − − − − → softball − − − − − − → fo otball direction. The ﬁve most extreme words (i.e. words with the highest p ositiv e or the low est negative pro jections onto 10 he she genius brilliant priest homemaker feminist divorce drafted earrings beautiful dress dancers modeling crafts dancer buddies guru sewing cocky pearls dance salon firepower ultrasound witch witches sassy builder tactical buddy burly tanning trimester mate scrimmage pageant babe command tote vases rule commit thighs journeyman brothers sisters queen beard breasts chap arrival browsing actresses fiance nuclear seconds caused drop seeking looks subject voters friend parts sites yard housing victims governor boys heavy slow user firms busy hoped ties letters identity folks quit sharply sons girlfriend daughters lobby grandmother cousin ladies cake treats wives nephew brass roses daddy fiancee dads reel girlfriends frost lust boyhood gals game wife lad ill ii pal hay vi rd son Figure 7: Selected w ords pro jected along tw o axes: x is a pro jection onto the diﬀerence b et ween the em b eddings of the words he and she , and y is a direction learned in the embedding that captures gender neutralit y , with gender neutral words ab o ve the line and gender sp eciﬁc words b elo w the line. Our hard debiasing algorithm remov es the gender pair asso ciations for gender neutral words. In this ﬁgure, the words ab o v e the horizontal line would all b e collapsed to the vertical line. − − − − → softball − − − − − − → fo otball ) are shown in the table. W ords such as r e c eptionist , waitr ess and homemaker are closer to softb al l than fo otb al l , and the β ’s b etw een these words and softb al l is substan tial (67%, 35%, 38%, resp ectiv ely). This suggests that the apparent similarity in the e m b eddings of these w ords to − − − − → softball can be largely explained b y gender biases in the embedding. Similarly , businessman and maestr o are closer to fo otb al l and this can also b e attributed largely to indirect gender bias, with β ’s of 31% and 42%, resp ectiv ely . 6 Debiasing algorithms The de biasing algorithms are deﬁned in terms of sets of words rather than just pairs, for generality , so that w e can consider other biases such as racial or religious biases. W e also assume that we ha ve a set of words to neutralize, which can come from a list or from the embedding as describ ed in Section 7. (In many cases it ma y b e easier to list the gender sp eciﬁc words not to neutralize as this set can b e muc h smaller.) The ﬁrst step, called Iden tify gender subspace , is to iden tify a direction (or, more generally , a subspace) of the embedding that captures the bias. F or the second step, we deﬁne tw o options: Neutralize and Equalize or Soften . Neutralize ensures that gender neutral words are zero in the gender subspace. Equalize p erfectly equalizes sets of words outside the subspace and thereby enforces the prop erty that any neutral word is equidistant to all words in each equality set. F or instance, if { grandmother , grandfather } and { guy , gal } w ere tw o equality sets, then after equalization b abysit would b e equidistan t to gr andmother and gr andfather and also equidistant to gal and guy , but presumably closer to the grandparents and further from the gal and guy . This is suitable for applications where one do es not wan t any such pair to display an y bias with resp ect to neutral words. The disadv antage of Equalize is that it remov es certain distinctions that are v aluable in certain applications. F or instance, one may wish a language mo del to assign a higher probability to the phrase to gr andfather a r e gulation ) than to gr andmother a r e gulation since gr andfather has a meaning that gr andmother do es not – equalizing the tw o remo ves this distinction. The Soften algorithm reduces the diﬀerences b et w een these sets 11 while maintaining as muc h similarity to the original embedding as p ossible, with a parameter that controls this trade-oﬀ. T o deﬁne the algorithms, it will b e conv enien t to introduce some further notation. A subspace B is deﬁned b y k orthogonal unit vectors B = { b 1 , . . . , b k } ⊂ R d . In the case k = 1 , the subspace is simply a direction. W e denote the pro jection of a vector v onto B by , v B = k X j =1 ( v · b j ) b j . This also means that v − v B is the pro jection onto the orthogonal subspace. Step 1: Identify gender subspace . Inputs: w ord sets W , deﬁning sets D 1 , D 2 , . . . , D n ⊂ W as well as em b edding  ~ w ∈ R d  w ∈ W and in teger parameter k ≥ 1 . Let µ i := X w ∈ D i ~ w / | D i | b e the means of the deﬁning sets. Let the bias subspace B be the ﬁrst k ro ws of SVD( C ) where C := n X i =1 X w ∈ D i ( ~ w − µ i ) T ( ~ w − µ i )  | D i | . Step 2a: Hard de-biasing (neutralize and equalize) . A dditional inputs: words to neutralize N ⊆ W , family of equality sets E = { E 1 , E 2 , . . . , E m } where each E i ⊆ W . F or each word w ∈ N , let ~ w b e re-embedded to ~ w := ( ~ w − ~ w B )  k ~ w − ~ w B k . F or eac h set E ∈ E , let µ := X w ∈ E w / | E | ν := µ − µ B F or eac h w ∈ E , ~ w := ν + p 1 − k ν k 2 ~ w B − µ B k ~ w B − µ B k Finally , output the subspace B and the new embedding  ~ w ∈ R d  w ∈ W . Equalize equates each set of w ords outside of B to their simple av erage ν and then adjusts vectors so that they are unit length. It is p erhaps easiest to understand by thinking separately of the tw o comp onents ~ w B and ~ w ⊥ B = ~ w − ~ w B . The latter ~ w ⊥ B are all simply equated to their av erage. Within B , they are centered (mo ved to mean 0) and then scaled so that each ~ w is unit length. T o motiv ate why we cen ter, b ey ond the fact that it is common in machine learning, consider the bias direction b eing the gender direction ( k = 1 ) and a gender pair such as E = { male , female } . As discussed, it so happ ens that b oth words are p ositiv e (female) in the gender direction, though female has a greater pro jection. One can only sp eculate as to wh y this is the case, e.g., p erhaps the frequency of text such as male nurse or male esc ort or she was assaulte d by the male . How ever, b ecause female has a greater gender comp onen t, after centering the tw o will b e symmetrically balanced across the origin. If in stead, we simply scaled each v ector’s comp onen t in the bias direciton without cen tering, male and female w ould hav e exactly the same embedding and we would lose analogies suc h as father:male :: mother:female . Before deﬁning the Soften alternative step, we note that Neutralizing and Equalizing completely remov e pair bias. Observ ation 1. After Steps 1 and 2a, for any gender neutr al wor d w any e quality set E , and any two wor ds e 1 , e 2 ∈ E , ~ w · ~ e 1 = w · ~ e 2 and k ~ w − ~ e 1 k = k ~ w − ~ e 2 k . F urthermor e, if E =  { x, y }| ( x, y ) ∈ P  ar e the sets of p airs deﬁning PairBias, then PairBias = 0 . 12 Pr o of. Step 1 ensures that ~ w B = 0 , while step 2a ensures that ~ e 1 − v ece 2 lies en tirely in B . Hence, their inner pro duct is 0 and ~ w · ~ e 1 = w · ~ e 2 . Lastly , k ~ w − ~ e 1 k = k ~ w − ~ e 2 k follo ws from the fact that k u 1 − u 2 k 2 = 2 − 2 u 1 · u 2 for unit v ectors u 1 , u 2 and P airBias b eing 0 follo ws trivially from the deﬁnition of PairBias. Step 2b: Soft bias correction . Overloading the notation, w e let W ∈ R d ×| v ocab | denote the matrix of all embedding vectors and N denote the matrix of the embedding vectors corresp onding to gender neutral w ords. W and N are learned from some corpus and are inputs to the algorithm. The desired debiasing transformation T ∈ R d × d is a linear transformation that seeks to preserve pairwise inner pro ducts b et ween all the word vectors while minimizing the pro jection of the gender neutral words onto the gender subspace. This can b e formalized as the following optimization problem min T k ( T W ) T ( T W ) − W T W k 2 F + λ k ( T N ) T ( T B ) k 2 F where B is the gender subspace learned in Step 1 and λ is a tuning parameter that balances the ob jective of preserving the original em b edding inner pro ducts with the goal of reducing gender bias. F or λ large, T w ould remov e the pro jection onto B from all the vectors in N , whic h corresp onds exactly to Step 2a. In the exp eriment, we use λ = 0 . 2 . The optimization problem is a semi-deﬁnite program and can b e solved eﬃcien tly . The output embedding is normalized to hav e unit length, ˆ W = { T w / k T w k 2 , w ∈ W } . 7 Determining gender neutral w ords F or practical purp oses, since there are man y fewer gender sp eciﬁc w ords, it is more eﬃcient to enumerate the set of gender sp eciﬁc words S and take the gender neutral w ords to be the compliment, N = W \ S . Using dictionary deﬁnitions, we derive a subset S 0 of 218 words out of the words in w2vNEWS. Recall that this em b edding is a subset of 26,377 words out of the full 3 million words in the embedding, as describ ed in Section 3. This base list S 0 is given in App endix C. Note that the choice of words is sub jective and ideally should b e customized to the application at hand. W e generalize this list to the entire 3 million words in the Go ogle News embedding using a linear classiﬁer, resulting in the set S of 6,449 gender-sp eciﬁc words. More sp eciﬁcally , w e trained a linear Supp ort V ector Mac hine (SVM) with the default regularization parameter of C = 1 . 0 . W e then ran this classiﬁer on the remaining w ords, taking S = S 0 ∪ S 1 , where S 1 w ere the words lab eled as gender sp eciﬁc by our classiﬁer among the w ords in the entire embedding that were not in the 26,377 words of w2vNEWS. Using 10-fold cross-v alidation to ev aluate the accuracy of this pro cess, we ﬁnd an F -score of . 627 ± . 102 based on stratiﬁed 10-fold cross-v alidation. The binary accuracy is well ov er 99% due to the im balanced nature of the classes. F or another test of how accurately the embedding agrees with our base set of 218 w ords, we ev aluate the class-balanced error by re-weigh ting the examples so that the p ositiv e and negative examples hav e equal w eights, i.e., w eighting each class in verse prop ortionally to the num b er of samples from that class. Here again, we use stratiﬁed 10-fold cross v alidation to ev aluate the error. Within each fold, the regularization parameter was also chosen by 10-fold (neste d) cross v alidation. The av erage (balanced) accuracy of the linear classiﬁers, across folds, w as 95 . 12% ± 1 . 46% with 95% conﬁdence. Figure 7 illustrates the results of the classiﬁer for separating gender-sp eciﬁc words from gender-neutral w ords. T o make the ﬁgure legible, we show a subset of the words. The x -axis corresp ond to pro jection of w ords onto the − → she − − → he direction and the y -axis corresp onds to the distance from the d ecision b oundary of the trained SVM. 8 Debiasing results W e ev aluated our debiasing algorithms to ensure that they preserve the desirable prop erties of the original em b edding while reducing b oth direct and indirect gender biases. 13 R G WS analogy Before 62.3 54.5 57.0 Hard-debiased 62.4 54.1 57.0 Soft-debiased 62.4 54.2 56.8 T able 1: The columns show the p erformance of the original w2vNEWS embedding (“b efore”) and the debiased w2vNEWS on the standard ev aluation metrics measuring coherence and analogy-solving abilities: RG [ 32 ], WS [ 12 ], MSR-analogy [ 26 ]. Higher is b etter. The results show that the p erformance do es not degrade after debiasing. Note that we use a subset of vocabulary in the exp erimen ts. Therefore, the p erformances are low er than the previously published results. Direct Bias. First we used the same analogy generation task as b efore: for b oth the hard-debiased and the soft-debiased embeddings, we automatically generated pairs of words that are analogous to she-he and asked cro wd-work ers to ev aluate whether these pairs reﬂect gender stereot yp es. Figure 8 shows the results. On the initial w2vNEWS embedding, 19% of the top 150 analogies were judged as showing gender stereotypes by a ma jority of the ten work ers. After applying our hard debiasing algorithm, only 6% of the new embedding w ere judged as stereotypical. As an example, consider the analogy puzzle, he to do ctor is as she to X . The original embedding returns X = nurse while the hard-debiased embedding ﬁnds X = physician . Moreov er the hard-debiasing algorithm preserved gender appropriate analogies such as she to ovarian c anc er is as he to pr ostate c anc er . This demonstrates that the hard-debiasing has eﬀectively reduced the gender stereotypes in the word embedding. Figure 8 also shows that the num ber of appropriate analogies remains similar as in the original embedding after executing hard-debiasing. This demonstrates that that the quality of the em b eddings is preserved. The details results are in App endix G. Soft-debiasing was less eﬀective in removing gender bias. T o further conﬁrms the qualit y of embeddings after debiasing, w e tested the debiased embedding on sev eral standard b enc hmarks that measu re whether related words hav e similar embeddings as well as how w ell the embedding p erforms in analogy tasks. T able 1 shows the results on the original and the new embeddings and the transformation do es not negatively impact the p erformance. Indirect bias. W e also inv estigated how the strict debiasing algorithm aﬀects indirect gender bias. Because w e do not hav e the ground truth on the indirect eﬀects of gender bias, it is challenging to quantify the p erformance of the algorithm in this regard. How ev er w e do see promising qualitativ e impro vemen ts, as sho wn in Figure 3 in the softb al l , fo otb al l example. After applying the strict debias algorithm, w e rep eated the exp erimen t and show the most extreme words in the − − − − → softball − − − − − − → fo otball direction. The most extreme words closer to softb al l are no w inﬁelder and major le aguer in addition to pitcher , which are more relev an t and do not exhibit gender bias. Gender stereotypic asso ciations such are r e c eptionist , waitr ess and homemaker are mo ved do wn the list. Similarly , words that clearly show male bias, e.g. businessman , are also no longer at the top of the list. Note that the tw o most extreme words in the − − − − → softball − − − − − − → fo otball direction are pitcher and fo otb al ler . The similarities b et ween pitcher and softb al l and b et ween fo otb al ler and fo otb al l comes from the actual functions of these words and hence hav e little gender con tribution. These tw o words are essentially unc hanged by the debiasing algorithm. 9 Discussion W ord embeddings help us further our understanding of bias in language. W e ﬁnd a single direction that largely captures gender, that helps us capture asso ciations b et ween gender neutral words and gender as well as indirect inequality .The pro jection of gender neutral words on this direction enables us to quantify their degree of female- or male-bias. T o reduce the bias in an embedding, we change the embeddings of gender neutral words, by remo ving 14 Figure 8: Number of stereotypical (Left) and appropriate (Right) analogies generated by w ordembeddings b efore and after debiasing. their gender asso ciations. F or instance, nurse is mov ed to to b e equally male and female in the direction g . In addition, we ﬁnd that gender-sp eciﬁc w ords hav e additional biases b ey ond g . F or instance, gr andmother and gr andfather are b oth closer to wisdom than gal and guy are, which do es not reﬂect a gender diﬀerence. On the other hand, the fact that b abysit is so muc h closer to gr andmother than gr andfather (more than for other gender pairs) is a gender bias sp eciﬁc to gr andmother . By equating gr andmother and gr andfather outside of gender, and since we’v e remov ed g from b abysit , b oth gr andmother and gr andfather and equally close to b abysit after debiasing. By retaining the gender comp onen t for gender-sp eciﬁc w ords, we main tain analogies suc h as she:gr andmother :: he:gr andfather . Through empirical ev aluations, we sho w that our hard-debiasing algorithm signiﬁcan tly reduces b oth direct and indirect gender bias while preserving the utility of the embedding. W e ha ve also developed a soft-embedding algorithm which balances reducing bias with preserving the original distances, and could b e appropriate in sp eciﬁc settings. One p ersp ectiv e on bias in word embeddings is that it merely reﬂects bias in so ciety , and therefore one should attempt to debias so ciet y rather than word em b eddings. How ever, by reducing the bias in to da y’s computer systems (or at least not amplifying the bias), whic h is increasingly relian t on word embeddings, in a small wa y debiased word embeddings can hop efully contribute to reducing gender bias in so ciet y . At the v ery least, machine learning should not b e used to inadverten tly amplify these biases, as we hav e seen can naturally happ en. In sp eciﬁc applications, one migh t argue that gender biases in the em b edding (e.g. c omputer pr o gr ammer is closer to he ) could capture useful statistics and that, in these sp ecial cases, the original bi ased embeddings could b e used. How ev er given the p oten tial risk of having machine learning algorithms that amplify gender stereot yp es and discriminations, we recommend that w e should err on the side of neutrality and use the debiased em b eddings pro vided here as muc h as p ossible. In this pap er, we focus on quantifying and reducing gender bias in word em b eddings. Corpus of do cumen ts often con tain other undesirable stereot yp es and these can also b e reﬂected in the embedding vectors. The same w2vNEWS also exhibits strong racial stereotype. F or example, pro jecting all the o ccupation words onto the direction − − − − − − − → minorities − − − − − → whites , we ﬁnd that the most extreme o ccupations closer to whites are p arliamentarian , advo c ate , deputy , chanc el lor , le gislator , and lawyer . In contrast, the most extreme occupations at the minorites end are butler , fo otb al ler , so cialite , and cr o oner . It is a subtle issue to understand the direct and indirect bias due to racial, ethnic and cultural stereotypes. An imp ortan t direction of future work would b e to quan tify and remo ve these biases. While w e fo cus on English w ord embeddings, it is also an interesting direction to consider how the approac h and ﬁndings here w ould apply to other languages, esp ecially languages with grammatical gender where the deﬁnitions of most nouns carry a gender marker. 15 References [1] J. Angwin, J. Larson, S. Mattu, and L. Kirchner. Mac hine bias: There’s soft ware used across the country to predict future criminals. and it’s biased against blacks., 2016. [2] C. Asch w anden. The ﬁnkb einer test: What matters in stories ab out women scientists? DoubleXScienc e , 2013. [3] S. Baro cas and A. D. Selbst. Big data’s disparate impact. Available at SSRN 2477899 , 2014. [4] E. Beigman and B. B. Klebano v. Learning with annotation noise. In A CL , 2009. [5] A. J. Cuddy , E. B. W olf, P . Glick, S. Crotty , J. Chong, and M. I. Norton. Men as cultural ideals: Cultural v alues mo derate gender stereotype conten t. Journal of p ersonality and so cial psycholo gy , 109(4):622, 2015. [6] A. Datta, M. C. T schan tz, and A. Datta. Automated exp erimen ts on ad priv acy settings. Pr o c e e dings on Privacy Enhancing T e chnolo gies , 2015. [7] C. Dwork, M. Hardt, T. Pitassi, O. Reingold, and R. Zemel. F airness through aw areness. In Innovations in The or etic al Computer Scienc e Confer enc e , 2012. [8] J. Eisenstein, B. O’Connor, N. A. Smith, and E. P . Xing. Diﬀusion of lexical change in so cial media. PL oS ONE , pages 1–13, 2014. [9] M. F aruqui, J. Do dge, S. K. Jauhar, C. Dyer, E. Hovy , and N. A. Smith. Retroﬁtting word vectors to seman tic lexicons. In NAACL , 2015. [10] M. F eldman, S. A. F riedler, J. Mo eller, C. Scheidegger, and S. V enk atasubramanian. Certifying and remo ving disparate impact. In KDD , 2015. [11] C. F ellbaum, editor. W or dnet: A n Ele ctr onic L exic al Datab ase . The MIT Press, Cambridge, MA, 1998. [12] L. Finkelstein, E. Gabrilovic h, Y. Matias, E. Rivlin, Z. Solan, G. W olfman, and E. Ruppin. Placing searc h in context: The concept revisited. In WWW . A CM, 2001. [13] P . Glick and S. T. Fiske. The ambiv alent sexism in ven tory: Diﬀerentiating hostile and b enev olent sexism. Journal of p ersonality and so cial psycholo gy , 70(3):491, 1996. [14] J. Gordon and B. V an Durme. Rep orting bias and knowledge extraction. Automate d Know le dge Base Construction (AKBC) , 2013. [15] A. G. Greenw ald, D. E. McGhee, and J. L. Sc hw artz. Measuring individual diﬀerences in implicit cognition: the implicit asso ciation test. Journal of p ersonality and so cial psycholo gy , 74(6):1464, 1998. [16] C. Hansen, M. T osik, G. Go ossen, C. Li, L. Bay ev a, F. Berbain, and M. Rotaru. How to get the b est w ord vectors for resume parsing. In SNN A daptive Intel ligenc e / Symp osium: Machine L e arning 2015, Nijme gen. [17] J. Holmes and M. Meyerhoﬀ. The handb o ok of language and gender , volume 25. John Wiley & Sons, 2008. [18] O. İrso y and C. Cardie. Deep recursive neural netw orks for comp ositionalit y in language. In NIPS . 2014. [19] R. Jak obson, L. R. W augh, and M. Monville-Burston. On language . Harv ard Univ Pr, 1990. [20] J. T. Jost and A. C. Ka y . Exposure to b enev olen t sexism and complemen tary gender stereot yp es: consequences for sp eciﬁc and diﬀuse forms of system justiﬁcation. Journal of p ersonality and so cial psycholo gy , 88(3):498, 2005. 16 [21] M. Ka y , C. Matuszek, and S. A. Munson. Unequal representation and gender stereotypes in image search results for o ccupations. In Human F actors in Computing Systems . ACM, 2015. [22] T. Lei, H. Joshi, R. Barzila y , T. Jaakk ola, A. M. Katerina Tymoshenk o, and L. Marquez. Semi-sup ervised question retriev al with gated conv olutions. In NAACL . 2016. [23] O. Levy and Y. Goldb erg. Linguistic regularities in sparse and explicit word representations. In CoNLL , 2014. [24] T. Mikolo v, K. Chen, G . Corrado, and J. Dean. Eﬃcien t estimation of word represen tations in vector space. In ICLR , 2013. [25] T. Mikolo v, I. Sutskev er, K. Chen, G. S. Corrado, and J. Dean. Distributed representations of words and phrases and their comp ositionalit y . In NIPS . [26] T. Mikolo v, W.-t. Yih, and G. Zw eig. Linguistic regularities in contin uous space word representations. In HL T-NAACL , pages 746–751, 2013. [27] E. Nalisnick, B. Mitra, N. Craswell, and R. Caruana. Improving do cumen t ranking with dual w ord em b eddings. In www , April 2016. [28] B. A. Nosek, M. Bana ji, and A. G. Greenw ald. Harvesting implicit group attitudes and b eliefs from a demonstration w eb site. Gr oup Dynamics: The ory, R ese ar ch, and Pr actic e , 6(1):101, 2002. [29] D. Pedreshi, S. Ruggieri, and F. T urini. Discrimination-a ware data mining. In Pr o c e e dings of the 14th A CM SIGKDD international c onfer enc e on Know le dge disc overy and data mining , pages 560–568. ACM, 2008. [30] J. Pennington, R. So c her, and C. D. Manning. Glov e: Global vectors for w ord representation. In EMNLP , 2014. [31] K. Ross and C. Carter. W omen and news: A long and winding road. Me dia, Cultur e & So ciety , 33(8):1148–1165, 2011. [32] H. Rub enstein and J. B. Go o denough. Contextual correlates of synonym y . Communic ations of the ACM , 8(10):627–633, 1965. [33] E. Sapir. Sele cte d writings of Edwar d Sapir in language, cultur e and p ersonality , volume 342. Univ of California Press, 1985. [34] B. Schmidt. Rejecting the gender binary: a vector-space op eration. http://bookworm.benschmidt. org/posts/2015- 10- 30- rejecting- the- gender- binary.html , 2015. [35] J. P . Stanley . P aradigmatic woman: The prostitute. Pap ers in language variation , pages 303–321, 1977. [36] L. Sw eeney . Discrimination in online ad delivery . Queue , 11(3):10, 2013. [37] A. T orralba and A. Efros. Unbiased lo ok at dataset bias. In CVPR , 2012. [38] P . D. T urney . Domain and function: A dual-space mo del of semantic relations and comp ositions. Journal of Artiﬁcial Intel ligenc e R ese ar ch , pages 533–585, 2012. [39] C. W agner, D. Garcia, M. Jadidi, and M. Strohmaier. It’s a man’s wikip edia? assessing gender inequality in an online encyclop edia. In Ninth International AAAI Confer enc e on W eb and So cial Me dia , 2015. [40] D. Y ogatama, M. F aruqui, C. Dy er, and N. A. Smith. Learning word representations with hierarchical sparse co ding. In ICML , 2015. [41] I. Zliobaite. A surv ey on measuring indirect discrimination in machine learning. arXiv pr eprint arXiv:1511.00148 , 2015. 17 A Generating analogies W e now expand on diﬀerent p ossible metho ds for generating ( x, y ) pairs, given ( a, b ) for generating analogies a : x :: b : y . The ﬁrst and simplest metric is to consider scoring an analogy by k ( ~ a − ~ b ) − ( ~ x − ~ y ) . This may b e called the p ar al lelo gr am approach and, for the purp ose of ﬁnding the b est single y giv en a, b, x , it is equiv alen t to the most common approach to ﬁnding single word analogies, namely maximizing cos ( ~ y , ~ x + ~ b − ~ a ) called c osA dd in earlier work [ 26 ] since we assume all v ectors are unit length. This works w ell in some cases, but a weakness can b e seen that, for man y triples ( a, b, x ) , the closest word to x is y = x , i.e., x = arg min y k ( ~ a − ~ b ) − ( ~ x − ~ y ) k . As a result, the deﬁnition explicitly excludes the p ossibilit y of returning x itself. In these cases, y is often a word very similar to x , and in most of these cases suc h an algorithm pro duces t wo opp osing analogies: a : x :: b : y as well as a : y :: b : x , which violates a desideratum of analogies (see [38], section 2.2). Related issues are discussed in [ 38 , 23 ], the latter of which prop oses the 3CosMul ob jective to ﬁnding y giv en ( a, b, x ) : max y (1 + cos( ~ x, ~ y ))(1 + cos( ~ x, ~ b ) 1 + cos( ~ y , ~ a ) +  . The additional  is necessary so that the denominator is p ositiv e. This approach is designed for ﬁnding a single w ord y and not directly applicable for the problem of generating b oth x and y as the ob jective is not symmetric in x and y . In the spirit of their work, we note that a desired prop ert y is that the direction ~ a − ~ b should b e similar (in angle) to the direction ~ x − ~ y ev en if the magnitudes diﬀer. Interestingly , given ( a, b, x ) , the y that maximizes cos ( ~ a − ~ b, ~ x − ~ y ) is generally an extreme. F or instance, for a = he and b = she, for the v ast ma jorit y of words x , the word her maximizes the expression for y . This is due to the fact that the most signiﬁcant diﬀerence b et w een a random word x and the word her is that her is likely muc h more feminine than x . Since, from a p erceptual p oint of view it is easier to compare and contrast similar items than very diﬀerent items, we instead seek x and y that are not semantically similar, whic h is wh y our deﬁnition is restricted to k ~ x − ~ y k ≤ δ . As δ v aries from small to large, the analogies v ary from generating very similar x and y to very lo osely related x and y where their relationship is v ague and more “creativ e”. Finally , Figure 9 highlights diﬀerences b et ween analogies generated from our approac h and the corre- sp onding analogies generated by the ﬁrst approach mentioned ab o v e, namely minimizing: min x,y : x 6 = a,y 6 = b,x 6 = y k ( ~ a − ~ b ) − ( ~ x − ~ y ) k , (2) T o compare, we to ok the ﬁrst 100 analogies generated using the tw o approaches that did not hav e any gender-sp eciﬁc words. W e then display the ﬁrst 10 analogies from each list which do not o ccur in the other list of 100. B Learning the linear transform In the soft debiasing algorithm, w e need to solve the following optimization problem. min T k ( T W ) T ( T W ) − W T W k 2 F + λ k ( T N ) T ( T B ) k 2 F . Let X = T T T , then this is equiv alen t to the following semi-deﬁnite programming problem min X k W T X W − W T W k 2 F + λ k N T X B k 2 F s.t. X  0 . (3) The ﬁrst term ensures that the pairwise inner pro ducts are preserved and the second term induces the biases of gender neutral w ords onto the gender subspace to b e small. The user-sp eciﬁed parameter λ balances the t wo terms. 18 Analogies generated using eq. (2) Analogies generated using our approac h, eq. (1) p etite-dimin utiv e p etite-lanky sev enth inning-eighth inning v olleyball-fo otball sev enth-sixth in terior designer-architect east-w est bitc h-bastard tripled-doubled bra-pan ts breast cancer-cancer n urse-surgeon meter h urdles-meter dash feminine-manly thousands-tens glamorous-ﬂash y eigh t-seven registered n urse-physician unemplo yment rate-jobless rate cup cak es-pizzas Figure 9: First 10 diﬀerent she-he analogies generated using the parallelogram approach and our approach, from the top 100 she-he analogies not containing gen der sp eciﬁc words. Most of the analogies on the left seem to ha ve little connection to gender. Directly solving this SDP optimization problem is challenging. In practice, the dimension of matrix W is in the scale of 300 × 400 , 000 . The dimensions of the matrices W T X W and W T W are 400 , 000 × 400 , 000 , causing computational and memory issues. W e p erform singular v alue decomp osition on W , such that W = U Σ V T , where U and V are orthogonal matrices and Σ is a diagonal matrix. k W T X W − W T W k 2 F = k W T ( X − I ) W k 2 F = k V Σ U T ( X − I ) U Σ V T k 2 F = k Σ U T ( X − I ) U Σ k 2 F . (4) The last equality follows the fact that V is an orthogonal matrix and ( k V Y V T k 2 F = tr ( V Y T V T V Y V T ) = tr ( V Y T Y V T ) = tr ( Y T Y V T V ) = tr ( Y T Y ) = k Y k 2 F .) Substituting Eq. (4) to Eq. (3) gives min X k Σ U T ( X − I ) U Σ k 2 F + λ k P X S T k 2 F s.t. X  0 . (5) Here Σ U T ( X − I ) U Σ is a 300 × 300 matrix and can b e solved eﬃciently . The solution T is the debiasing transformation of the w ord embedding. C Details of gender sp eciﬁc w ords base set This section gives precise details of how we derived our list of gender neutral words. Note that the choice of gender neutral w ords is partly sub jectiv e. Some words are most often asso ciated with females or males but ha ve exceptions, such as b e ar d (b earded w omen), estr o gen (men hav e small amounts of the hormone estrogen), and r abbi (reformed Jewish congregations recognize female rabbis). There are also many words that hav e m ultiple senses, some of which are gender neutral and others of which are gender sp eciﬁc. F or instance, the profession of nursing is gender neutral while nursing a baby (i.e., breastfeeding) is only p erformed b y women. T o derive the base subset of words from w2vNEWS, for each of the 26,377 w ords in the ﬁltered embedding, w e selected words whose deﬁnitions include any of the following w ords in their singular or plural forms: female, male, woman, man, girl, b oy, sister, br other, daughter, son, gr andmother, gr andfather, wife, husb and . Deﬁnitions were taken from W ordnet [ 11 ] (in the case where a word had multiple senses/synsets, we chose the deﬁnition whose corresp onding lemma had greatest frequency in terms of its coun t). This lis t of hundreds of words contains most gender sp eciﬁc words of interest but also contains some gender neutral words, e.g., the deﬁnition of mating is “the act of pairing a male and female for repro ductiv e purp oses.” Even though the word female is in the deﬁnition, mating is not gender sp eciﬁc. W e wen t through this list and manually selected those words that were clearly gender sp eciﬁc. Motiv ated by the application of improving web search, 19 w e used a strict deﬁnition of gender sp eciﬁcit y , so that when in doubt a word w as deﬁned to b e gender neutral. F or instance, clothing words (e.g., the deﬁnition of vest is “a collarless men’s undergarmen t for the upp er part of the b ody”) were classiﬁed as gender neutral since there are undoubtedly p eople of every gender that w ear any giv en t yp e of clothing. After this ﬁltering, we were left with the following list of 218 gender-sp eciﬁc w ords (sorted by word frequency): he, his, her, she, him, man, women, men, woman, sp okesman, wife, himself, son, mother, father, chairman, daughter, husb and, guy, girls, girl, b oy, b oys, br other, sp okeswoman, female, sister, male, herself, br others, dad, actr ess, mom, sons, girlfriend, daughters, lady, b oyfriend, sisters, mothers, king, businessman, gr andmother, gr andfather, de er, ladies, uncle, males, c ongr essman, gr andson, bul l, que en, businessmen, wives, widow, nephew, bride, females, aunt, pr ostate c anc er, lesbian, chairwoman, fathers, moms, maiden, gr anddaughter, younger br other, lads, lion, gentleman, fr aternity, b achelor, nie c e, bul ls, husb ands, princ e, c olt, salesman, hers, dude, b e ar d, ﬁl ly, princ ess, lesbians, c ouncilman, actr esses, gentlemen, stepfather, monks, ex girlfriend, lad, sp erm, testoster one, nephews, maid, daddy, mar e, ﬁanc e, ﬁanc e e, kings, dads, waitr ess, maternal, her oine, nie c es, girlfriends, sir, stud, mistr ess, lions, estr ange d wife, womb, gr andma, maternity, estr o gen, ex b oyfriend, widows, gelding, diva, te enage girls, nuns, czar, ovarian c anc er, c ountrymen, te enage girl, p enis, bloke, nun, brides, housewife, sp okesmen, suitors, menop ause, monastery, motherho o d, br ethr en, stepmother, pr ostate, hostess, twin br other, scho olb oy, br otherho o d, ﬁl lies, stepson, c ongr esswoman, uncles, witch, monk, viagr a, p aternity, suitor, sor ority, macho, businesswoman, eldest son, gal, statesman, scho olgirl, father e d, go ddess, hubby, step daughter, blokes, dudes, str ongman, uterus, gr andsons, studs, mama, go dfather, hens, hen, mommy, estr ange d husb and, elder br other, b oyho o d, b aritone, gr andmothers, gr andp a, b oyfriends, feminism, c ountryman, stal lion, heir ess, que ens, witches, aunts, semen, fel la, gr anddaughters, chap, widower, salesmen, c onvent, vagina, b e au, b e ar ds, handyman, twin sister, maids, gals, housewives, horsemen, obstetrics, fatherho o d, c ouncilwoman, princ es, matriar ch, c olts, ma, fr aternities, p a, fel las, c ouncilmen, dowry, b arb ershop, fr aternal, b al lerina D Questionnaire for generating gender stereot ypical w ords T ask: for each category , please en ter 10 or more words, separated b y commas. W e are lo oking for a v ariet y of creative answers – this is a mentally c hallenging HIT that will make you think. • 10 or more comma-separated words deﬁnitionally asso ciated with males. Examples: dude, menswe ar, king, p enis , ... • 10 or more comma-separated words deﬁnitionally asso ciated with females. Examples: que en, Jane, girl , ... • 10 or more comma-separated words stereot ypically asso ciated with males Examples: fo otb al l, janitor, c o cky , ... • 10 or more comma-separated words stereot ypically asso ciated with females Examples: pink, sewing, c aring, sassy, nurse , ... Thank y ou for your help in making Artiﬁcially Intelligen t systems that aren’t prejudiced. :-) E Questionnaire for generating gender stereot ypical analogies An analogy describ es tw o pairs of words where the relationship b et w een the tw o words in eac h pair is the same. An example of an analogy is apple is to fruit as asp ar agus is to ve getable (denoted as ap- ple:fruit::asparagus:v egetable). W e need your help to improv e our analogy generating system. 20 T ask: please enter 10 or more analogies reﬂecting gender stereotypes, separated by commas . W e are lo oking for a v ariety of creative answ ers – this is a mentally challenging HIT that will make y ou think. Examples of stereotypes • tall : man :: short : woman reﬂects a cultural stereotype that men are tall and women are short. • do ctor : man :: n urse : woman reﬂects a stereot yp e that do ctors are typically men and n urses are t ypically women. F Questionnaire for rating stereot ypical analogies An analogy describ es tw o pairs of words where the relationship b et w een the tw o words in eac h pair is the same. An example of an analogy is apple is to fruit as asp ar agus is to ve getable (denoted as ap- ple:fruit::asparagus:v egetable). W e need your help to improv e our analogy generating system. T ask: Whic h analogies are stereot yp es? Which ones are appropriate analogies? • Examples of stereotype analogies tall : man :: short : w oman do ctor : man :: nurse : woman • Examples of appropriate analogies King: man :: Queen : w oman brother : man :: sister : w oman John : man :: Mary : w oman His : man :: Hers : w oman salesman : man :: saleswoman : w oman p enis : man :: v agina : w oman W ARNING: This HIT may contain adult conten t. W ork er discretion is advised. Chec k the analogies that are stereotypes ... Chec k the analogies that are nonsensical ... Chec k the analogies that are nonsensical ... An y suggestions or comments on the hit? Optional feedback G Analogies Generated by W ord Em b eddings After executing hard debiasing Before executing debiasing Analogy Appropriate Biased Analogy Appropriate Biased hostess:bartender 1 8 midwife:doctor 1 10 ballerina:dancer 0 7 sewing:carpentry 2 9 colts:mares 6 7 pediatrician:orthop edic_surgeon 0 9 ma:na 8 7 registered_n urse:physician 1 9 salesp erson:salesman 1 7 housewife:shopkeeper 1 9 21 div a:superstar 4 7 skirts:shorts 0 9 witc hes:v ampires 1 7 n urse:surgeon 1 9 hair_salon:barb ershop 4 6 in terior_designer:architect 1 8 maid:housek eep er 3 6 softball:baseball 4 8 soprano:baritone 4 5 blond:burly 2 8 fo ot y:blok es 0 5 nanny:c hauﬀeur 1 8 maids:serv an ts 4 5 feminism:conserv atism 2 8 dictator:strongman 0 5 adorable:go ofy 0 8 bac helor:bachelor_degree 7 4 vocalists:guitarists 0 8 witc h:witchcraft 0 4 cosmetics:pharmaceuticals 1 8 gaﬀer:lads 1 3 whore:cow ard 0 7 con ven t:monastery 8 3 vocalist:guitarist 1 7 hen:co c k 8 2 p etite:lanky 1 7 aldermen:councilmen 0 2 salesp erson:salesman 1 7 girlfriend:friend 0 2 sassy:snappy 2 7 housewife:homemak er 2 2 div a:sup erstar 4 7 maternal:infan t_mortality 1 2 charming:aﬀable 2 6 b eau:lo v er 1 2 giggle:ch uckle 1 6 mistress:prostitute 0 2 witch:demon 2 6 heroine:protagonist 2 2 volleyball:football 1 6 heiress:so cialite 2 2 feisty:mild_mannered 0 6 teenage_girl:teenager 3 2 cup cak es:pizzas 1 6 estrogen:testosterone 9 2 dolls:replicas 0 6 actresses:actors 10 1 netball:rugb y 0 6 blok es:bloke 1 1 hairdresser:barb er 6 5 girlfriends:buddies 6 1 soprano:baritone 4 5 compatriot:coun tryman 3 1 gown:blazer 6 5 compatriots:coun trymen 2 1 glamorous:ﬂashy 2 5 gals:dudes 10 1 sweater:jersey 0 5 eldest:elder_brother 1 1 feminist:lib eral 0 5 sp erm:em bry os 2 1 bra:pants 2 5 mother:father 10 1 reb ounder:pla ymaker 0 5 w edlo c k:fathered 0 1 n ude:shirtless 0 5 mama:fella 7 1 judgmental:arrogan t 1 4 lesbian:ga y 8 1 b oobs:ass 1 4 kid:guy 1 1 salon:barb ershop 7 4 carp en ter:handyman 5 1 lov ely:brilliant 0 4 she:he 9 1 practicality:durabilit y 0 4 herself:himself 10 1 singer:frontman 0 4 her:his 10 1 gorgeous:magniﬁcent 2 4 uterus:in testine 1 1 p on ytail:mustac he 2 4 queens:kings 10 1 feminists:so cialists 0 4 female:male 9 1 bras:trousers 5 4 w omen:men 10 1 wedding_dress:tuxedo 6 4 pa:mo 9 1 violinist:virtuoso 0 4 n un:monk 7 1 handbag:briefcase 8 3 matriarc h:patriarch 9 1 giggling:grinning 0 3 n uns:priests 9 1 kids:guys 3 3 menopause:pub ert y 2 1 b eautiful:ma jestic 1 3 ﬁance:ro ommate 0 1 feminine:manly 8 3 daugh ter:son 9 1 conv ent:monastery 8 3 22 daugh ters:sons 10 1 sexism:racism 0 3 sp ok esw oman:sp ok esman 10 1 pink:red 0 3 p olitician:statesman 1 1 blouse:shirt 6 3 stallion:stud 7 1 bitch:bastard 8 2 suitor:tak eov er_bid 8 1 wig:b eard 4 2 w aitress:waiter 10 1 hysterical:comical 0 2 lady:w aitress 0 1 male_counterparts:coun terparts 1 2 bride:w edding 0 1 b eaut y:grandeur 0 2 wido wer:wido wed 3 1 cheerful:jo vial 0 2 h usband:younger_brother 3 1 breast_cancer:lymphoma 3 2 actress:actor 9 0 heiress:magnate 6 2 m ustache:beard 0 0 estrogen:testosterone 9 2 facial_hair:b eards 0 0 starlet:youngster 2 2 suitors:bidders 6 0 Mary:John 9 1 girl:b o y 9 0 actresses:actors 10 1 c hildho o d:boyhoo d 1 0 middle_aged:b earded 0 1 girls:b o ys 10 0 mums:blok es 5 1 coun terparts:brethren 4 0 girlfriends:buddies 6 1 brides:bridal 1 0 mammogram:colonoscopy 0 1 sister:brother 10 0 compatriot:coun tryman 3 1 friendship:brotherho od 3 0 luscious:crisp 0 1 sisters:brothers 9 0 gals:dudes 10 1 businessw oman:businessman 9 0 siblings:elder_brother 1 1 businessp eople:businessmen 1 0 mother:father 10 1 c hairwoman:c hairman 10 0 bab e:fella 9 1 bastard:c hap 0 0 lesbian:gay 8 1 hens:c hick ens 3 0 breasts:genitals 0 1 viagra:cialis 1 0 wonderful:great 0 1 ﬁlly:colt 9 0 she:he 9 1 ﬁllies:colts 8 0 herself:himself 10 1 congressw oman:congressman 9 0 her:his 10 1 councilw oman:councilman 9 0 mommy:kid 0 1 wife:cousin 0 0 queens:kings 10 1 mom:dad 10 0 female:male 9 1 momm y:daddy 10 0 women:men 10 1 moms:dads 9 0 b o yfriend:pal 0 1 wido w:deceased 0 0 matriarch:patriarc h 9 1 gal:dude 9 0 nun:priest 10 1 stepmother:eldest_son 3 0 breast:prostate 9 1 deer:elk 1 0 daughter:son 9 1 estranged_h usband:estranged 0 0 daughters:sons 10 1 ex_b o yfriend:ex_girlfriend 7 0 sp ok eswoman:spokesman 10 1 wido ws:families 4 0 fabulous:terriﬁc 3 1 motherho od:fatherho od 10 0 headscarf:turban 6 1 mothers:fathers 10 0 w aitress:waiter 10 1 guys:fellas 1 0 husband:y ounger_brother 3 1 feminism:feminist 1 0 hers:yours 2 1 w omb:fetus 0 0 teenage_girls:youths 0 1 sororit y:fraternity 9 0 actress:actor 9 0 lesbians:ga ys 9 0 blonde:blond 4 0 mare:gelding 7 0 girl:b o y 9 0 23 fella:gen tleman 1 0 childhoo d:b o yho od 1 0 ladies:gen tlemen 10 0 girls:b oys 10 0 b o yfriends:girlfriend 3 0 sister:brother 10 0 go ddess:god 9 0 sisters:brothers 9 0 grandmother:grandfather 10 0 businesswoman:businessman 9 0 grandma:grandpa 9 0 chairw oman:chairman 10 0 grandmothers:grandparen ts 5 0 ﬁlly:colt 9 0 granddaugh ter:grandson 10 0 ﬁllies:colts 8 0 granddaugh ters:grandsons 9 0 congresswoman:congressman 9 0 me:him 2 0 councilwoman:councilman 9 0 queen:king 10 0 mom:dad 10 0 y oungster:lad 1 0 moms:dads 9 0 elephan t:lion 0 0 gal:dude 9 0 elephan ts:lions 0 0 motherho od:fatherho o d 10 0 manly:mac ho 4 0 mothers:fathers 10 0 females:males 10 0 sorority:fraternit y 9 0 w oman:man 8 0 mare:gelding 7 0 ﬁancee:married 4 0 lady:gentleman 9 0 maternit y:midwives 1 0 ladies:gentlemen 10 0 monks:monasteries 0 0 go ddess:go d 9 0 niece:nephew 9 0 grandmother:grandfather 10 0 nieces:nephews 9 0 grandma:grandpa 9 0 h ubby:pal 1 0 granddaughter:grandson 10 0 obstetrics:p ediatrics 3 0 granddaughters:grandsons 9 0 v agina:penis 10 0 kinda:guy 1 0 princess:prince 9 0 heroine:hero 9 0 colon:prostate 6 0 me:him 2 0 o v arian_cancer:prostate_cancer 10 0 queen:king 10 0 salesp eople:salesmen 2 0 females:males 10 0 semen:saliv a 7 0 woman:man 8 0 sc ho olgirl:sc ho olb o y 8 0 niece:nephew 9 0 replied:sir 0 0 nieces:nephews 9 0 sp ok esp eople:spokesmen 0 0 v agina:p enis 10 0 b o yfriend:stepfather 1 0 princess:prince 9 0 step daugh ter:stepson 9 0 ov arian_cancer:prostate_cancer 10 0 teenage_girls:teenagers 1 0 schoolgirl:schoolb o y 8 0 hers:theirs 0 0 sp okespeople:sp ok esmen 0 0 t win_sister:twin_brother 9 0 step daugh ter:stepson 9 0 aun t:uncle 9 0 t win_sister:twin_brother 9 0 aun ts:uncles 10 0 aunt:uncle 9 0 h usbands:wives 7 0 aunts:uncles 10 0 H Debiasing the full w2vNEWS em b edding. In the main text, w e fo cused on the results from a cleaned version of w2vNEWS consisting of 26,377 low er-case w ords. W e hav e also applied our hard debiasing algorithm to the full w2vNEWS dataset. Ev alution based on the standard metrics sho ws that the debiasing do es not degrade the utility of the em b edding (T able 3). 24 R G WS analogy Before 76.1 70.0 71.2 Hard-debiased 76.5 69.7 71.2 Soft-debiased 76.9 69.7 71.2 T able 3: The columns show the p erformance of the original, complete w2vNEWS embedding (“b efore”) and the debiased w2vNEWS on the standard ev aluation metrics measuring coherence and analogy-solving abilities: R G [ 32 ], WS [ 12 ], MSR-analogy [ 26 ]. Higher is b etter. The results show that the p erformance do es not degrade after debiasing. 25

Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings

Original Paper

Comments & Academic Discussion

Leave a Comment

Original Paper

Related Papers

Comments & Academic Discussion

Leave a Comment