AI and gender diversity

AI can reinforce discrimination when it learns from unequal data or is deployed without adequate checks. It can also help identify patterns of exclusion. The outcome depends on what a system is designed to do, how it is tested and who can challenge its decisions.

When Joy Buolamwini was a graduate student at MIT, facial-analysis software could not detect her dark-skinned face until she put on a white mask. The technology worked, but not for her. She described that experience as an example of the cost of systems that fail to account for human diversity.

In 2018, Buolamwini and Timnit Gebru’s Gender Shades study tested three commercial systems that classified faces using binary gender labels. Error rates reached 34.7% for darker-skinned women, compared with a maximum of 0.8% for lighter-skinned men. This study examined gender classification, not whether systems could identify a particular person. Its binary categories also did not capture the full range of gender identities.

The lesson reaches beyond those systems: an overall accuracy score can hide substantial differences in who a technology serves well.

How does bias enter a system

Bias can enter through training data, the outcomes a model is taught to predict, how it is evaluated, and the decisions people make when using it. Improving the dataset matters, but it is only part of the work. NIST’s research on AI bias highlights human and institutional factors alongside computational ones.

“The answer to whether AI hinders diversity is a potential yes. It depends on the algorithm and who makes it. A supervised learning algorithm considers multiple factors as inputs when deciding on an output,” said Nagarro technologist Roopshree Das Gupta when interviewed for this article. Her question remains useful: when a system uses gender as an input, why does it need it?

Consider a hiring model trained to recognize patterns among previously successful candidates. If the organization has historically favored one group, the model may learn to reproduce that pattern. It could become efficient at repeating an unfair decision.

Simply removing gender from the input does not settle the question. Other information may preserve patterns associated with exclusion. Teams need to examine what the system is learning and whose opportunities its recommendations affect.

There is also a difference between using demographic data to rank candidates and using it, with appropriate safeguards, to check whether a selection process treats groups differently. Otherwise, organizations risk making disparities harder to measure rather than making the system fairer.

How language model can reproduce stereotypes

Language carries assumptions about who belongs in which role. Models trained on that language can pick up the associations too.

In research published in 2016, Tolga Bolukbasi and colleagues examined gender stereotypes in word embeddings, numerical representations of words learned from text. Their work showed how such representations could encode associations between gender and occupations. It was evidence about the models studied, not a finding that every language model behaves identically.

“Let's take the example of Google translator,” said Nagarro’s Kanupriya Saxena. In the translation examples she shared for this article, Hindi sentences about doctors, engineers, and nurses did not specify gender, yet the reported English outputs assigned different pronouns to different occupations. These observations illustrate the issue; they are not current product test results.

These examples reflect Google Translate outputs observed when the article was originally written. Results may differ today.

And the most disturbing one to find was:
Language Given Phrase Google Translator (in English) gender-neutral 

Translation tools have changed over time. Google Translate now documents gender-specific alternatives for some words and phrases, with availability varying by language and input length. Its 2021 translation research also explored using surrounding sentences to improve gender accuracy.

But pronouns are only one part of the problem. A system could use neutral language and still repeatedly cast men as decision-makers and women as supporters.

Vanessa Causemann, a machine learning engineer at Nagarro, when interviewed, drew a parallel with her own learning:

“I learned from a biased society, and I try to improve by using expert knowledge I gained in recent years (correcting my thoughts before I express or infer upon them).”

She distinguished between stereotypes expressed through individual words and those conveyed through broader context. Her examples of the first were:

“Her job is being [a housemaker]”

“His job is to be [a programmer]”

Her contextual examples were:

“She dealt with the issue by [perceiving the positions of the people around her and finding a balanced solution everyone could accept]”

“He dealt with the issue by [calculating the different outputs and finding the most efficient solution].”

“Therefore, I believe global biases contribute more subtly to reproducing stereotypes, and just censoring specific words (local biases) will not solve the problem – neither for society nor in AI,” she said.

For a business using AI to write job ads or describe employees, that means reviewing the roles, qualities and authority assigned to people, as well as the words used for them.

Where AI can help organizations become mroe progressive

AI can help teams investigate uneven outcomes and review content for potentially exclusionary patterns. These are uses to evaluate, not benefits to assume.

For example, an organization could analyze its hiring funnel to see where candidates from different groups are disproportionately screened out. That pattern would be a reason to investigate the process, not proof of discrimination or an instruction to automate the next decision.

Tools can also help review job descriptions. However, not every tool that flags biased language uses AI. Gender Decoder checks text against research-based lists of gender-coded words. It can support a review of wording; evaluating a hiring process requires checking what happens to candidates throughout that process.

Diverse teams can bring additional perspectives to the questions an organization asks. Their involvement must extend to setting requirements, choosing evaluation criteria and challenging deployment decisions. Representation alone does not certify a model as fair.

The business case also needs precise evidence. A 2018 BCG study found that companies reporting above-average management diversity also reported a higher share of revenue from recently launched products and services: 45%, compared with 26%. The 19-percentage-point difference concerns innovation revenue share. The study reports a correlation, so it does not establish that diversity alone caused the difference.

Addressing bias in AI

As AI becomes more widely used, it becomes increasingly important to consider how different genders and racial groups are represented in the data and in the teams developing these systems.

There are three points at which bias can be addressed:

• Reduce bias in the data.

• Address bias while training the model.

• Adjust for bias after modeling.

Debiasing these systems is not simple. Recognizing biases in data and algorithms is a starting point. But the data isn't going to fix itself, is it? Nor can we move forward by simply hoping or assuming that the data will improve with better gender representation in the future.

“The need is to have strong compliances to enforce regulation of such biases at every model, every organization level, else the bias will keep on building in the AI models,” says Kanupriya.

The policy landscape has changed since these interviews. The European Commission’s AI Act overview, for example, identifies employment-related AI among high-risk uses. The applicable requirements and implementation dates depend on the particular system and jurisdiction.

How to reduce harmful bias in practice?

There are always two sides to a coin. So, just as AI can foster biases, it can also promote diversity and inclusion. AI can make organizations more diverse and inclusive by enabling logical and intelligent behavior and actions at the workplace. It can facilitate the transition to more inclusive work cultures, bereft of prejudices. Algorithms can learn to detect and remove biases in databases – another brick in the wall towards removing any unintentional and unconscious biases.

AI-powered hiring tools can alert the human resource department about any preferences in the candidate selection process. Intelligent text editors are already in use to rewrite job descriptions to appeal to candidates from under-represented groups. Gender Decoder, for instance, allows companies to remove 'linguistic gender coding' from job descriptions, which can discourage women candidates from applying for the job. HR departments can use deep neural networks to spot patterns in data and view any hidden preferences to see if an organization has favored a specific set of candidates over the other.

The significance of research and funding in reducing AI bias

Vanessa also emphasized the investment needed to make progress. In the original interviews, she said:

“Training with expert data is a costly project. Therefore, we need to equip top researchers with the means required to do so and offer incentives to encourage them. I am certain they have the intrinsic motivation required for the task. However, a lot of AI research is driven by huge companies such as Google, Facebook, and Open AI. It would be helpful if governments provided guidelines and funding to smaller research groups.

Optimally, we could create language models that are less biased than the average human and could serve as role models or tools to help check human language. ”

Expert feedback and research need to be supported by evaluation. A promising correction should be tested on the task and population it is meant to serve, including the errors or trade-offs it may introduce.

Inclusion must show in outcomes 

AI can make an unequal process faster. It can also help expose patterns people have overlooked. Neither outcome follows simply from calling a system intelligent.

The useful question is whether the organization can show who benefits, who bears the errors and what happens when someone challenges a result. That is where inclusion becomes something a business can assess and improve.