Red Flag 5. Algorithmic decision-making that can result in discrimination

Right to non-discrimination and associated impacts on economic, social and cultural rights, such as housing, employment opportunities, livelihoods and healthcare. The use of algorithms to automate decision-making in industries as diverse as online advertising, recruitment, healthcare, retail, and consumer finance is rarely – if ever – intended to undermine individuals’ right to non- discrimination. In fact, these tools have the potential to reduce or remove human bias from decision-making. Nonetheless, the opposite can also be true. Examples include:

• Social media, search and websites selling targeted advertising where:
  • Landlords have been enabled to exclude users based on race, age or gender. This has occurred when tools allowed agents to explicitly exclude certain groups from seeing housing ads. It can also happen in more subtle ways when companies allow targeting based on categories, such as age, marital status, and ZIP code, that are de facto proxies for certain groups. A series of court cases have led to many companies, including Facebook, committing to change their policies.
  • Ads for jobs placed on search platforms result in higher paying jobs being shown to more men than women, as in a 2015 case involving Google, reported in the Washington Post. Google has since taken steps to seek to address these, and similar examples, by updating its ad targeting policies.
  • Elderly populations have been targeted with fraudulent products or services to trick them out of cash or savings, ranging from anti-ageing products to funeral insurance and reverse mortgages. In one case, retired, politically conservative individuals in the United States were tricked into using much of their retirement savings to buy marked up gold and silver coins to “protect their money from the deep state.” Even though this broke the company’s rules, Facebook showed ads supporting this scheme more than 45 million times over a 21-month period.

• Discrimination in credit and insurance decision-making, for example:
  • Where loan providers rely on algorithms to analyze credit worthiness. In 2020, a report from the US-based Student Borrower Protection Center found that two lending institutions were effectively raising the cost of credit for students at academic institutions serving predominantly Hispanic and Black students.
  • Where insurance companies use algorithms to set the price of cover. 2018 reports alleged that UK car insurance firms were using algorithms that quoted higher premiums to people with non-Western names. The International Association of Insurance Providers published a paper cautioning the industry about these risks.
  • Where individuals’ credit limits are influenced by their social connections. Some companies are requesting mobile phone data and social media records in order to make judgements about credit worthiness. Where individuals’ do not have a credit history this can be one way to positively increase financial inclusion. But is also risks bringing down minorities’ scores if, for example, an individual has friends and family members who have not paid past debts.

• Recruitment industry tools that discriminate. The recruitment industry increasingly integrates automated decision-making as part of its value proposition to employers. In this context, discrimination can occur in a range of ways that have been well highlighted by researchers. High profile examples have included companies offering tools that:
  • Examine social media timelines and online postings about candidates with the risk that data which should not legally or ethically exclude an individual from a job – such as political opinion, sexual orientation or having family members convicted of a crime – ends up doing so.
  • Use Natural Language Processing to screen out candidate resumes that don’t fit an employer’s prior hiring patterns, which can perpetuate, racial, gender and other discrimination.
  • Allow employers conducting video interviews to grade verbal responses, tone, and facial expressions against high- performing employees potentially reinforcing biases and being unable to interpret non-white faces.
• Discrimination in healthcare. Healthcare professionals are increasingly looking to leverage the power of artificial intelligence to achieve breakthroughs in disease detection, diagnosis and patient care plans. The WHO has begun to flag associated ethical risks. In one case, an algorithmic tool sold to hospitals and insurers to predict health care needs was found to underestimate the needs of Black patients.

*For an explanation of how companies can be involved in human rights impacts, and their related responsibilities, see here.

Companies that make decisions and pursue actions based on algorithmic profiling and predictions can cause adverse impacts on human rights. An example would be a bank denying credit based on a tool that makes discriminatory recommendations.

Companies whose value proposition is to sell the capability to profile and predict to public or private third parties can contribute to adverse human rights impacts that those actors cause where their tools embed discriminatory biases. Contribution might arise due to the ways that customers are empowered to use these tools (such as by excluding certain groups) or may be more subtle such as when an algorithmic system has bias built into the data set.

An added complexity is that a single algorithmic system may integrate a number of inputs from different actors. For example, a data broker might provide training data; an AI research firm might license an algorithm and a developer might design the customer interface. Depending on the specific circumstances, each of these companies could contribute to adverse impacts.

In situations where companies have taken reasonable steps to prevent their tools contributing to discrimination and other human rights harms, they may nevertheless be linked to adverse impacts that business or government customers are causing.

Algorithmic systems may be used to advance a number of SDGs such as those listed below. Addressing impacts to people associated with this red flag can contribute to ensuring that this is done in ways that do not simultaneously impact people’s rights to non-discrimination, privacy and physical and mental health and well-being.

Unless otherwise indicated, the following questions draw heavily on Ranking Digital Rights’ Best Practices: Algorithms, Machine Learning and Automated Decision-Making, and the World Economic Forum’s White Paper on How to Prevent Discriminatory Outcomes in Machine Learning.

• Do we have a clear policy that describes how the company identifies and manages human rights risks related to the algorithmic system(s) we use?
• Do we inform customers or users about the existence of algorithmic profiling, describe how this works, explain the variables that influence the algorithm, and explain how users and customers may be impacted?
• Have we mapped and understood if any particular groups may be at an advantage or disadvantage in the context in which the system is being deployed? Do we have a method for checking if the output from an algorithm is decorrelated from protected or sensitive features?
• Do we seek a diversity of views about the potential risks of proposed models, especially from specific populations affected by the outcomes of algorithmic systems we use?
• Have we established robust diversity and inclusion policies at every level of the company, and notably in teams that develop algorithms, machine learning models, or other automated decision-making tools?
• Have we consulted with all the relevant domain experts whose interdisciplinary insights allow us to understand potential sources of bias or unfairness, and to design ways to counteract them?

• Do we assess whether any uses or use-cases of our algorithmic tools pose risks to human rights? Where we identify these, are we:
  • Creating clear and enforceable terms of use?
  • Engaging enterprise and government customers/users to educate and train them about how to use the tools without increasing human rights risks?
  • Do we have systems in place to monitor and review how customers are using our tools?
  • Are we clear about the actions we will take if we discover that our tools are being used in ways that lead to, or increase the likelihood of, adverse human rights impacts?
• Do we apply “rigorous pre-release trials to ensure that algorithmic systems will not amplify biases and error due to any issues with the training data, algorithms, or other elements of system design?”
• Have we outlined an ongoing system for evaluating fairness throughout the life cycle of our product? Do we have an escalation/emergency procedure to correct unforeseen cases of unfairness when we uncover them?
• Are we clearly committed to only buying and/or using training datasets that comprise data whose data subjects have provided informed content to having their data included in datasets used for this purpose?
  • Are we making dataset(s) used to train machine learning models, terms of use, and APIs available to allow third parties to provide and review the behavior of our system?
  • What reporting, grievance or redress processes and recourse do we have in place? Do we have a process in place to make necessary fixes to the design of the system based on reported issues or concerns?

Mitigation examples are current or historical examples for reference, but do not offer insight into their relative maturity or effectiveness. Moreover, some examples listed below are proposals for mitigating actions that have come from data science and engineering research institutes.

• Racial Discrimination and Emerging Digital Technologies: A Human Rights Analysis (A/HRC/44/57), Report from the UN Special Rapporteur on contemporary forms of racism.
• European Union Agency for Fundamental Rights (2018) Big Data: Discrimination in data-supported decision making.
• Harvard School of Law and Corporate Governance Artificial Intelligence and Ethics: An Emerging Area of Board Oversight Responsibility.
• UK Information Commissioners Office GDPR: Rights Related to Automated Decision-Making including Profiling.
• UN Secretary-General’s Roadmap for Digital Cooperation.