Three ways inequity gets in

This module builds on Bias in AI from the AI Literate course. Researchers look for bias at three points.

The data. A system learns from past decisions and from text people wrote. In 2018, Reuters reported that Amazon dropped a hiring tool that penalized resumes containing the word “women’s.” In 2024, University of Washington researchers ran identical resumes through AI models with different names attached. The models favored white-associated names 85% of the time.

The target. Builders choose what to predict, and that choice can carry bias on its own. A 2019 study in Science examined a health tool that predicted future medical costs as a stand-in for need. Less money is spent on Black patients with the same need, so the tool underrated how sick they were. Fixing the target would have raised the share of Black patients flagged for extra help from 17.7% to 46.5%.

Where it is used. In 2020, England replaced cancelled exams with a formula that leaned on each school’s past results. It lowered 35.6% of grades by one level, and students at larger state schools were hit harder. After protests, regulators dropped it on August 17, 2020.

A proxy variable (show definition) lets bias in even after a protected trait is removed. In 2022, the US Justice Department settled with Meta over housing ads. It alleged that the ad delivery system relied in part on race, national origin, and sex to decide who saw each ad.