Algorithms and other computational models that appear to "think" are trained on massive datasets to learn how to identify and process information. The type of data depends on the goal of the researchers developing the model, but available datasets may raise issues of confidentiality and fairness, according to Penn State Institute for Computational and Data Sciences (ICDS) co-hire G. Gary Tan.
Tan, professor of computer science and engineering in the Penn State College of Engineering, is on the second year of a three-year project - funded by a $600,000 grant from the U.S. National Science Foundation - focused on helping researchers customize fair models for their projects by mitigating biases in existing machine learning models in open-source libraries such as Tensorflow or Scikit-Learn. The project is also focused on creating fairness customization recommendations for researchers in this space.
A machine learning model trains on a lot of data to learn how to make decisions based on various inputs, Tan said. However, that data and how the people involved label it or feed it to the models may be biased, even unconsciously, against specific groups of people, according to Tan. These unbalanced or insufficient data contain what researchers call "fairness bias," which could lead to unequal treatment of different groups by the models.
Tan said that these biases can affect the fairness of outcomes across various demographic categories such as race, sex, age or socioeconomic status. A model is considered unfair if it predicts different outcomes for two individuals that have the same features except for a protected attribute, like a model involved in hiring that tends to recommend men more than women, even though all other attributes are equal.
The research team, which includes Ashish Kumar and Vishnu Asutosh Dasu, Penn State doctoral students; Saeid Tizpaz-Niari, assistant professor of computer science at the University of Texas El Paso; Verya Monjezi, University of Texas El Paso doctoral student; and Ashutosh Trivedi, associate professor of computer science at the University of Colorado Boulder, applied software testing and fuzzing, a process of generating random inputs such as demographical information to check the fairness of machine learning models like deep neural networks and large language models. These types of inputs vary depending on the training of the model. For example, on a model that needs users' features to infer their income levels, inputs would include occupation, sex or race.
"We want to understand what customizations within the models may produce fair or unfair models," Tan said. "After we test and better understand what customizations can result in a fair model, we can recommend what customizations users should stay away from as they could result in an unfair model."
To measure model fairness, the research team used metrics such as equal opportunity difference and average odd difference, which measure the difference of statistics between two protected groups such as how likely a job applicant is hired between a male and a female group.
According to Tan, researchers take open-source machine learning libraries and customize the model by providing a set of hyperparameters - or variables to configure the model before training it - that fit the needs of the researchers' study. However, the space of the hyperparameters is large, and researchers often need guidance on how to customize those hyperparameters to build fair and functioning models.
Tan said interacting with researchers within ICDS encouraged him to think more broadly about the potential impact of his research.
"I presented this work to the ICDS community last year and received a lot of good feedback. A lot of the faculty build models from their data and are concerned about fairness. This kind of work can help them navigate the customizable space better," Tan said.
Future work will aim to better understand how neural networks - a type of machine learning model containing layers of artificial neurons that produce input signals and create outputs - could play a role in unfairness. The researchers plan to study if removing some of the neurons could correct biases without degrading the overall performance of the model.
Tan has presented his work at various conferences, including the 33rd Association for Computing Machinery SIGSOFT International Symposium on Software Testing and Analysis in 2024, the 45th International Conference on Software Engineering in 2023 and the 44th International Conference on Software Engineering in 2022.
