Computer Science Department , PhD Dissertation Defense, Mallak Alkhathlan " Group Fairness of Ranking-Based Decisions: Do Metrics Reflect Human Perception?"

Mallak Alkhathlan

 Ph.D. Candidate

WPI – Computer Science Department 

Tuesday, February 18, 2025

Time: 12:00PM - 1:00PM

Location: Gordon Library Conference Room

Dissertation Committee:

Dr. Elke Rundensteiner, Professor, WPI (Advisor)

Dr. Lane Harrison, Associate Professor, WPI (Co-Advisor)

Dr. Fabricio Murai, Assistant Professor, WPI

Dr. Mahsan Nourani, Assistant Professor, Northeastern University

 As automated decision-making systems increasingly influence critical aspects of daily life, concerns have grown over their potential to perpetuate biases and disproportionately harm marginalized groups. Ranking-based decision-making, widely used in hiring, university admissions, and bail decisions, can embed disparities due to reliance on historical data and predefined criteria. While fairness metrics aim to mitigate bias, a fundamental challenge remains: do these metrics align with human perceptions of fairness? This dissertation investigates this alignment through HCI studies on group fairness, ultimately guiding the design of ranking systems that are not only mathematically fair but also widely accepted.  

 This dissertation is structured into three parts. First, we study how ranking fairness metrics align with people’s subjective perceptions of fairness. 480 participants evaluated a scholarship distribution across 12 conditions, varying by group sensitivity, size, and performance scores. Results show a strong reliance on explicit score values, with group sensitivity influencing perceptions, particularly in unbalanced groups. Fairness perceptions were more consistent when candidates had similar grades, regardless of group sensitivity or size. Second, using JND, the minimum detectable difference between stimuli, and 2AFC methods, we address the limitations of Likert-scale studies in assessing fairness perceptions. With 224 participants and 170,000 judgments, we find that longer lists lead to more consistent and precise fairness judgments, revealing cases where fairness metrics diverge from human intuition. Third, a thematic analysis of 6,000 crowdsourced responses revealed concerns about merit-based discontent and transparency in socio-economic adjustments. Some participants relied on cognitive strategies and visual cues to assess fairness. These findings highlight the challenges of aligning fairness metrics with people’s intuition and the need for fairness-aware ranking methods that balance quantitative metrics with qualitative perceptions.