Computer Science PhD Proposal Defense , Hilson Shrestha " Fairness Reasoning in Practice: Designing and Evaluating Systems Supporting Human-in-the-Loop Consensus Ranking "

Hilson Shrestha

PhD Candidate

WPI – Computer Science Department

Friday, July 31, 2026

Time: 11:00 a.m. – 12:00 p.m.

Location:  Fuller Labs 311

Zoom Link: https://wpi.zoom.us/my/hilsonshrestha?omn=93794973879

 Committee members :

Prof. Lane Harrison, Advisor - WPI - Computer Science Department 

Prof. Elke Rundensteiner, WPI - Computer Science Department 

Prof. Erin Solovey,  WPI - Computer Science Department 

Prof. Alexander Lex, University of Utah, (External Advisor)

Abstract:

Algorithmic rankings increasingly shape critical decisions from admissions to hiring, raising concerns about fairness and transparency. While visualization enabled human-in-the-loop systems provide more interpretable and trustworthy fair consensus rankings, they only work if human participants genuinely reason through the decision making process rather than passively deferring to algorithmic output. Prior work offers limited empirical guidance on whether users meaningfully participate or simply "rubber-stamp" algorithm suggested rankings.

This work makes three main contributions. First, it develops interactive visualization systems that support the construction and exploration of fair consensus rankings. Second, it conducts a crowdsourced user study to investigate whether visual encodings of fairness metrics improve users' understanding of fairness trade-offs and lead to fairer consensus outcomes. Third, it proposes an in-depth study combining screen recording and think-aloud protocols to examine how users interpret fairness information, interact with visualization interfaces, and make ranking decisions in practice. Overall, this work shifts the focus from asking whether human-in-the-loop systems produce fair outcomes to asking whether humans meaningfully participate in producing those outcomes.