Speaker: Ivan Stelmakh

Abstract: We will consider a problem of testing for fairness of human decisions in application to conference peer review. Specifically, we are interested in testing for biases in single-blind setup, where reviewers observe identities of authors. We will show that various idiosyncrasies of peer review (including non-random assignment, noise and miscalibration of reviewers) do not allow to perform fully randomized controlled trials and may undermine Type-I error guarantees of popular parametric approaches to testing. To overcome this negative result, we will consider a general framework for performing such tests and show that a simple non-parametric procedure leads to a more reliable test, requiring weaker assumptions to control for the Type-I error and having non-trivial power. This is a joint work with Nihar and Aarti.

Draft: http://www.cs.cmu.edu/afs/cs.cmu.edu/user/istelmak/www/papers/bias.pdf