Home Call For Papers Organization KeyNote Accepted Papers Schedule

Multiple Attribute Fairness: Application to Fraud Detection

Meghanath MY, Deepak Pai, Anish Narang, Vijay Srivastava PDF
Abstract: We propose a fairness measure relaxing the equality conditions in the popular equal odds fairness regime for classification. We design an iterative, model-agnostic, grid-based heuristic that calibrates the outcomes per sensitive attribute value to conform to the measure. The heuristic is designed to handle high arity attribute values and performs a per attribute sanitization of outcomes across different protected attribute values. We also extend our heuristic for multiple attributes. Highlighting our motivating application, fraud detection, we show that the proposed heuristic is able to achieve fairness across multiple values of a single protected attribute, multiple protected attributes. When compared to current fairness techniques, that focus on two groups, we achieve comparable performance across several public data sets.

Democratizing Ethical Assessment of Natural Language Generation Models

Amin Rasekh, Ian Eisenberg
Abstract: Natural language generation models are computer systems that generate coherent language when prompted with a sequence of words as context. Despite their ubiquity and many beneficial applications, language generation models also have the potential to inflict social harms by generating discriminatory language, hateful speech, profane content, and other harmful material. Ethical assessment of these models is therefore critical. But it is also a challenging task, requiring a expertise in several specialized domains, such as computational linguistics and social justice. While significant strides have been made by the research community in this domain, accessibility of such ethical assessments to the wider population is limited due to the high entry barriers. This article introduces a new tool to democratize and standardize ethical assessment of natural language generation models: Tool for Ethical Assessment of Language generation models (TEAL), a component of Credo AI Lens, an open-source assessment framework.

Consensus-determinacy Space and Moral Components for Ethical Dilemmas

Yongxu Liu, Yan Liu, Gong Chen, Yuexian Hou, Sheng-hua Zhong PDF
Abstract: This paper models ethical dilemmas, which take place in a decision-making context where any of the available options requires the agent to violate or compromise on their ethical standards. A famous dilemma, the "Trolley Problem," has been studied quantitatively and systematically since it represents machine ethics and has many applications in autonomous vehicles. We design a psychological paradigm to collect the data on the decisions and confidence of the participants facing the dilemmas, and a consensus-determinacy space is defined for human ethics. Then, we formulate the moral principle analysis as the matrix factorization problem, and a new model is proposed to discover moral components. Based on the embedded moral principles, we explore the possibility of providing decisions that are more consistent with the behavior of human beings, even if the available dataset is small and incomplete. Several experiments have been conducted on the proposed model to discuss the necessity and feasibility of the research in machine ethics.

Fair Collective Classification in Networked Data

Karuna Bhaila, Yongkai Wu, Xintao Wu PDF
Abstract: Collective classification utilizes network structure information via label propagation to improve prediction accuracy for node classification tasks. Because these models use information from labeled nodes which often contain historical bias, they may result in predictions that are biased w.r.t. the sensitive attributes of nodes. Throughout inference, this bias may even be amplified due to propagation. Despite past and ongoing research on fair classification, research to ensure fair collective classification still remains unexplored. In this paper, we present a fair collective classification (FairCC) framework and formulate various methodologies based on reweighting, threshold adjustment, and postprocessing to achieve fair prediction. Experiments on semi-synthetic datasets demonstrate the effectiveness of the proposed heuristics in significantly reducing prediction bias.

Information Theoretic Framework For Evaluation of Task Level Fairness

Surbhi Rathore, Sarah M Brown PDF
Abstract: The social impact of any algorithm deployed in the world, is limited by the broader context of the abstract goal, or task, it aims to solve. In the case of a prediction problem, this task guides the collection of training data and selection of the target. Problem formulation includes these steps and includes important formalization of the goal. This process is both a technical and normative process. We propose an information theoretic framework for examining problem formulations in terms of fairness through a reflective and heuristic process. We evaluate the framework as a practical tool by applying it to established benchmark in fair machine learning. We find that the heuristics we have developed reliably rank problem formulations by the severity of bias in resultant models. These results unify prior results on limitations of fairness interventions and motivate new classes of interventions for algorithmic fairness.

Stress-testing Fairness Mitigation Techniques under Distribution Shift using Synthetic Data

Karan Bhanot, Ioana Baldini, Dennis Wei, Jiaming Zeng, Kristin Bennett PDF
Abstract: Machine learning (ML) models may suffer from biases against certain subgroups defined by protected attributes. ML fairness mitigation techniques aim to resolve these biases. But how well do these fairness methods work in practice? To date, their evaluation has largely been limited to a few overly-used datasets that do not consider distribution shifts. In this paper, we propose the design of an "auditor" that uses synthetic data generation to create a grid of scenarios with distribution shifts to stress-test these techniques. We provide an explanation of the design of this auditor along with fairness audits of the reweighing method for fairness mitigation, using synthetic versions of MIMIC-III and Adult Income datasets. The paper also highlights the importance and potential benefits of doing fairness auditing of algorithms.