Unfairness in Computer Vision Models

Computer vision models have known performance disparities across attributes such as gender and skin tone. This means during tasks such as classification and detection, model performance differs for certain classes based on the demographics of the people in the image. FACET (FAirness in Computer Vision EvaluaTion) is a large, publicly available evaluation set of 32k images for some of the most common vision tasks - image classification, object detection and segmentation. For every image in FACET, expert reviewers manually annotated person-related attributes such as perceived skin tone and hair type, manually draw bounding boxes and label fine-grained person-related classes such as disk jockey or guitarist. FACET is used to benchmark state-of-the-art vision models and present a deeper understanding of potential performance disparities and challenges across sensitive demographic attributes. The authors probe models using single demographics attributes as well as multiple attributes using an intersectional approach (e.g. hair color and perceived skin tone). Results show that classification, detection, segmentation, and visual grounding models exhibit performance disparities across demographic attributes and intersections of attributes. These harms suggest that not all people represented in datasets receive fair and equitable treatment in these vision tasks. FACET is available publicly at https://facet.metademolab.com.

In this video, I will talk briefly about the following: Are computer vision models biased? FACET dataset curation and statistics. Bias in classification, person detection, segmentation, Open Vocabulary Detection and Visual grounding.

For more details, please look at https://ai.meta.com/research/publications/facet-fairness-in-computer-vision-evaluation-benchmark/

Laura Gustafson, Chloe Rolland, Nikhila Ravi, Quentin Duval, Aaron Adcock, Cheng-Yang Fu, Melissa Hall, Candace Ross. FACET: Fairness in Computer Vision Evaluation Benchmark. Meta AI Research, FAIR. 2023

Видео Unfairness in Computer Vision Models канала Data Science Gems