FeDa4Fair: Client-Level Federated Datasets for Fairness Evaluation

Federated Learning (FL) enables collaborative training while preserving privacy, yet it introduces a critical challenge: the “illusion of fairness”. A global model, usually evaluated on the server, appears fair on average while keeping persistent unfairness at the client level. Current fairness-enhancing FL solutions often fall short, as they typically mitigate biases for a single, usually binary, sensitive attribute, while ignoring two realistic and conflicting scenarios: Attribute-Bias (where clients are unfair toward different sensitive attributes) and Value-Bias (where clients exhibit conflicting biases toward different values of the same attribute). To support more robust and reproducible fairness research in FL, we introduce FeDa4Fair, the first benchmarking framework designed to stress-test FL fairness methods under these heterogeneous conditions. Our contributions are three-fold: (1) We introduce FeDa4Fair, a library designed to create datasets tailored to evaluating fair FL methods under heterogeneous client bias; (2) we release a benchmark suite generated by the FeDa4Fair library to standardize the evaluation of fair FL methods; (3) we provide ready-to-use functions for evaluating fairness outcomes for these datasets.