NIID-Bench is a benchmark designed to evaluate federated learning (FL) algorithms under non-independent and identically distributed (non-IID) data conditions. We implement and compare four FL algorithms (FedAvg, FedProx, SCAFFOLD, and FedNova) across three types of non-IID settings: label distribution skew, feature distribution skew, and quantity skew. The benchmark supports various datasets, including MNIST, CIFAR-10, and generated datasets, providing a comprehensive platform for research in FL.
NIID-Bench distinguishes itself by offering a wide range of non-IID settings, enabling researchers to thoroughly test algorithm robustness. It includes diverse datasets and flexible partitioning strategies catering to various research needs. The project's clear documentation and easy-to-use implementation make it accessible to a broad audience. The integration of MOON and additional datasets enhances the benchmark's applicability and scope.
- Algorithm Support: Implements FedAvg, FedProx, SCAFFOLD, and FedNova FL algorithms.
- Non-IID Settings: Offers label distribution skew, feature distribution skew, and quantity skew.
- Dataset Variety: Supports MNIST, CIFAR-10, Fashion-MNIST, SVHN, and custom datasets.
- Flexible Partitioning: Provides various partitioning strategies including homo, non-IID label distribution, and others.
- Extensible Architecture: Supports easy integration with new FL algorithms and datasets.
- User-Friendly Interface: Simple command-line interface for easy experiment execution.
- Comprehensive Documentation: Clear instructions for setting up and running experiments.
NIID-Bench is an active project with ongoing development and maintenance. Recent commits indicate continuous improvements and bug fixes. The project benefits from a growing community and has been cited in recent research papers. Regular updates and test data integration demonstrates commitment to reliability. The data partition map is well-defined and actively maintained.
Researchers, developers, and practitioners benefit from NIID-Bench by providing a standardized platform to evaluate and compare FL algorithms in realistic non-IID scenarios. It enables the development of more robust and effective FL systems for real-world applications where data is inherently non-IID. It offers a valuable tool for understanding and mitigating the challenges posed by data heterogeneity in federated learning.