FL-bench is a repository dedicated to benchmarking federated learning (FL) techniques. It offers a standardized platform for comparing the performance of various FL algorithms across different datasets and scenarios. The project aims to accelerate research progress by providing a consistent and reproducible environment for evaluating advancements in distributed machine learning. Currently, it includes a comprehensive collection of both standard and personalized FL methods.
FL-bench distinguishes itself through its extensive collection of both traditional and personalized FL algorithms, catering to a wide range of research interests. It focuses on providing clear, reproducible experiments, emphasizing modularity and ease of extension for incorporating new models and datasets. The project structure facilitates easy integration with existing FL frameworks like PyTorch and Ray. The benchmark emphasizes reproducibility and clear reporting practices.
- Algorithm Implementations: Includes implementations of various core FL algorithms like FedAvg, FedProx, and personalized variants.
- Dataset Support: Supports multiple datasets, enabling fair comparisons across different data distribution scenarios.
- Performance Metrics: Provides a variety of standard metrics like accuracy, convergence speed, and communication cost.
- Scalability Testing: Facilitates experiments on different numbers of clients, assessing the scalability of algorithms.
- Configuration Flexibility: Highly configurable to experiment with various hyperparameters, communication protocols, and system settings.
FL-bench is an actively maintained project with a growing collection of benchmark methods and datasets. It has a reasonable release history, with updates being added based on the latest research. The code is well-documented, and the community engagement is positive, with contributions from researchers and developers. The project has been presented in various conferences and workshops, indicating its relevance and utility within the FL research community.
FL-bench benefits researchers and developers by offering a unified platform for evaluating FL algorithms. It simplifies the comparison process, reduces development effort, and promotes reproducibility. Researchers can quickly assess the effectiveness of new FL methods, while practitioners can benchmark and select the most suitable algorithms for their specific use cases. By providing a standardized testing environment, FL-bench fosters the advancement of federated learning techniques in healthcare, privacy-preserving AI, and other fields.