This repository provides the datasets and code for benchmarking neural network robustness to common image corruptions and perturbations, as detailed in the ICLR 2019 paper. It focuses on evaluating the resilience of deep learning models to real-world image degradations. The project includes ImageNet-C and ImageNet-P datasets used to assess model performance under various attacks and noise conditions, particularly for models like ResNet-50. The primary technology employed is the application of various image augmentation techniques to simulate real-world corruption.
The project offers a comprehensive evaluation of robustness against a diverse set of image corruptions, including blur, noise, and geometric transformations. It is particularly notable for its standardized methodology and readily available datasets. The leaderboard facilitates community contributions and advances in robustness research. The inclusion of ImageNet-P expands the scope to real-world perturbations commonly encountered in non-synthetic data.
- Data Sets: Includes ImageNet-C (with varying corruption levels) and ImageNet-P (with real-world and synthetic perturbations).
- Evaluation Metrics: Provides mean Corruption Error (mCE) and mean mT5D for robust classification metrics.
- Leaderboard: Features a leaderboard to track the state-of-the-art performance on the ImageNet-C and ImageNet-P benchmarks.
- Code: Includes code for calculating mCE and mT5D, enabling easy evaluation of models.
- Perturbations: Covers a wide range of perturbations including blur, noise, color distortion, and geometric transformations.
- Usability: Offers clear instructions and links for downloading datasets and accessing relevant resources.
- Community Friendly: Encourages contribution by inviting PRs for state-of-the-art results.
The project is well-established with a published paper and readily accessible datasets. The quality and availability of training and evaluation components are solid. An active leaderboard allows the community to contribute advancements and maintain the relevance of the benchmarks. The documentation, though concise, effectively guides users through the methodology and evaluation procedures.
This repository is valuable for researchers and practitioners seeking to assess and improve the robustness of their neural network models. It offers a standardized framework and comprehensive datasets for evaluating performance under various corruption scenarios. It is particularly beneficial for those working on model safety, domain generalization, and real-world deployment of deep learning systems.