Representation Alignment for Generation (REPA) proposes aligning noisy input states in diffusion models with representations extracted from pretrained visual encoders. This alignment strategy demonstrably improves training efficiency, achieving a 17.5x speedup in SiT training while also improving image generation quality, as evidenced by a state-of-the-art FID of 1.42. The core idea is to reduce the discrepancy between the noisy states and the encoded representations, making training more stable and faster.
REPA offers a simplified approach to training diffusion models by leveraging pretrained visual encoders, reducing the need for large-scale training datasets. The method is effective across various ViT encoder architectures (DINOv2, MocoV3, JPEPA, MAE) and can be applied to both ImageNet and text-to-image generation tasks. Furthermore, the changes to the training process are relatively minor, making it easy to integrate into existing diffusion model training pipelines.
- Core Functionality: Aligns noisy input states in diffusion models with pretrained visual encoder representations to improve training efficiency and generation quality.
- Supported Platforms: Compatible with various ViT encoder architectures including DINOv2, MocoV3, JPEPA, and MAE.
- Configuration: Offers flexible configuration options for model size, encoder type, projection coefficient, and encoder depth.
- Performance: Achieves significant speedups in training (up to 17.5x) and state-of-the-art FID scores.
- Developer Experience: Provides clear instructions for environment setup, data preparation, and training, with pre-trained checkpoints available.
The project is actively developed and demonstrated through successful experiments on ImageNet and text-to-image generation. The code is relatively new, and occasional inconsistencies may arise during reproduction. Regular updates and improvements are expected based on community feedback and ongoing research.
REPA benefits researchers and practitioners working with diffusion models by offering a straightforward way to accelerate training and improve generation quality. This method is especially valuable for those with limited computational resources or who want to quickly evaluate different diffusion model architectures. It reduces training time and resource requirements without sacrificing generation fidelity, making it a valuable tool for creative and scientific applications.