This repository provides a PyTorch implementation of the World Models algorithm, as described in the 2018 paper by Ha and Schmidhuber. The project focuses on learning a predictive model of the environment, encompassing a Variational Auto-Encoder (VAE), a Mixture-Density Recurrent Network (MDN-RNN), and a Controller. It addresses the challenge of sample efficiency in reinforcement learning by enabling the agent to learn a compact internal representation of the world.
The implementation highlights a modular architecture for the VAE, MDN-RNN, and Controller, allowing for independent training and experimentation. It utilizes CMA-ES for controller optimization, demonstrating an effective approach to policy learning. The inclusion of a data generation script facilitates customizable training datasets, enabling flexibility in environment simulation. The code provides essential tools for model training, testing, and evaluation.
- VAE: Learns a compressed latent representation of the environment's visual inputs.
- MDN-RNN: Predicts the latent encoding of the next frame based on past states and actions.
- Controller: A linear actor trained using CMA-ES to maximize cumulative reward.
- Data Generation: Includes a script for generating random rollouts for training.
- Modular Training: Separate scripts for training VAE, MDN-RNN, and Controller.
- CMA-ES Optimization: Utilizes Covariance Matrix Adaptation Evolution Strategy for controller training.
- Extensible Architecture: Designed to be easily adapted to different environments and tasks.
The project is complete and functional, with clear instructions for setup, training, and evaluation. The repository includes necessary scripts for data generation, model training, and policy testing. Active development appears to be minimal, suggesting a stable and well-tested implementation.
This project benefits researchers and practitioners interested in model-based reinforcement learning. It provides a readily available and well-documented implementation of World Models, enabling experiments and further development in this field. By learning internal world models, it addresses sample efficiency challenges and facilitates improved planning and decision-making in complex environments.