AgentCPM-GUI is an open-source on-device LLM agent built upon MiniCPM-V. It is designed to operate Android applications by receiving smartphone screenshots as input and autonomously executing user-defined tasks. The project addresses the challenge of creating efficient and robust agents for interacting with diverse mobile interfaces, combining GUI grounding with advanced reasoning capabilities via Reinforcement Fine-Tuning (RFT). The primary focus is on enabling users to control apps without manual intervention.
AgentCPM-GUI offers high-quality GUI grounding through pre-training on a large bilingual dataset, enabling operation of Chinese and English apps. Its RFT-enhanced reasoning significantly improves task success on complex scenarios. The compact action-space design minimizes token length, enhancing on-device inference efficiency. The project provides clear documentation, example usage, and inference methods via both Hugging Face Inference and vLLM.
- GUI Grounding: Pre-trained on a large bilingual Android dataset for robust widget understanding.
- Chinese App Support: Specifically fine-tuned for 30+ popular Chinese apps.
- Reinforcement Learning: Employs RFT for improved planning and complex task execution.
- Efficient Inference: Compact action space minimizes token length for faster on-device processing.
- Flexible Deployment: Supports both Hugging Face Inference and vLLM for flexible deployment options.
- Detailed Documentation: Includes quick start guides, schema definitions, and usage examples.
- Extensible Schema: JSON schema allows definition of actions and constraints.
The project is actively developed, with recent releases including a technical report and the initial open-source release. The GitHub repository demonstrates ongoing maintenance with regular commits and engagement through issue tracking. The project maintains a comprehensive schema definition, supporting standardized actions and ensuring predictable behavior. The active community and clear documentation suggest sustained development and reliability.
AgentCPM-GUI benefits developers seeking to integrate intelligent agents into Android applications or users desiring automated app control. It addresses use cases requiring automation of tasks within mobile apps, increasing efficiency and streamlining workflows. It provides a powerful alternative to manual app interaction or simpler rule-based automation systems, offering advanced reasoning capabilities and improved accuracy.