GPT-4V-Act enables GPT-4V(ision) to interact with web-based user interfaces by mimicking human input through mouse and keyboard actions. The project addresses the difficulty of automating dynamic UIs, offering a solution for workflow automation, accessibility improvements, and UI testing. It uses a novel auto-labeler combined with set-of-marks prompting to achieve this.
This project uniquely combines visual understanding with direct UI interaction, bridging the gap between AI reasoning and practical application in web environments. The auto-labeling system simplifies the process of identifying interactive elements, and the JSON output format allows for easy integration with other systems. The project also demonstrates a practical application of set-of-marks prompting for precise control.
- DOM Auto-labeler: Generates a JavaScript DOM auto-labeler with COCO export for identifying interactive elements. Supports both JS DOM auto-labeler and AI auto-labeler labeling (currently the former is implemented).
- Clicking: Enables precise mouse clicks on UI elements identified by the auto-labeler. Provides accurate pixel coordinates for reliable interaction.
- Typing: Allows for typing of text strings into UI elements, including letters, numbers, and strings. Partially supports typing of special keycodes.
- JSON Output: Consistently outputs actions in a structured JSON format, facilitating easy integration with other systems and workflows.
- Demo & Instructions: Provides a clear demonstration prompt and usage instructions for testing and experimentation.
- Accessibility Focus: Aims to improve UI accessibility by automating interactions and reducing the need for manual input.
- JavaScript Implementation: Built in JavaScript, allowing for easy integration into existing web applications and systems.
The project is actively developed with recent commits indicating ongoing maintenance and improvement. Documentation is present via the README, and the actively maintained GitHub repository suggests community involvement. The project is functional, though certain features are marked as incomplete.
GPT-4V-Act benefits developers and users looking to automate web tasks, improve UI accessibility, or create automated UI testing frameworks. It provides a powerful way to leverage GPT-4V(ision) to interact with dynamic web UIs, offering a valuable alternative to traditional UI automation tools and manual processes.