An agent that enables multimodal models to operate a computer by viewing the screen and deciding on a series of mouse and keyboard actions to reach an objective, using the same inputs and outputs as a human operator.
Enabling multimodal models to operate a computer (full computer-use), integrated with models such as GPT-4o, GPT-4.1, o1, Gemini Pro Vision, Claude 3, Qwen-VL and LLaVa.
This is a computer-use framework where a multimodal model views the screen and autonomously decides on a series of mouse and keyboard actions to reach an objective. The README explicitly describes "full computer-use" with the model deciding on "a series of mouse and keyboard actions" — an autonomous multi-step action loop with no per-step confirmation. The user provides an objective once at the start, then the agent executes. Code surface confirms it executes shell commands and writes files. No code-enforced approval gate exists per action.