Computer-use agent
A computer-use agent looks at screenshots and operates a computer like a person, by clicking, typing and scrolling, and sometimes by running shell commands or code.
Instead of calling a fixed set of APIs, the model receives a screenshot, chooses the next action, and a harness carries it out and returns a new screenshot. Microsoft's 4B Mage-VL-Cowork shows one design: it chooses between graphical actions and a persistent shell with Python, and clicks only when it needs to see something. Microsoft reports 66.0% on OSWorld-Verified with a 50-step budget, which is its own measurement.
Such an agent takes real actions, so run it in a sandbox or virtual machine that has no access to accounts or files you care about. The model alone clicks nothing, because you also need the harness and the screen it controls. Scores depend on the step budget and the harness, so compare models only on the same setup.
Related terms
- AI agent An AI agent is a system in which a model plans and carries out multi-step tasks by calling tools, such as running code, searching the web or editing files.
- Tool calling (function calling) Tool calling lets a model ask your program to run a function, such as a search or a calculator, and then use the result in its answer.
- Vision-language model (VLM) A vision-language model reads images, and often video, together with text, so you can ask questions about a picture, a screenshot or a document page.
- Agentic coding Agentic coding means a model works on software like a developer: it reads the codebase, edits files, runs commands and tests, and keeps going until the task is done.