self-operating-computer
A framework to enable a multimodal model to operate a computer.
What is self-operating-computer?
A framework to enable a multimodal model to operate a computer.
Overview of self-operating-computer
What it is
Self-Operating Computer Framework is a framework that enables multimodal models to operate a computer. It uses the same inputs and outputs as a human operator: the model views the screen and decides on mouse and keyboard actions to achieve an objective. Released in November 2023, it was one of the first examples of full computer-use.
Key features
- Compatibility with various multimodal models.
- Integrated with GPT-4o, GPT-4.1, o1, Gemini Pro Vision, Claude 3, Qwen-VL, and LLaVa.
- Future plans to support additional models.
- Supports voice mode for objective input.
- Optical Character Recognition (OCR) mode for clickable element detection.
- Set-of-Mark (SoM) prompting mode for enhanced visual grounding.
Use cases
Automating computer tasks by allowing multimodal AI models to control the mouse and keyboard based on screen observations. Suitable for testing and experimenting with different models' ability to perform computer operations.
Getting started
Install the project:
pip install self-operating-computer
Run the project:
operate
Enter your OpenAI key when prompted. Grant Terminal app permissions for Screen Recording and Accessibility on macOS.
License and pricing
License: MIT. Pricing: open source (free).
Based on the project's public README and repository metadata.