self-operating-computer

A framework to enable a multimodal model to operate a computer.

What is self-operating-computer?

A framework to enable a multimodal model to operate a computer.

Overview of self-operating-computer

What it is

Self-Operating Computer Framework is a framework that enables multimodal models to operate a computer. It uses the same inputs and outputs as a human operator: the model views the screen and decides on mouse and keyboard actions to achieve an objective. Released in November 2023, it was one of the first examples of full computer-use.

Key features

  • Compatibility with various multimodal models.
  • Integrated with GPT-4o, GPT-4.1, o1, Gemini Pro Vision, Claude 3, Qwen-VL, and LLaVa.
  • Future plans to support additional models.
  • Supports voice mode for objective input.
  • Optical Character Recognition (OCR) mode for clickable element detection.
  • Set-of-Mark (SoM) prompting mode for enhanced visual grounding.

Use cases

Automating computer tasks by allowing multimodal AI models to control the mouse and keyboard based on screen observations. Suitable for testing and experimenting with different models' ability to perform computer operations.

Getting started

Install the project:

pip install self-operating-computer

Run the project:

operate

Enter your OpenAI key when prompted. Grant Terminal app permissions for Screen Recording and Accessibility on macOS.

License and pricing

License: MIT. Pricing: open source (free).

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/othersideaiself-operating-computer.svg)](https://yunodb.com/tools/othersideaiself-operating-computer?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/othersideaiself-operating-computer?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/othersideaiself-operating-computer.svg" alt="Featured on YunoDB" height="20"></a>