Skip to content

Secure AI computer use powered by E2B Desktop Sandbox

License

Notifications You must be signed in to change notification settings

e2b-dev/open-computer-use

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

91 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Open Computer Use

A secure cloud Linux computer powered by E2B Desktop Sandbox and controlled by open-source LLMs.

Desktop.Use.+.Streaming.mp4

Features

  • Uses E2B for secure Desktop Sandbox
  • Operates the computer via the keyboard, mouse, and shell commands
  • Supports 10+ LLMs, OS-Atlas/ShowUI and any other models you want to integrate!
  • Live streams the display of the sandbox on the client computer
  • User can pause and prompt the agent at any time
  • Uses Ubuntu, but designed to work with any operating system

Design

Open Computer Use Architecture Open Computer Use Architecture

The details of the design are laid out in this article: How I taught an AI to use a computer

LLM support

Open Computer Use is designed to make it easy to swap in and out new LLMs. The LLMs used by the agent are specified in config.py like this:

grounding_model = providers.OSAtlasProvider()
vision_model = providers.GroqProvider("llama3.2")
action_model = providers.GroqProvider("llama3.3")

The providers are imported from providers.py and include:

  • Fireworks, OpenRouter, Llama API:
    • Llama 3.2 (vision only), Llama 3.3 (action only)
  • Groq:
    • Llama 3.2 (vision + action), Llama 3.3 (action only)
  • DeepSeek:
    • DeepSeek (action only)
  • Google:
    • Gemini 2.0 Flash (vision + action)
  • OpenAI:
    • GPT-4o and GPT-4o mini (vision + action)
  • Anthropic:
    • Claude (vision + action)
  • HuggingFace Spaces:
    • OS-Atlas (grounding)
    • ShowUI (grounding)

If you add a new model or provider, please make a PR to this repository with the updated providers.py!

Get started

Prerequisites

  • Python 3.10 or later
  • git
  • E2B API key
  • API key for an LLM provider (see above)

1. Install the prerequisites

In your terminal:

brew install poetry ffmpeg

2. Clone the repository

In your terminal:

git clone https://github.com/e2b-dev/open-computer-use/

3. Set the environment variables

Enter the project directory:

cd open-computer-use

Create a .env file in open-computer-use and set the following:

# Get your API key here: https://e2b.dev/
E2B_API_KEY="your-e2b-api-key"

Additionally, add API key(s) for any LLM providers you're using:

# You only need the API key for the provider(s) selected in config.py:
# Hugging Face Spaces do not require an API key.
FIREWORKS_API_KEY=...
OPENROUTER_API_KEY=...
LLAMA_API_KEY=...
GROQ_API_KEY=...
GEMINI_API_KEY=...
OPENAI_API_KEY=...
ANTHROPIC_API_KEY=...

4. Start the web interface

Run the following command to start the agent:

poetry install
poetry run start

The agent will open and prompt you for its first instruction.

To start the agent with a specified prompt, run:

poetry run start --prompt "use the web browser to get the current weather in sf"

The display stream should be visible a few seconds after the Python program starts.