Mobile Computer-Using Agent

droid-cua

Minimal AI agent that controls Android devices using OpenAI’s computer-use-preview model.

---

🚀 How It Works

1. Connects to a running Android emulator.

2. Captures full-screen device screenshots.

3. Scales down the screenshots for OpenAI model compatibility.

4. Sends screenshots and user instructions to OpenAI’s computer-use-preview model.

5. Receives structured actions (click, scroll, type, keypress, wait, drag).

6. Rescales model outputs back to real device coordinates.

7. Executes the actions on the device.

8. Repeats until you type exit.

Demo Video

---

🛠 Setup

First, clone this project - link

1. Install dependencies:

   npm install

2. Create a .env file with your OpenAI API key:

   echo "OPENAI_API_KEY=your-api-key" > .env

3. Make sure Android Debug Bridge (ADB) is available in your system PATH:

   adb version

4. Start your Android emulator manually (optional):

   emulator -avd Your_AVD_Name

5. Run the agent:

   node index.js --avd=Your_AVD_Name

   If no --avd is provided, the agent will try to connect to the first running device.

---

🧠 Features

• Captures screenshots directly from the device (adb exec-out screencap -p).

• Dynamically scales screenshots for OpenAI compatibility.

• Maps model-generated actions (click, scroll, drag, type, keypress, wait) back to real device coordinates.

• Connects automatically to a running emulator or launches it if needed.

• Pretends the device screen is embedded inside a browser page for environment compatibility.

---

📄 Command Line Flags

Flag	Description
--avd=AVD_NAME	Select the emulator device by AVD name.
--instructions=FILENAME	Load user instructions from a text file.
--record	Save every screenshot into a folder for later review or video creation.

---

📋 Example Usage

Start your emulator:

emulator -avd Pixel_5_API_34

Run the agent:

node index.js --avd=Pixel_5_API_34

Run with an instructions file:

node index.js --avd=Pixel_5_API_34 --instructions=example.txt

---

✅ Assertions

Assertions allow you to validate the state of the device during script execution. If an assertion fails, the script stops and reports the failure.

Syntax

Add assertions to your script using the assert keyword (case-insensitive, colon optional):

assert: <condition to validate>

Example Script

Open Chrome
Open a new tab
Open the Google built-in Calculator by searching "calculator"
Type "5"
Click the plus button
Type "3"
Click the equals button
assert: the result shows 8
exit

---

📦 Requirements

• Node.js 18 or higher

• A running Android emulator (AVD)

• Android Debug Bridge (ADB) installed and available in system PATH

• OpenAI Tier 3 access for the computer-use-preview model

Your OpenAI account must be Tier 3 to access the computer-use-preview model. Learn more: OpenAI Computer Use Preview.

If you do not have access to a Tier 3 OpenAI API key, please reach out to your Loadmill contact.

---

🎞️ Convert Screenshots to Video (Optional)

If you run the agent with the --record flag, it saves all screenshots to a folder like:

droid-cua-recording-1715098765432/

You can convert the frames into a video using ffmpeg:

ffmpeg -framerate 1 -pattern_type glob -i 'droid-cua-recording-*/frame_*.png' \
  -vf "pad=ceil(iw/2)*2:ceil(ih/2)*2" \
  -c:v libx264 -pix_fmt yuv420p session.mp4