diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 7d8a47c..a010a05 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -2,6 +2,7 @@ default_install_hook_types: - pre-commit - commit-msg exclude: '^phone_agent/config/apps\.py$' +exclude: '^README_en\.md$' default_stages: - pre-commit # Run locally repos: diff --git a/README.md b/README.md index f4e17df..7e3cc62 100644 --- a/README.md +++ b/README.md @@ -46,7 +46,11 @@ ADB 调试能力，可通过 WiFi 或网络连接设备，实现灵活的远程 建议使用 Python 3.10 及以上版本。 -### 2. ADB (Android Debug Bridge) +### 2. 手机调试命令行工具 + +根据你的设备类型选择相应的工具： + +#### 对于 Android 设备 - 使用 ADB 1. 下载官方 ADB [安装包](https://developer.android.com/tools/releases/platform-tools?hl=zh-cn)，并解压到自定义路径 2. 配置环境变量 @@ -60,7 +64,22 @@ ADB 调试能力，可通过 WiFi 或网络连接设备，实现灵活的远程 - Windows 配置方法：可参考 [第三方教程](https://blog.csdn.net/x2584179909/article/details/108319973) 进行配置。 -### 3. Android 7.0+ 的设备或模拟器，并启用 `开发者模式` 和 `USB 调试` +#### 对于鸿蒙设备 (HarmonyOS NEXT版本以上) - 使用 HDC + +1. 下载 HDC 工具： + - 从 [HarmonyOS SDK](https://developer.huawei.com/consumer/cn/download/) 下载 +2. 配置环境变量 + +- MacOS/Linux 配置方法： + + ```bash + # 假设解压后的目录为 ~/Downloads/harmonyos-sdk/toolchains。请根据实际路径调整。 + export PATH=${PATH}:~/Downloads/harmonyos-sdk/toolchains + ``` + +- Windows 配置方法：将 HDC 工具所在目录添加到系统 PATH 环境变量 + +### 3. Android 7.0+ 或 HarmonyOS 设备，并启用 `开发者模式` 和 `USB 调试` 1. 开发者模式启用：通常启用方法是，找到 `设置-关于手机-版本号` 然后连续快速点击 10 次左右，直到弹出弹窗显示“开发者模式已启用”。不同手机会有些许差别，如果找不到，可以上网搜索一下教程。 @@ -72,7 +91,11 @@ ADB 调试能力，可通过 WiFi 或网络连接设备，实现灵活的远程 ![权限](resources/screenshot-20251209-181423.png) -### 4. 安装 ADB Keyboard(用于文本输入) +### 4. 安装 ADB Keyboard(仅 Android 设备需要，用于文本输入) + +**注意：鸿蒙设备使用原生输入方法，无需安装 ADB Keyboard。** + +如果你使用的是 Android 设备： 下载 [安装包](https://github.com/senzhk/ADBKeyBoard/blob/master/ADBKeyboard.apk) 并在对应的安卓设备中进行安装。 注意，安装完成后还需要到 `设置-输入法` 或者 `设置-键盘列表` 中启用 `ADB Keyboard` 才能生效(或使用命令`adb shell ime enable com.android.adbkeyboard/.AdbIME`[How-to-use](https://github.com/senzhk/ADBKeyBoard/blob/master/README.md#how-to-use)) @@ -86,7 +109,9 @@ pip install -r requirements.txt pip install -e . ``` -### 2. 配置 ADB +### 2. 配置 ADB 或 HDC + +#### 对于 Android 设备 确认 **USB数据线具有数据传输功能**, 而不是仅有充电功能 @@ -101,6 +126,20 @@ adb devices # emulator-5554 device ``` +#### 对于鸿蒙设备 + +确认 **USB数据线具有数据传输功能**, 而不是仅有充电功能 + +确保已安装 HDC 并使用 **USB数据线** 连接设备： + +```bash +# 检查已连接的设备 +hdc list targets + +# 输出结果应显示你的设备，如： +# 7001005458323933328a01bce01c2500 +``` + ### 3. 启动模型服务 你可以选择自行部署模型服务，或使用第三方模型服务商。 @@ -234,23 +273,32 @@ python scripts/check_deployment_cn.py --base-url http://你的IP:你的端口/v1 ### 命令行 -根据你部署的模型, 设置 `--base-url` 和 `--model` 参数. 例如: +根据你部署的模型, 设置 `--base-url` 和 `--model` 参数, 设置 `--device-type` 指定是安卓设备或鸿蒙设备 (默认值 adb 表示安卓设备, hdc 表示鸿蒙设备). 例如: ```bash -# 交互模式 +# Android 设备 - 交互模式 python main.py --base-url http://localhost:8000/v1 --model "autoglm-phone-9b" -# 指定模型端点 +# Android 设备 - 指定任务 python main.py --base-url http://localhost:8000/v1 "打开美团搜索附近的火锅店" +# 鸿蒙设备 - 交互模式 +python main.py --device-type hdc --base-url http://localhost:8000/v1 --model "autoglm-phone-9b" + +# 鸿蒙设备 - 指定任务 +python main.py --device-type hdc --base-url http://localhost:8000/v1 "打开美团搜索附近的火锅店" + # 使用 API Key 进行认证 python main.py --apikey sk-xxxxx # 使用英文 system prompt python main.py --lang en --base-url http://localhost:8000/v1 "Open Chrome browser" -# 列出支持的应用 +# 列出支持的应用（Android） python main.py --list-apps + +# 列出支持的应用（鸿蒙） +python main.py --device-type hdc --list-apps ``` ### Python API @@ -275,30 +323,47 @@ print(result) ## 远程调试 -Phone Agent 支持通过 WiFi/网络进行远程 ADB 调试，无需 USB 连接即可控制设备。 +Phone Agent 支持通过 WiFi/网络进行远程 ADB/HDC 调试，无需 USB 连接即可控制设备。 ### 配置远程调试 #### 在手机端开启无线调试 +##### Android 设备 + 确保手机和电脑在同一个WiFi中，如图所示 ![开启无线调试](resources/setting.png) -#### 在电脑端使用标准 ADB 命令 +##### 鸿蒙设备 + +确保手机和电脑在同一个WiFi中： +1. 进入 `设置 > 系统和更新 > 开发者选项` +2. 开启 `USB 调试` 和 `无线调试` +3. 记录显示的 IP 地址和端口号 + +#### 在电脑端使用标准 ADB/HDC 命令 ```bash - -# 通过 WiFi 连接, 改成手机显示的 IP 地址和端口 +# Android 设备 - 通过 WiFi 连接, 改成手机显示的 IP 地址和端口 adb connect 192.168.1.100:5555 # 验证连接 adb devices # 应显示：192.168.1.100:5555 device + +# 鸿蒙设备 - 通过 WiFi 连接 +hdc tconn 192.168.1.100:5555 + +# 验证连接 +hdc list targets +# 应显示：192.168.1.100:5555 ``` ### 设备管理命令 +#### Android 设备（ADB） + ```bash # 列出所有已连接设备 adb devices @@ -313,8 +378,26 @@ adb disconnect 192.168.1.100:5555 python main.py --device-id 192.168.1.100:5555 --base-url http://localhost:8000/v1 --model "autoglm-phone-9b" "打开抖音刷视频" ``` +#### 鸿蒙设备（HDC） + +```bash +# 列出所有已连接设备 +hdc list targets + +# 连接远程设备 +hdc tconn 192.168.1.100:5555 + +# 断开指定设备 +hdc tdisconn 192.168.1.100:5555 + +# 指定设备执行任务 +python main.py --device-type hdc --device-id 192.168.1.100:5555 --base-url http://localhost:8000/v1 --model "autoglm-phone-9b" "打开抖音刷视频" +``` + ### Python API 远程连接 +#### Android 设备（ADB） + ```python from phone_agent.adb import ADBConnection, list_devices @@ -339,6 +422,27 @@ print(f"设备 IP: {ip}") conn.disconnect("192.168.1.100:5555") ``` +#### 鸿蒙设备（HDC） + +```python +from phone_agent.hdc import HDCConnection, list_devices + +# 创建连接管理器 +conn = HDCConnection() + +# 连接远程设备 +success, message = conn.connect("192.168.1.100:5555") +print(f"连接状态: {message}") + +# 列出已连接设备 +devices = list_devices() +for device in devices: + print(f"{device.device_id} - {device.connection_type.value}") + +# 断开连接 +conn.disconnect("192.168.1.100:5555") +``` + ### 远程连接问题排查 **连接被拒绝：** @@ -370,14 +474,15 @@ conn.disconnect("192.168.1.100:5555") ### 环境变量 -| 变量 | 描述 | 默认值 | -|-------------------------|------------------|----------------------------| -| `PHONE_AGENT_BASE_URL` | 模型 API 地址 | `http://localhost:8000/v1` | -| `PHONE_AGENT_MODEL` | 模型名称 | `autoglm-phone-9b` | -| `PHONE_AGENT_API_KEY` | 模型认证 API Key | `EMPTY` | -| `PHONE_AGENT_MAX_STEPS` | 每个任务最大步数 | `100` | -| `PHONE_AGENT_DEVICE_ID` | ADB 设备 ID | (自动检测) | -| `PHONE_AGENT_LANG` | 语言 (`cn` 或 `en`) | `cn` | +| 变量 | 描述 | 默认值 | +|-----------------------------|------------------------|----------------------------| +| `PHONE_AGENT_BASE_URL` | 模型 API 地址 | `http://localhost:8000/v1` | +| `PHONE_AGENT_MODEL` | 模型名称 | `autoglm-phone-9b` | +| `PHONE_AGENT_API_KEY` | 模型认证 API Key | `EMPTY` | +| `PHONE_AGENT_MAX_STEPS` | 每个任务最大步数 | `100` | +| `PHONE_AGENT_DEVICE_ID` | ADB/HDC 设备 ID | (自动检测) | +| `PHONE_AGENT_DEVICE_TYPE` | 设备类型 (`adb` 或 `hdc`) | `adb` | +| `PHONE_AGENT_LANG` | 语言 (`cn` 或 `en`) | `cn` | ### 模型配置 @@ -449,6 +554,8 @@ config = AgentConfig( ## 支持的应用 +### Android 应用 + Phone Agent 支持 50+ 款主流中文应用： | 分类 | 应用 | @@ -464,6 +571,25 @@ Phone Agent 支持 50+ 款主流中文应用： 运行 `python main.py --list-apps` 查看完整列表。 +### 鸿蒙应用 + +Phone Agent 支持 60+ 款鸿蒙原生应用和系统应用： + +| 分类 | 应用 | +|---------|------------------------------------------| +| 社交通讯 | 微信、QQ、微博、飞书、企业微信 | +| 电商购物 | 淘宝、京东、拼多多、唯品会、得物、闲鱼 | +| 美食外卖 | 美团、美团外卖、大众点评、海底捞 | +| 出行旅游 | 12306、滴滴出行、同程旅行、高德地图、百度地图 | +| 视频娱乐 | bilibili、抖音、快手、腾讯视频、爱奇艺、芒果TV | +| 音乐音频 | QQ音乐、汽水音乐、喜马拉雅 | +| 生活服务 | 小红书、知乎、今日头条、58同城、中国移动 | +| AI与工具 | 豆包、WPS、UC浏览器、扫描全能王、美图秀秀 | +| 系统应用 | 浏览器、日历、相机、时钟、云空间、文件管理器、相册、联系人、短信、设置等 | +| 华为服务 | 应用市场、音乐、视频、阅读、主题、天气 | + +运行 `python main.py --device-type hdc --list-apps` 查看完整列表。 + ## 可用操作 Agent 可以执行以下操作： diff --git a/README_en.md b/README_en.md index e2e7820..0b3af12 100644 --- a/README_en.md +++ b/README_en.md @@ -39,7 +39,11 @@ Phone Agent is a mobile intelligent assistant framework built on AutoGLM. It und Python 3.10 or higher is recommended. -### 2. ADB (Android Debug Bridge) +### 2. Device Debug Tools + +Choose the appropriate tool based on your device type: + +#### For Android Devices - Using ADB 1. Download the official ADB [installation package](https://developer.android.com/tools/releases/platform-tools) and extract it to a custom path 2. Configure environment variables @@ -53,7 +57,22 @@ Python 3.10 or higher is recommended. - Windows configuration: Refer to [third-party tutorials](https://blog.csdn.net/x2584179909/article/details/108319973) for configuration. -### 3. Android 7.0+ Device or Emulator with `Developer Mode` and `USB Debugging` Enabled +#### For HarmonyOS Devices - Using HDC + +1. Download HDC tool: + - From [HarmonyOS SDK](https://developer.huawei.com/consumer/en/download/) +2. Configure environment variables + +- MacOS/Linux configuration: + + ```bash + # Assuming the extracted directory is ~/Downloads/harmonyos-sdk/toolchains. Adjust according to actual path. + export PATH=${PATH}:~/Downloads/harmonyos-sdk/toolchains + ``` + +- Windows configuration: Add the HDC tool directory to the system PATH environment variable + +### 3. Android 7.0+ or HarmonyOS Device with `Developer Mode` and `USB Debugging` Enabled 1. Enable Developer Mode: The typical method is to find `Settings > About Phone > Build Number` and tap it rapidly about 10 times until a popup shows "Developer mode has been enabled." This may vary slightly between phones; search online for tutorials if you can't find it. 2. Enable USB Debugging: After enabling Developer Mode, go to `Settings > Developer Options > USB Debugging` and enable it @@ -63,7 +82,11 @@ Python 3.10 or higher is recommended. ![Permissions](resources/screenshot-20251210-120416.png) -### 4. Install ADB Keyboard (for Text Input) +### 4. Install ADB Keyboard (Required for Android Devices Only, for Text Input) + +**Note: HarmonyOS devices use native input methods and do not require ADB Keyboard.** + +If you are using an Android device: Download the [installation package](https://github.com/senzhk/ADBKeyBoard/blob/master/ADBKeyboard.apk) and install it on the corresponding Android device. Note: After installation, you need to enable `ADB Keyboard` in `Settings > Input Method` or `Settings > Keyboard List` for it to work.(or use command `adb shell ime enable com.android.adbkeyboard/.AdbIME`[How-to-use](https://github.com/senzhk/ADBKeyBoard/blob/master/README.md#how-to-use)) @@ -77,7 +100,9 @@ pip install -r requirements.txt pip install -e . ``` -### 2. Configure ADB +### 2. Configure ADB or HDC + +#### For Android Devices Make sure your **USB cable supports data transfer**, not just charging. @@ -92,6 +117,20 @@ adb devices # emulator-5554 device ``` +#### For HarmonyOS Devices + +Make sure your **USB cable supports data transfer**, not just charging. + +Ensure HDC is installed and connect the device via **USB cable**: + +```bash +# Check connected devices +hdc list targets + +# Output should show your device, e.g.: +# 7001005458323933328a01bce01c2500 +``` + ### 3. Start Model Service You can choose to deploy the model service yourself or use a third-party model service provider. @@ -188,20 +227,29 @@ Upon successful execution, the script will display the model's inference result Set the `--base-url` and `--model` parameters according to your deployed model. For example: ```bash -# Interactive mode +# Android device - Interactive mode python main.py --base-url http://localhost:8000/v1 --model "autoglm-phone-9b-multilingual" -# Specify model endpoint +# Android device - Specify task python main.py --base-url http://localhost:8000/v1 "Open Maps and search for nearby coffee shops" +# HarmonyOS device - Interactive mode +python main.py --device-type hdc --base-url http://localhost:8000/v1 --model "autoglm-phone-9b-multilingual" + +# HarmonyOS device - Specify task +python main.py --device-type hdc --base-url http://localhost:8000/v1 "Open Maps and search for nearby coffee shops" + # Use API key for authentication python main.py --apikey sk-xxxxx # Use English system prompt python main.py --lang en --base-url http://localhost:8000/v1 "Open Chrome browser" -# List supported apps +# List supported apps (Android) python main.py --list-apps + +# List supported apps (HarmonyOS) +python main.py --device-type hdc --list-apps ``` ### Python API @@ -226,29 +274,47 @@ print(result) ## Remote Debugging -Phone Agent supports remote ADB debugging via WiFi/network, allowing device control without a USB connection. +Phone Agent supports remote ADB/HDC debugging via WiFi/network, allowing device control without a USB connection. ### Configure Remote Debugging #### Enable Wireless Debugging on Phone +##### Android Devices + Ensure the phone and computer are on the same WiFi network, as shown below: ![Enable Wireless Debugging](resources/screenshot-20251210-120630.png) -#### Use Standard ADB Commands on Computer +##### HarmonyOS Devices + +Ensure the phone and computer are on the same WiFi network: +1. Go to `Settings > System & Updates > Developer Options` +2. Enable `USB Debugging` and `Wireless Debugging` +3. Note the displayed IP address and port number + +#### Use Standard ADB/HDC Commands on Computer ```bash -# Connect via WiFi, replace with the IP address and port shown on your phone +# Android device - Connect via WiFi, replace with the IP address and port shown on your phone adb connect 192.168.1.100:5555 # Verify connection adb devices # Should show: 192.168.1.100:5555 device + +# HarmonyOS device - Connect via WiFi +hdc tconn 192.168.1.100:5555 + +# Verify connection +hdc list targets +# Should show: 192.168.1.100:5555 ``` ### Device Management Commands +#### Android Devices (ADB) + ```bash # List all connected devices adb devices @@ -263,8 +329,26 @@ adb disconnect 192.168.1.100:5555 python main.py --device-id 192.168.1.100:5555 --base-url http://localhost:8000/v1 --model "autoglm-phone-9b-multilingual" "Open TikTok and browse videos" ``` +#### HarmonyOS Devices (HDC) + +```bash +# List all connected devices +hdc list targets + +# Connect to remote device +hdc tconn 192.168.1.100:5555 + +# Disconnect specific device +hdc tdisconn 192.168.1.100:5555 + +# Execute task on specific device +python main.py --device-type hdc --device-id 192.168.1.100:5555 --base-url http://localhost:8000/v1 --model "autoglm-phone-9b-multilingual" "Open TikTok and browse videos" +``` + ### Python API Remote Connection +#### Android Devices (ADB) + ```python from phone_agent.adb import ADBConnection, list_devices @@ -289,6 +373,27 @@ print(f"Device IP: {ip}") conn.disconnect("192.168.1.100:5555") ``` +#### HarmonyOS Devices (HDC) + +```python +from phone_agent.hdc import HDCConnection, list_devices + +# Create connection manager +conn = HDCConnection() + +# Connect to remote device +success, message = conn.connect("192.168.1.100:5555") +print(f"Connection status: {message}") + +# List connected devices +devices = list_devices() +for device in devices: + print(f"{device.device_id} - {device.connection_type.value}") + +# Disconnect +conn.disconnect("192.168.1.100:5555") +``` + ### Remote Connection Troubleshooting **Connection Refused:** @@ -320,14 +425,15 @@ You can directly modify the corresponding config files to enhance model capabili ### Environment Variables -| Variable | Description | Default Value | -|---------------------------|---------------------------|----------------------------| -| `PHONE_AGENT_BASE_URL` | Model API URL | `http://localhost:8000/v1` | -| `PHONE_AGENT_MODEL` | Model name | `autoglm-phone-9b` | -| `PHONE_AGENT_API_KEY` | API key for authentication| `EMPTY` | -| `PHONE_AGENT_MAX_STEPS` | Maximum steps per task | `100` | -| `PHONE_AGENT_DEVICE_ID` | ADB device ID | (auto-detect) | -| `PHONE_AGENT_LANG` | Language (`cn` or `en`) | `en` | +| Variable | Description | Default Value | +|-----------------------------|---------------------------|----------------------------| +| `PHONE_AGENT_BASE_URL` | Model API URL | `http://localhost:8000/v1` | +| `PHONE_AGENT_MODEL` | Model name | `autoglm-phone-9b` | +| `PHONE_AGENT_API_KEY` | API key for authentication| `EMPTY` | +| `PHONE_AGENT_MAX_STEPS` | Maximum steps per task | `100` | +| `PHONE_AGENT_DEVICE_ID` | ADB/HDC device ID | (auto-detect) | +| `PHONE_AGENT_DEVICE_TYPE` | Device type (`adb` or `hdc`)| `adb` | +| `PHONE_AGENT_LANG` | Language (`cn` or `en`) | `en` | ### Model Configuration @@ -399,6 +505,8 @@ This allows you to clearly see the AI's reasoning process and specific operation ## Supported Apps +### Android Apps + Phone Agent supports 50+ mainstream Chinese applications: | Category | Apps | @@ -411,6 +519,25 @@ Phone Agent supports 50+ mainstream Chinese applications: Run `python main.py --list-apps` to see the complete list. +### HarmonyOS Apps + +Phone Agent supports 60+ HarmonyOS native apps and system apps: + +| Category | Apps | +|--------------------------|----------------------------------------------------------------------------------------| +| Social & Messaging | WeChat, QQ, Weibo, Feishu, Enterprise WeChat | +| E-commerce & Shopping | Taobao, JD.com, Pinduoduo, Vipshop, Dewu, Xianyu | +| Food & Delivery | Meituan, Meituan Waimai, Dianping, Haidilao | +| Travel & Navigation | 12306, Didi, Tongcheng, Amap, Baidu Maps | +| Video & Entertainment | Bilibili, Douyin, Kuaishou, Tencent Video, iQIYI, Mango TV | +| Music & Audio | QQ Music, Qishui Music, Ximalaya | +| Lifestyle & Social | Xiaohongshu, Zhihu, Toutiao, 58.com, China Mobile | +| AI & Tools | Doubao, WPS, UC Browser, CamScanner, Meitu | +| System Apps | Browser, Calendar, Camera, Clock, Cloud, File Manager, Gallery, Contacts, SMS, Settings | +| Huawei Services | AppGallery, Music, Video, Books, Themes, Weather | + +Run `python main.py --device-type hdc --list-apps` to see the complete list. + ## Available Actions The Agent can perform the following actions: