Commit Graph

25 Commits

Author SHA1 Message Date
let5sne.win10
b97d3f3a9f Improve Video Learning Agent with action-based detection and analysis toggle
- Change video detection from screenshot hash to action-based (Swipe detection)
- Add enable_analysis toggle to disable VLM screenshot analysis
- Improve task prompt to prevent VLM from stopping prematurely
- Add debug logging for action detection troubleshooting
- Fix ModelResponse attribute error (content -> raw_content)

Co-Authored-By: Claude <noreply@anthropic.com>
2026-01-10 01:47:09 +08:00
let5sne.win10
6b770832aa Skip app startup screens with warmup logic
- Added warmup counter to skip first 3 steps after entering app
- Reset counter when leaving target app
- Prevents recording splash screens as videos

Co-Authored-By: Claude <noreply@anthropic.com>
2026-01-09 23:56:21 +08:00
let5sne.win10
a823c03788 Improve screenshot analysis prompt and add debug logs
- Simplified prompt to force JSON-only response
- Added debug logs to track VLM response and parsing
- Better error messages for troubleshooting

Co-Authored-By: Claude <noreply@anthropic.com>
2026-01-09 23:37:35 +08:00
let5sne.win10
c4325d57d4 Fix record_video method to accept analysis fields
Added missing parameters: shares, tags, category, elements
Now analysis results are properly saved to VideoRecord

Co-Authored-By: Claude <noreply@anthropic.com>
2026-01-09 23:23:57 +08:00
let5sne.win10
195a93b7e0 Add screenshot content analysis using VLM
Features:
- ScreenshotAnalyzer class for VLM-based image analysis
- Real-time analysis during video recording
- Extract likes, comments, tags, category from screenshots
- Frontend display for category badges and tags
- Batch analysis API endpoint

Co-Authored-By: Claude <noreply@anthropic.com>
2026-01-09 23:20:52 +08:00
let5sne.win10
5b3f214e20 Add Video Learning Agent for short video platforms
Features:
- VideoLearningAgent for automated video watching on Douyin/Kuaishou/TikTok
- Web dashboard UI for video learning sessions
- Real-time progress tracking with screenshot capture
- App detection using get_current_app() for accurate recording
- Session management with pause/resume/stop controls

Technical improvements:
- Simplified video detection logic using direct app detection
- Full base64 hash for sensitive screenshot change detection
- Immediate stop when target video count is reached
- Fixed circular import issues with ModelConfig

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-09 22:54:57 +08:00
let5sne.win10
3552df23d6 Add Web Dashboard with multi-device control and callback hooks
Features:
- Web Dashboard: FastAPI-based dashboard with Vue.js frontend
  - Multi-device support (ADB, HDC, iOS)
  - Real-time WebSocket updates for task progress
  - Device management with status tracking
  - Task queue with execution controls (start/stop/re-execute)
  - Detailed task information display (thinking, actions, completion messages)
  - Screenshot viewing per device
  - LAN deployment support with configurable CORS

- Callback Hooks: Interrupt and modify task execution
  - step_callback: Called after each step with StepResult
  - before_action_callback: Called before executing action
  - Support for task interruption and dynamic task switching
  - Example scripts demonstrating callback usage

- Configuration: Environment-based configuration
  - .env file support for all settings
  - .env.example template with documentation
  - Model API configuration (base URL, model name, API key)
  - Dashboard configuration (host, port, CORS, device type)
  - Phone agent configuration (delays, max steps, language)

Technical improvements:
- Fixed forward reference issue with StepResult
- Added package exports for callback types and configs
- Enhanced dependencies with FastAPI, WebSocket support
- Thread-safe task execution with device locking
- Async WebSocket broadcasting from sync thread pool

Co-Authored-By: Claude <noreply@anthropic.com>
2026-01-09 02:20:06 +08:00
floatingstarZ
75f0e26ae4 修复hdc的get_current_app函数 2025-12-22 10:29:57 +08:00
liuyongbin
780b756e21 fix format 2025-12-19 18:56:43 +08:00
yongbin-buaa
7729568ae0 Merge pull request #143 from gekowa/ios-support-3
feat: Added iOS support
2025-12-19 18:15:50 +08:00
floatingstarZ
9bbf112dda 优化HDC文本输入:支持多行文本和简化接口
- 在hdc/input.py中实现多行文本支持,使用HarmonyOS keyEvent 2054处理换行
- 移除type_text函数的x/y坐标参数,简化接口
- 将多行文本处理逻辑从handler.py移至hdc/input.py,统一处理
- 优化parse_action函数,支持Type动作的text参数提取

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-16 19:54:59 +08:00
floatingstarZ
c0573c097f fix the entry ability of apps 2025-12-16 16:38:09 +08:00
floatingstarZ
95f5921887 支持鸿蒙OSNEXT_HDC 2025-12-16 14:28:49 +08:00
floatingstarZ
4d427bcd31 解决issue154中的model_output有换行的问题以及多行文本输出。解决subprocessing编码问题 2025-12-15 18:10:46 +08:00
xiaoman-kb
172d3e8e51 Fix Didi error 2025-12-15 15:17:53 +08:00
liuyongbin
430c13d22d support delay config 2025-12-15 11:56:48 +08:00
liuyongbin
b1ddd98552 fix format 2025-12-14 14:03:41 +08:00
yongbin-buaa
8c10bf7e99 replace eval with ast 2025-12-13 00:57:22 +08:00
yongbin-buaa
0653d6ea65 support stream thinking 2025-12-13 00:41:40 +08:00
xhguo
7c23ca549b feat: Added iOS support 2025-12-12 17:58:20 +08:00
liuyongbin
2a77c511ff update action parser 2025-12-11 15:07:42 +08:00
liuyongbin
2a5ee7a2d2 fix: code style and typo fix
- Fix typo: throught -> thought in prompts_en.py
- Apply ruff-format to main.py, prompts_en.py, prompts_zh.py

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-12-10 18:23:35 +08:00
zRzRzRzRzRzRzR
d4fbb4649c update 2025-12-09 23:30:39 +08:00
zRzRzRzRzRzRzR
a5675f380c format using pre-commit 2025-12-08 23:55:00 +08:00
zRzRzRzRzRzRzR
7e1785e08e draft init 2025-12-08 23:54:29 +08:00