初始化 AutoAgent 代码与文档
This commit is contained in:
@@ -0,0 +1,8 @@
|
||||
.gradle/
|
||||
.idea/
|
||||
local.properties
|
||||
**/build/
|
||||
*.iml
|
||||
captures/
|
||||
.externalNativeBuild/
|
||||
.cxx/
|
||||
@@ -0,0 +1,191 @@
|
||||
# AutoAgent 开发规则
|
||||
|
||||
本仓库采用需求驱动的轻量开发流程。当前项目是内部系统,不设置额外的敏感数据规则、安全审批门禁、工单门禁、原型门禁或 Wiki 同步门禁。
|
||||
|
||||
## 1. 基本原则
|
||||
|
||||
- 以用户当前确认的需求为实施依据。
|
||||
- 收到需求后,先分析目标、现状、范围、约束和验收方式。
|
||||
- 存在多种实现路径或关键事实不明确时,先与用户讨论方案,不直接猜测。
|
||||
- 方案确认后按确认方案实施;需求或方案发生变化时,先重新确认再继续。
|
||||
- 只修改当前需求涉及的内容,不主动扩展无关功能或重构。
|
||||
- 保留与当前需求无关的工作区改动,不重置、不覆盖。
|
||||
- 测试结果必须真实;未执行或无法覆盖的验证必须明确说明。
|
||||
|
||||
## 2. 标准流程
|
||||
|
||||
1. **理解需求**:整理目标、输入、输出、范围、非目标和验收标准。
|
||||
2. **检查现状**:读取相关代码、配置和文档,确认当前实现,不凭假设修改。
|
||||
3. **分析方案**:说明模块边界、数据流、接口、持久化、异常处理和测试方式。
|
||||
4. **讨论确认**:方案存在重要取舍、接口不明确或可能改变既有行为时,等待用户确认。
|
||||
5. **实施改动**:按照确认方案进行最小且完整的实现。
|
||||
6. **执行验证**:运行受影响范围的单元测试、构建或其他必要检查。
|
||||
7. **交付说明**:列出改动文件、实现结果、验证结果和未完成或未验证事项。
|
||||
|
||||
对于范围清楚、实现方式唯一、容易回退的小改动,可以在完成分析后直接实施,不必重复请求确认。
|
||||
|
||||
## 3. 项目标识
|
||||
|
||||
- Gradle 项目名:`AutoAgent`
|
||||
- Android 包名 / Application ID:`cn.auto.agent`
|
||||
- Kotlin namespace:`cn.auto.agent`
|
||||
- 应用显示名:`Auto Agent`
|
||||
- 无障碍服务显示名:`autoagent`
|
||||
|
||||
未经用户明确要求,不修改上述标识。
|
||||
|
||||
## 4. 系统、Java 与依赖版本
|
||||
|
||||
### 开发环境基线
|
||||
|
||||
以下版本已在当前开发机实际核验:
|
||||
|
||||
| 项目 | 版本 / 路径 |
|
||||
|---|---|
|
||||
| 操作系统 | Windows 10,版本 `10.0.19045.3803`,amd64 |
|
||||
| PowerShell 7 | `7.3.12`,有条件时优先使用 |
|
||||
| Windows PowerShell | `5.1.19041.3803` |
|
||||
| Java / JDK | Eclipse Temurin OpenJDK `17.0.13+11`,64 位 |
|
||||
| Git | `2.49.0.windows.1` |
|
||||
| Android SDK | `C:\Users\ila20\AppData\Local\Android\Sdk` |
|
||||
|
||||
开发和构建使用 Java 17;除非依赖升级明确要求,不自行切换 Java 主版本。Android SDK 的本机绝对路径只描述当前开发环境,不写入可提交的 `local.properties` 或业务代码。
|
||||
|
||||
### 构建工具版本
|
||||
|
||||
| 项目 | 版本 |
|
||||
|---|---|
|
||||
| Gradle Wrapper | `8.2` |
|
||||
| Android Gradle Plugin | `8.2.0` |
|
||||
| Kotlin Android Plugin | `1.9.22` |
|
||||
| Java source compatibility | `17` |
|
||||
| Java target compatibility | `17` |
|
||||
| Kotlin JVM target | `17` |
|
||||
|
||||
`gradlew --version` 显示的 Kotlin `1.8.20` 是 Gradle 自身内嵌的 Kotlin 版本;项目 Kotlin 源码编译版本以 `build.gradle.kts` 声明的 Kotlin Android Plugin `1.9.22` 为准。
|
||||
|
||||
### Android SDK 与应用版本
|
||||
|
||||
| 项目 | 当前值 |
|
||||
|---|---:|
|
||||
| `compileSdk` | `34` |
|
||||
| `targetSdk` | `34` |
|
||||
| `minSdk` | `23` |
|
||||
| `versionCode` | `1` |
|
||||
| `versionName` | `0.1.0` |
|
||||
|
||||
### 生产依赖
|
||||
|
||||
| 依赖 | 版本 | 用途 |
|
||||
|---|---:|---|
|
||||
| `androidx.appcompat:appcompat` | `1.7.0` | Android 兼容 UI 和 `AppCompatActivity` |
|
||||
| `com.google.android.material:material` | `1.12.0` | Material Components 主题和控件 |
|
||||
|
||||
### 测试依赖
|
||||
|
||||
| 依赖 | 版本 | 用途 |
|
||||
|---|---:|---|
|
||||
| `junit:junit` | `4.13.2` | JVM 单元测试 |
|
||||
| `org.json:json` | `20240303` | JVM 测试中的 JSON 实现 |
|
||||
|
||||
版本的可执行事实来源依次为 `gradle/wrapper/gradle-wrapper.properties`、根目录 `build.gradle.kts` 和 `app/build.gradle.kts`;本节用于开发约束和快速查阅。升级系统基线、Java、Gradle、AGP、Kotlin、Android SDK 级别或依赖时,必须先分析兼容性,按需求确认方案,然后同步修改构建文件、本节和受影响测试。不得只修改本文档来宣称升级完成。
|
||||
|
||||
## 5. 架构与解耦
|
||||
|
||||
- 通用任务模型、协议解析、任务同步、持久化、执行器和 Android 无障碍驱动保持分层。
|
||||
- 核心执行器依赖抽象接口,不直接依赖具体服务端或目标 App 的业务逻辑。
|
||||
- 外部服务端差异通过 `TaskGateway` 或后续等价适配器隔离。
|
||||
- Android 无障碍实现通过 `ActionDriver` 向执行器提供能力。
|
||||
- 目标 App 的包名、页面识别和业务步骤放入独立适配层,不写入通用协议和执行内核。
|
||||
- 不把 GoAuto 的 PDD、采集、采购、规格、地址或订单业务逻辑带入本项目,除非用户提出明确需求。
|
||||
- 优先进行满足当前需求的简单实现,不为未确认的未来需求预建复杂框架。
|
||||
|
||||
## 6. 当前任务规则
|
||||
|
||||
- Agent 从外部服务获取任务列表。
|
||||
- 任务使用 `assignedAgentName` 指定设备名。
|
||||
- 只有 `assignedAgentName` 与本机配置的 Agent 设备名精确一致时,任务才允许保存到 SQLite。
|
||||
- 设备名比较区分大小写,不使用包含、前缀、后缀或模糊匹配。
|
||||
- 不属于本设备的任务仅忽略,不写入 SQLite,也不作为本设备执行失败处理。
|
||||
- 格式无效的任务不写入 SQLite,并在同步结果中计入无效任务。
|
||||
- SQLite 当前以 `task_id + revision` 作为任务版本的幂等键。
|
||||
- 任务是否入库与任务是否开始执行是两个独立阶段;自动执行、领取、租约和结果提交按后续需求实现。
|
||||
|
||||
上述规则可由用户后续需求修改;修改时同步更新实现、测试和本文件中的长期规则。
|
||||
|
||||
## 7. 任务执行约定
|
||||
|
||||
- 任务步骤使用类型化动作,不通过字符串分支散落实现。
|
||||
- 新动作需要同步更新模型、协议解析、执行器和测试。
|
||||
- 控件选择器和执行结果使用通用模型。
|
||||
- 默认要求控件唯一;匹配多个控件时明确失败,不随意选择第一个。
|
||||
- 每个步骤必须具有有限超时,任务执行不得无限等待或无限循环。
|
||||
- 包名和已声明的 Activity 必须在操作前核对。
|
||||
- 业务特定的页面判断、重试和恢复策略由对应适配层实现。
|
||||
|
||||
当前动作:
|
||||
|
||||
- `WAIT`
|
||||
- `CLICK`
|
||||
- `INPUT`
|
||||
- `BACK`
|
||||
- `EXTRACT_TEXT`
|
||||
|
||||
当前未实现的动作或能力不能仅通过修改任务 JSON 宣称可用。
|
||||
|
||||
## 8. 代码修改规则
|
||||
|
||||
- 开始修改前读取相关文件;不要只依据文件名推断实现。
|
||||
- 优先修改已有抽象,避免复制相同逻辑。
|
||||
- 保持包职责清晰:
|
||||
- `core/model`:通用数据模型;
|
||||
- `core/protocol`:任务协议解析和校验;
|
||||
- `core/scheduler`:任务获取与同步;
|
||||
- `core/persistence`:SQLite 和本地状态;
|
||||
- `core/executor`:任务步骤执行;
|
||||
- `core/accessibility`:Android 无障碍能力;
|
||||
- `app`:界面、生命周期和依赖装配。
|
||||
- 数据库表结构、任务协议或接口发生变化时,同步更新相关测试和 README。
|
||||
- 不生成无必要的备份文件、临时脚本或重复文档。
|
||||
|
||||
## 9. Git 规则
|
||||
|
||||
- 提交前查看工作区状态,确认没有混入无关文件。
|
||||
- 一个提交只包含一个清晰的交付目标。
|
||||
- 提交信息简洁描述实际变化。
|
||||
- 未经用户要求,不自动推送、不发布 APK、不创建标签。
|
||||
- 不改写用户已有提交,不重置或强制清理工作区。
|
||||
|
||||
## 10. 验证
|
||||
|
||||
默认从仓库根目录执行:
|
||||
|
||||
```powershell
|
||||
.\gradlew.bat test
|
||||
.\gradlew.bat assembleDebug
|
||||
```
|
||||
|
||||
根据改动范围选择验证:
|
||||
|
||||
- 模型、协议、同步或执行逻辑:运行相关单元测试,交付前运行 `test`。
|
||||
- Android Manifest、资源、依赖或应用代码:运行 `assembleDebug`。
|
||||
- SQLite 结构变化:补充升级或读写测试,并验证旧版本升级路径(存在已发布旧版本时)。
|
||||
- 无障碍真实操作变化:自动化测试和构建之外,明确列出需要真机验证的目标 App、页面和步骤。
|
||||
|
||||
APK 默认输出:
|
||||
|
||||
```text
|
||||
app/build/outputs/apk/debug/app-debug.apk
|
||||
```
|
||||
|
||||
## 11. 完成条件
|
||||
|
||||
满足以下条件后停止当前任务:
|
||||
|
||||
- 已实现用户确认的需求;
|
||||
- 相关代码和测试保持一致;
|
||||
- 必要验证已经通过;
|
||||
- README 或本文件需要记录的长期事实已同步;
|
||||
- 已向用户说明改动、验证结果以及未验证事项。
|
||||
|
||||
发现与当前需求无关的问题时只简要提示,不自动混入本次实现。
|
||||
@@ -0,0 +1,33 @@
|
||||
# AutoAgent
|
||||
|
||||
独立的原生 Kotlin Android 任务 Agent。项目标识:
|
||||
|
||||
- 项目名:`AutoAgent`
|
||||
- 包名:`cn.auto.agent`
|
||||
- 应用名:`Auto Agent`
|
||||
- 无障碍服务名:`autoagent`
|
||||
|
||||
## 当前提取范围
|
||||
|
||||
- 类型化任务协议:`WAIT`、`CLICK`、`INPUT`、`BACK`、`EXTRACT_TEXT`
|
||||
- 基于 `AccessibilityService` 的包名、Activity 和唯一节点操作驱动
|
||||
- 外部服务适配接口 `TaskGateway`
|
||||
- 按 `assignedAgentName` 与本地设备名进行区分大小写的精确匹配
|
||||
- 不属于本设备和格式无效的任务不写入本地数据库
|
||||
- SQLite `task_inbox`,以 `task_id + revision` 幂等保存
|
||||
- 单进程任务互斥和可单元测试的执行器
|
||||
|
||||
## 重要边界
|
||||
|
||||
当前外部服务端接口尚未提供,因此项目只定义 `TaskGateway`,没有臆造 API 地址、认证和轮询协议。设备名过滤是当前需求的兼容机制;正式多设备调度仍建议增加不可变 `agentId`、服务端原子 claim、任务租约和幂等结果提交。
|
||||
|
||||
当前版本不会自动开始任务,也未实现后台轮询、结果 Outbox、任意脚本、Shell、坐标盲点、订单或支付动作。
|
||||
|
||||
## 验证
|
||||
|
||||
```powershell
|
||||
.\gradlew.bat test
|
||||
.\gradlew.bat assembleDebug
|
||||
```
|
||||
|
||||
APK:`app/build/outputs/apk/debug/app-debug.apk`
|
||||
@@ -0,0 +1,40 @@
|
||||
plugins {
|
||||
id("com.android.application")
|
||||
id("org.jetbrains.kotlin.android")
|
||||
}
|
||||
|
||||
android {
|
||||
namespace = "cn.auto.agent"
|
||||
compileSdk = 34
|
||||
|
||||
defaultConfig {
|
||||
applicationId = "cn.auto.agent"
|
||||
minSdk = 23
|
||||
targetSdk = 34
|
||||
versionCode = 1
|
||||
versionName = "0.1.0"
|
||||
testInstrumentationRunner = "androidx.test.runner.AndroidJUnitRunner"
|
||||
}
|
||||
|
||||
buildTypes {
|
||||
release {
|
||||
isMinifyEnabled = false
|
||||
proguardFiles(getDefaultProguardFile("proguard-android-optimize.txt"), "proguard-rules.pro")
|
||||
}
|
||||
}
|
||||
|
||||
compileOptions {
|
||||
sourceCompatibility = JavaVersion.VERSION_17
|
||||
targetCompatibility = JavaVersion.VERSION_17
|
||||
}
|
||||
|
||||
kotlinOptions { jvmTarget = "17" }
|
||||
}
|
||||
|
||||
dependencies {
|
||||
implementation("androidx.appcompat:appcompat:1.7.0")
|
||||
implementation("com.google.android.material:material:1.12.0")
|
||||
|
||||
testImplementation("junit:junit:4.13.2")
|
||||
testImplementation("org.json:json:20240303")
|
||||
}
|
||||
Vendored
+1
@@ -0,0 +1 @@
|
||||
# Add project-specific ProGuard rules here.
|
||||
@@ -0,0 +1,31 @@
|
||||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<manifest xmlns:android="http://schemas.android.com/apk/res/android">
|
||||
<uses-permission android:name="android.permission.INTERNET" />
|
||||
|
||||
<application
|
||||
android:allowBackup="false"
|
||||
android:label="@string/app_name"
|
||||
android:supportsRtl="true"
|
||||
android:theme="@style/Theme.AutoAgent">
|
||||
<activity
|
||||
android:name=".app.MainActivity"
|
||||
android:exported="true">
|
||||
<intent-filter>
|
||||
<action android:name="android.intent.action.MAIN" />
|
||||
<category android:name="android.intent.category.LAUNCHER" />
|
||||
</intent-filter>
|
||||
</activity>
|
||||
<service
|
||||
android:name=".core.accessibility.AutoAccessibilityService"
|
||||
android:exported="true"
|
||||
android:label="@string/accessibility_service_name"
|
||||
android:permission="android.permission.BIND_ACCESSIBILITY_SERVICE">
|
||||
<intent-filter>
|
||||
<action android:name="android.accessibilityservice.AccessibilityService" />
|
||||
</intent-filter>
|
||||
<meta-data
|
||||
android:name="android.accessibilityservice"
|
||||
android:resource="@xml/accessibility_service_config" />
|
||||
</service>
|
||||
</application>
|
||||
</manifest>
|
||||
@@ -0,0 +1,37 @@
|
||||
package cn.auto.agent.app
|
||||
|
||||
import android.content.Intent
|
||||
import android.os.Bundle
|
||||
import android.provider.Settings
|
||||
import android.widget.Button
|
||||
import android.widget.LinearLayout
|
||||
import android.widget.TextView
|
||||
import androidx.appcompat.app.AppCompatActivity
|
||||
import cn.auto.agent.R
|
||||
import cn.auto.agent.core.accessibility.AutoAccessibilityService
|
||||
|
||||
class MainActivity : AppCompatActivity() {
|
||||
private lateinit var state: TextView
|
||||
|
||||
override fun onCreate(savedInstanceState: Bundle?) {
|
||||
super.onCreate(savedInstanceState)
|
||||
title = getString(R.string.app_name)
|
||||
state = TextView(this).apply { textSize = 18f }
|
||||
val button = Button(this).apply {
|
||||
text = "打开无障碍设置"
|
||||
setOnClickListener { startActivity(Intent(Settings.ACTION_ACCESSIBILITY_SETTINGS)) }
|
||||
}
|
||||
setContentView(LinearLayout(this).apply {
|
||||
orientation = LinearLayout.VERTICAL
|
||||
val padding = (24 * resources.displayMetrics.density).toInt()
|
||||
setPadding(padding, padding, padding, padding)
|
||||
addView(state)
|
||||
addView(button)
|
||||
})
|
||||
}
|
||||
|
||||
override fun onResume() {
|
||||
super.onResume()
|
||||
state.text = if (AutoAccessibilityService.instance == null) "autoagent 未连接" else "autoagent 已就绪"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
package cn.auto.agent.core.accessibility
|
||||
|
||||
internal class ActivityTracker {
|
||||
private val activities = mutableMapOf<String, String>()
|
||||
|
||||
@Synchronized
|
||||
fun observe(packageName: String?, className: String?) {
|
||||
if (!packageName.isNullOrBlank() && !className.isNullOrBlank()) activities[packageName] = className
|
||||
}
|
||||
|
||||
@Synchronized
|
||||
fun current(packageName: String): String? = activities[packageName]
|
||||
}
|
||||
+73
@@ -0,0 +1,73 @@
|
||||
package cn.auto.agent.core.accessibility
|
||||
|
||||
import android.accessibilityservice.AccessibilityService
|
||||
import android.os.Bundle
|
||||
import android.view.accessibility.AccessibilityEvent
|
||||
import android.view.accessibility.AccessibilityNodeInfo
|
||||
import cn.auto.agent.core.executor.ActionDriver
|
||||
import cn.auto.agent.core.executor.UiNodeRef
|
||||
import cn.auto.agent.core.model.NodeSelector
|
||||
|
||||
class AutoAccessibilityService : AccessibilityService(), ActionDriver {
|
||||
private val activityTracker = ActivityTracker()
|
||||
|
||||
override fun onServiceConnected() { instance = this }
|
||||
|
||||
override fun onAccessibilityEvent(event: AccessibilityEvent?) {
|
||||
if (event?.eventType == AccessibilityEvent.TYPE_WINDOW_STATE_CHANGED) {
|
||||
activityTracker.observe(event.packageName?.toString(), event.className?.toString())
|
||||
}
|
||||
}
|
||||
|
||||
override fun onInterrupt() = Unit
|
||||
|
||||
override fun onDestroy() {
|
||||
if (instance === this) instance = null
|
||||
super.onDestroy()
|
||||
}
|
||||
|
||||
override fun currentPackage(): String? = rootInActiveWindow?.packageName?.toString()
|
||||
override fun currentActivity(): String? = currentPackage()?.let(activityTracker::current)
|
||||
|
||||
override fun find(selector: NodeSelector): List<UiNodeRef> {
|
||||
val root = rootInActiveWindow ?: return emptyList()
|
||||
val result = mutableListOf<UiNodeRef>()
|
||||
walk(root) { node ->
|
||||
if (node.matches(selector)) result += UiNodeRef(node, (node.text ?: node.contentDescription)?.toString())
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
override fun click(node: UiNodeRef): Boolean {
|
||||
var target = node.opaqueId as? AccessibilityNodeInfo ?: return false
|
||||
while (!target.isClickable) target = target.parent ?: return false
|
||||
return target.performAction(AccessibilityNodeInfo.ACTION_CLICK)
|
||||
}
|
||||
|
||||
override fun input(node: UiNodeRef, value: String): Boolean {
|
||||
val arguments = Bundle().apply {
|
||||
putCharSequence(AccessibilityNodeInfo.ACTION_ARGUMENT_SET_TEXT_CHARSEQUENCE, value)
|
||||
}
|
||||
return (node.opaqueId as? AccessibilityNodeInfo)
|
||||
?.performAction(AccessibilityNodeInfo.ACTION_SET_TEXT, arguments) == true
|
||||
}
|
||||
|
||||
override fun back(): Boolean = performGlobalAction(GLOBAL_ACTION_BACK)
|
||||
|
||||
private fun walk(node: AccessibilityNodeInfo, visit: (AccessibilityNodeInfo) -> Unit) {
|
||||
visit(node)
|
||||
for (index in 0 until node.childCount) node.getChild(index)?.let { walk(it, visit) }
|
||||
}
|
||||
|
||||
private fun AccessibilityNodeInfo.matches(selector: NodeSelector): Boolean =
|
||||
(selector.resourceId == null || viewIdResourceName == selector.resourceId) &&
|
||||
(selector.text == null || text?.toString() == selector.text) &&
|
||||
(selector.contentDescription == null || contentDescription?.toString() == selector.contentDescription) &&
|
||||
(selector.className == null || className?.toString() == selector.className) &&
|
||||
(selector.clickable == null || isClickable == selector.clickable)
|
||||
|
||||
companion object {
|
||||
@Volatile var instance: AutoAccessibilityService? = null
|
||||
private set
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,15 @@
|
||||
package cn.auto.agent.core.executor
|
||||
|
||||
import java.util.concurrent.atomic.AtomicReference
|
||||
|
||||
class TaskExecutionMutex {
|
||||
private val activeTaskId = AtomicReference<String?>(null)
|
||||
|
||||
fun tryAcquire(taskId: String): Boolean {
|
||||
require(taskId.isNotBlank()) { "taskId 不能为空" }
|
||||
return activeTaskId.compareAndSet(null, taskId) || activeTaskId.get() == taskId
|
||||
}
|
||||
|
||||
fun currentTaskId(): String? = activeTaskId.get()
|
||||
fun release(taskId: String): Boolean = activeTaskId.compareAndSet(taskId, null)
|
||||
}
|
||||
@@ -0,0 +1,70 @@
|
||||
package cn.auto.agent.core.executor
|
||||
|
||||
import cn.auto.agent.core.model.ActionType
|
||||
import cn.auto.agent.core.model.AgentTask
|
||||
import cn.auto.agent.core.model.NodeSelector
|
||||
import cn.auto.agent.core.model.TaskExecutionResult
|
||||
|
||||
data class UiNodeRef(val opaqueId: Any, val text: String?)
|
||||
|
||||
interface ActionDriver {
|
||||
fun currentPackage(): String?
|
||||
fun currentActivity(): String?
|
||||
fun find(selector: NodeSelector): List<UiNodeRef>
|
||||
fun click(node: UiNodeRef): Boolean
|
||||
fun input(node: UiNodeRef, value: String): Boolean
|
||||
fun back(): Boolean
|
||||
}
|
||||
|
||||
class TaskExecutor(
|
||||
private val driver: ActionDriver,
|
||||
private val now: () -> Long = System::currentTimeMillis,
|
||||
private val pause: (Long) -> Unit = Thread::sleep,
|
||||
) {
|
||||
fun execute(task: AgentTask): TaskExecutionResult {
|
||||
val output = linkedMapOf<String, MutableList<String>>()
|
||||
task.steps.forEach { step ->
|
||||
val deadline = now() + step.timeoutMs
|
||||
var lastFailure = "控件未出现"
|
||||
while (now() <= deadline) {
|
||||
if (driver.currentPackage() != step.packageName) {
|
||||
lastFailure = "当前应用与步骤不匹配"
|
||||
} else if (step.activityName != null && driver.currentActivity() != step.activityName) {
|
||||
lastFailure = "当前 Activity 与步骤不匹配"
|
||||
} else if (step.action == ActionType.BACK) {
|
||||
if (driver.back()) break
|
||||
lastFailure = "返回操作失败"
|
||||
} else {
|
||||
val matches = driver.find(requireNotNull(step.selector))
|
||||
when {
|
||||
matches.size > 1 -> return failure(step.id, "TASK_AMBIGUOUS", "控件匹配到多个节点", output)
|
||||
matches.isEmpty() -> lastFailure = "控件未出现"
|
||||
else -> {
|
||||
val node = matches.single()
|
||||
val successful = when (step.action) {
|
||||
ActionType.WAIT -> true
|
||||
ActionType.CLICK -> driver.click(node)
|
||||
ActionType.INPUT -> driver.input(node, requireNotNull(step.value))
|
||||
ActionType.EXTRACT_TEXT -> node.text?.trim()?.takeIf(String::isNotEmpty)?.also {
|
||||
output.getOrPut(requireNotNull(step.outputField)) { mutableListOf() }.add(it)
|
||||
} != null
|
||||
ActionType.BACK -> error("handled above")
|
||||
}
|
||||
if (successful) break
|
||||
lastFailure = "操作执行失败"
|
||||
}
|
||||
}
|
||||
}
|
||||
if (now() >= deadline) {
|
||||
if (step.optional) break
|
||||
return failure(step.id, "TASK_NOT_MATCHED", lastFailure, output)
|
||||
}
|
||||
pause(minOf(100, (deadline - now()).coerceAtLeast(1)))
|
||||
}
|
||||
}
|
||||
return TaskExecutionResult(true, "OK", "任务执行完成", output.mapValues { it.value.toList() })
|
||||
}
|
||||
|
||||
private fun failure(id: String, code: String, detail: String, output: Map<String, List<String>>) =
|
||||
TaskExecutionResult(false, code, "步骤 $id 失败:$detail", output)
|
||||
}
|
||||
@@ -0,0 +1,42 @@
|
||||
package cn.auto.agent.core.model
|
||||
|
||||
enum class TaskStatus { RECEIVED, RUNNING, SUCCEEDED, FAILED }
|
||||
enum class ActionType { WAIT, CLICK, INPUT, BACK, EXTRACT_TEXT }
|
||||
|
||||
data class NodeSelector(
|
||||
val resourceId: String? = null,
|
||||
val text: String? = null,
|
||||
val contentDescription: String? = null,
|
||||
val className: String? = null,
|
||||
val clickable: Boolean? = null,
|
||||
) {
|
||||
fun isEmpty(): Boolean = resourceId == null && text == null && contentDescription == null &&
|
||||
className == null && clickable == null
|
||||
}
|
||||
|
||||
data class TaskStep(
|
||||
val id: String,
|
||||
val action: ActionType,
|
||||
val packageName: String,
|
||||
val activityName: String? = null,
|
||||
val selector: NodeSelector? = null,
|
||||
val value: String? = null,
|
||||
val outputField: String? = null,
|
||||
val timeoutMs: Long = 5_000,
|
||||
val optional: Boolean = false,
|
||||
)
|
||||
|
||||
data class AgentTask(
|
||||
val taskId: String,
|
||||
val revision: Long,
|
||||
val assignedAgentName: String,
|
||||
val steps: List<TaskStep>,
|
||||
val rawPayload: String,
|
||||
)
|
||||
|
||||
data class TaskExecutionResult(
|
||||
val successful: Boolean,
|
||||
val code: String,
|
||||
val message: String,
|
||||
val extracted: Map<String, List<String>> = emptyMap(),
|
||||
)
|
||||
@@ -0,0 +1,66 @@
|
||||
package cn.auto.agent.core.persistence
|
||||
|
||||
import android.content.ContentValues
|
||||
import android.content.Context
|
||||
import android.database.sqlite.SQLiteDatabase
|
||||
import android.database.sqlite.SQLiteOpenHelper
|
||||
import cn.auto.agent.core.model.AgentTask
|
||||
import cn.auto.agent.core.model.TaskStatus
|
||||
import cn.auto.agent.core.protocol.TaskParser
|
||||
|
||||
class SqliteTaskInbox(context: Context) : SQLiteOpenHelper(context, DATABASE_NAME, null, DATABASE_VERSION), TaskInbox {
|
||||
override fun onCreate(db: SQLiteDatabase) {
|
||||
db.execSQL(
|
||||
"""
|
||||
CREATE TABLE task_inbox (
|
||||
task_id TEXT NOT NULL,
|
||||
revision INTEGER NOT NULL,
|
||||
assigned_agent_name TEXT NOT NULL,
|
||||
payload_json TEXT NOT NULL,
|
||||
status TEXT NOT NULL,
|
||||
received_at INTEGER NOT NULL,
|
||||
updated_at INTEGER NOT NULL,
|
||||
PRIMARY KEY (task_id, revision)
|
||||
)
|
||||
""".trimIndent(),
|
||||
)
|
||||
}
|
||||
|
||||
override fun onUpgrade(db: SQLiteDatabase, oldVersion: Int, newVersion: Int) = Unit
|
||||
|
||||
override fun save(task: AgentTask) {
|
||||
val now = System.currentTimeMillis()
|
||||
writableDatabase.insertWithOnConflict(
|
||||
"task_inbox",
|
||||
null,
|
||||
ContentValues().apply {
|
||||
put("task_id", task.taskId)
|
||||
put("revision", task.revision)
|
||||
put("assigned_agent_name", task.assignedAgentName)
|
||||
put("payload_json", task.rawPayload)
|
||||
put("status", TaskStatus.RECEIVED.name.lowercase())
|
||||
put("received_at", now)
|
||||
put("updated_at", now)
|
||||
},
|
||||
SQLiteDatabase.CONFLICT_IGNORE,
|
||||
)
|
||||
}
|
||||
|
||||
override fun find(taskId: String, revision: Long): AgentTask? = readableDatabase.query(
|
||||
"task_inbox",
|
||||
arrayOf("payload_json"),
|
||||
"task_id = ? AND revision = ?",
|
||||
arrayOf(taskId, revision.toString()),
|
||||
null,
|
||||
null,
|
||||
null,
|
||||
"1",
|
||||
).use { cursor ->
|
||||
if (!cursor.moveToFirst()) null else TaskParser.parse(cursor.getString(0))
|
||||
}
|
||||
|
||||
private companion object {
|
||||
const val DATABASE_NAME = "auto_agent.db"
|
||||
const val DATABASE_VERSION = 1
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,8 @@
|
||||
package cn.auto.agent.core.persistence
|
||||
|
||||
import cn.auto.agent.core.model.AgentTask
|
||||
|
||||
interface TaskInbox {
|
||||
fun save(task: AgentTask)
|
||||
fun find(taskId: String, revision: Long): AgentTask?
|
||||
}
|
||||
@@ -0,0 +1,68 @@
|
||||
package cn.auto.agent.core.protocol
|
||||
|
||||
import cn.auto.agent.core.model.ActionType
|
||||
import cn.auto.agent.core.model.AgentTask
|
||||
import cn.auto.agent.core.model.NodeSelector
|
||||
import cn.auto.agent.core.model.TaskStep
|
||||
import org.json.JSONObject
|
||||
|
||||
class TaskProtocolException(message: String) : IllegalArgumentException(message)
|
||||
|
||||
object TaskParser {
|
||||
fun parse(raw: String): AgentTask {
|
||||
val root = runCatching { JSONObject(raw) }
|
||||
.getOrElse { throw TaskProtocolException("任务不是有效的 JSON 对象") }
|
||||
if (root.optInt("schemaVersion", -1) != 1) throw TaskProtocolException("仅支持 schemaVersion=1")
|
||||
val taskId = root.requiredString("taskId")
|
||||
val revision = root.optLong("revision", 1).also {
|
||||
if (it <= 0) throw TaskProtocolException("revision 必须为正整数")
|
||||
}
|
||||
val assignedAgentName = root.requiredString("assignedAgentName")
|
||||
val items = root.optJSONArray("steps") ?: throw TaskProtocolException("steps 必须是数组")
|
||||
if (items.length() !in 1..200) throw TaskProtocolException("steps 必须包含 1..200 项")
|
||||
val ids = mutableSetOf<String>()
|
||||
val steps = (0 until items.length()).map { index ->
|
||||
val item = items.optJSONObject(index) ?: throw TaskProtocolException("steps[$index] 必须是对象")
|
||||
val id = item.requiredString("id")
|
||||
if (!ids.add(id)) throw TaskProtocolException("步骤 id 不能重复")
|
||||
val action = runCatching { ActionType.valueOf(item.requiredString("action").uppercase()) }
|
||||
.getOrElse { throw TaskProtocolException("步骤 $id 的 action 不受支持") }
|
||||
val selector = item.optJSONObject("selector")?.let {
|
||||
NodeSelector(
|
||||
resourceId = it.optionalString("resourceId"),
|
||||
text = it.optionalString("text"),
|
||||
contentDescription = it.optionalString("contentDescription"),
|
||||
className = it.optionalString("className"),
|
||||
clickable = if (it.has("clickable")) it.getBoolean("clickable") else null,
|
||||
)
|
||||
}
|
||||
if (action !in setOf(ActionType.BACK) && (selector == null || selector.isEmpty())) {
|
||||
throw TaskProtocolException("步骤 $id 缺少 selector")
|
||||
}
|
||||
val value = item.optionalString("value")
|
||||
val outputField = item.optionalString("outputField")
|
||||
if (action == ActionType.INPUT && value == null) throw TaskProtocolException("输入步骤 $id 缺少 value")
|
||||
if (action == ActionType.EXTRACT_TEXT && outputField == null) throw TaskProtocolException("读取步骤 $id 缺少 outputField")
|
||||
TaskStep(
|
||||
id = id,
|
||||
action = action,
|
||||
packageName = item.requiredString("packageName"),
|
||||
activityName = item.optionalString("activityName"),
|
||||
selector = selector,
|
||||
value = value,
|
||||
outputField = outputField,
|
||||
timeoutMs = item.optLong("timeoutMs", 5_000).also {
|
||||
if (it !in 100..30_000) throw TaskProtocolException("步骤 $id 的 timeoutMs 必须为 100..30000")
|
||||
},
|
||||
optional = item.optBoolean("optional", false),
|
||||
)
|
||||
}
|
||||
return AgentTask(taskId, revision, assignedAgentName, steps, raw)
|
||||
}
|
||||
|
||||
private fun JSONObject.requiredString(name: String): String = optionalString(name)
|
||||
?: throw TaskProtocolException("$name 必填")
|
||||
|
||||
private fun JSONObject.optionalString(name: String): String? =
|
||||
optString(name).trim().takeIf { it.isNotEmpty() }
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
package cn.auto.agent.core.scheduler
|
||||
|
||||
/** External service adapter. Implement this interface after the server API is confirmed. */
|
||||
interface TaskGateway {
|
||||
fun fetchTaskPayloads(): List<String>
|
||||
}
|
||||
|
||||
data class TaskSyncSummary(
|
||||
val fetched: Int,
|
||||
val accepted: Int,
|
||||
val discarded: Int,
|
||||
val invalid: Int,
|
||||
)
|
||||
@@ -0,0 +1,31 @@
|
||||
package cn.auto.agent.core.scheduler
|
||||
|
||||
import cn.auto.agent.core.persistence.TaskInbox
|
||||
import cn.auto.agent.core.protocol.TaskParser
|
||||
|
||||
class TaskSynchronizer(
|
||||
private val gateway: TaskGateway,
|
||||
private val inbox: TaskInbox,
|
||||
) {
|
||||
fun sync(localAgentName: String): TaskSyncSummary {
|
||||
require(localAgentName.isNotBlank()) { "设备名不能为空" }
|
||||
val payloads = gateway.fetchTaskPayloads()
|
||||
var accepted = 0
|
||||
var discarded = 0
|
||||
var invalid = 0
|
||||
payloads.forEach { raw ->
|
||||
val task = runCatching { TaskParser.parse(raw) }.getOrElse {
|
||||
invalid++
|
||||
return@forEach
|
||||
}
|
||||
// Exact, case-sensitive comparison is intentional. Non-target tasks are never persisted.
|
||||
if (task.assignedAgentName != localAgentName) {
|
||||
discarded++
|
||||
return@forEach
|
||||
}
|
||||
inbox.save(task)
|
||||
accepted++
|
||||
}
|
||||
return TaskSyncSummary(payloads.size, accepted, discarded, invalid)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,5 @@
|
||||
<resources>
|
||||
<string name="app_name">Auto Agent</string>
|
||||
<string name="accessibility_service_name">autoagent</string>
|
||||
<string name="accessibility_service_description">根据已授权的任务步骤执行点击、输入、返回、滑动和信息读取操作。</string>
|
||||
</resources>
|
||||
@@ -0,0 +1,7 @@
|
||||
<resources>
|
||||
<style name="Theme.AutoAgent" parent="Theme.MaterialComponents.DayNight.DarkActionBar">
|
||||
<item name="colorPrimary">#2563EB</item>
|
||||
<item name="colorPrimaryVariant">#1D4ED8</item>
|
||||
<item name="colorSecondary">#22C55E</item>
|
||||
</style>
|
||||
</resources>
|
||||
@@ -0,0 +1,9 @@
|
||||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<accessibility-service xmlns:android="http://schemas.android.com/apk/res/android"
|
||||
android:accessibilityEventTypes="typeWindowStateChanged|typeWindowContentChanged"
|
||||
android:accessibilityFeedbackType="feedbackGeneric"
|
||||
android:accessibilityFlags="flagReportViewIds|flagIncludeNotImportantViews|flagRetrieveInteractiveWindows"
|
||||
android:canPerformGestures="true"
|
||||
android:canRetrieveWindowContent="true"
|
||||
android:description="@string/accessibility_service_description"
|
||||
android:notificationTimeout="100" />
|
||||
@@ -0,0 +1,42 @@
|
||||
package cn.auto.agent
|
||||
|
||||
import cn.auto.agent.core.executor.ActionDriver
|
||||
import cn.auto.agent.core.executor.TaskExecutor
|
||||
import cn.auto.agent.core.executor.UiNodeRef
|
||||
import cn.auto.agent.core.model.NodeSelector
|
||||
import cn.auto.agent.core.protocol.TaskParser
|
||||
import org.junit.Assert.assertEquals
|
||||
import org.junit.Assert.assertTrue
|
||||
import org.junit.Test
|
||||
|
||||
class TaskExecutorTest {
|
||||
@Test
|
||||
fun executesUniqueClick() {
|
||||
var clicked = false
|
||||
val driver = object : ActionDriver {
|
||||
override fun currentPackage() = "com.example.target"
|
||||
override fun currentActivity(): String? = null
|
||||
override fun find(selector: NodeSelector) = listOf(UiNodeRef("node", "登录"))
|
||||
override fun click(node: UiNodeRef): Boolean { clicked = true; return true }
|
||||
override fun input(node: UiNodeRef, value: String) = false
|
||||
override fun back() = false
|
||||
}
|
||||
val result = TaskExecutor(driver).execute(TaskParser.parse(TaskParserTest.taskJson("phone-01")))
|
||||
assertTrue(result.successful)
|
||||
assertTrue(clicked)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun rejectsAmbiguousNodes() {
|
||||
val driver = object : ActionDriver {
|
||||
override fun currentPackage() = "com.example.target"
|
||||
override fun currentActivity(): String? = null
|
||||
override fun find(selector: NodeSelector) = listOf(UiNodeRef(1, null), UiNodeRef(2, null))
|
||||
override fun click(node: UiNodeRef) = true
|
||||
override fun input(node: UiNodeRef, value: String) = true
|
||||
override fun back() = true
|
||||
}
|
||||
val result = TaskExecutor(driver).execute(TaskParser.parse(TaskParserTest.taskJson("phone-01")))
|
||||
assertEquals("TASK_AMBIGUOUS", result.code)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,40 @@
|
||||
package cn.auto.agent
|
||||
|
||||
import cn.auto.agent.core.model.ActionType
|
||||
import cn.auto.agent.core.protocol.TaskParser
|
||||
import cn.auto.agent.core.protocol.TaskProtocolException
|
||||
import org.junit.Assert.assertEquals
|
||||
import org.junit.Test
|
||||
|
||||
class TaskParserTest {
|
||||
@Test
|
||||
fun parsesTypedTask() {
|
||||
val task = TaskParser.parse(taskJson("phone-01"))
|
||||
assertEquals("task-1", task.taskId)
|
||||
assertEquals("phone-01", task.assignedAgentName)
|
||||
assertEquals(ActionType.CLICK, task.steps.single().action)
|
||||
}
|
||||
|
||||
@Test(expected = TaskProtocolException::class)
|
||||
fun rejectsUnsupportedAction() {
|
||||
TaskParser.parse(taskJson("phone-01").replace("CLICK", "SHELL"))
|
||||
}
|
||||
|
||||
companion object {
|
||||
fun taskJson(agentName: String) = """
|
||||
{
|
||||
"schemaVersion": 1,
|
||||
"taskId": "task-1",
|
||||
"revision": 1,
|
||||
"assignedAgentName": "$agentName",
|
||||
"steps": [{
|
||||
"id": "click-login",
|
||||
"action": "CLICK",
|
||||
"packageName": "com.example.target",
|
||||
"selector": {"resourceId": "com.example.target:id/login"},
|
||||
"timeoutMs": 1000
|
||||
}]
|
||||
}
|
||||
""".trimIndent()
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,37 @@
|
||||
package cn.auto.agent
|
||||
|
||||
import cn.auto.agent.core.model.AgentTask
|
||||
import cn.auto.agent.core.persistence.TaskInbox
|
||||
import cn.auto.agent.core.scheduler.TaskGateway
|
||||
import cn.auto.agent.core.scheduler.TaskSynchronizer
|
||||
import org.junit.Assert.assertEquals
|
||||
import org.junit.Test
|
||||
|
||||
class TaskSynchronizerTest {
|
||||
@Test
|
||||
fun onlyPersistsTasksAssignedToExactLocalName() {
|
||||
val inbox = MemoryInbox()
|
||||
val gateway = object : TaskGateway {
|
||||
override fun fetchTaskPayloads() = listOf(
|
||||
TaskParserTest.taskJson("phone-01"),
|
||||
TaskParserTest.taskJson("Phone-01").replace("task-1", "task-2"),
|
||||
TaskParserTest.taskJson("phone-02").replace("task-1", "task-3"),
|
||||
"invalid",
|
||||
)
|
||||
}
|
||||
|
||||
val result = TaskSynchronizer(gateway, inbox).sync("phone-01")
|
||||
|
||||
assertEquals(4, result.fetched)
|
||||
assertEquals(1, result.accepted)
|
||||
assertEquals(2, result.discarded)
|
||||
assertEquals(1, result.invalid)
|
||||
assertEquals(listOf("task-1"), inbox.tasks.map { it.taskId })
|
||||
}
|
||||
|
||||
private class MemoryInbox : TaskInbox {
|
||||
val tasks = mutableListOf<AgentTask>()
|
||||
override fun save(task: AgentTask) { tasks += task }
|
||||
override fun find(taskId: String, revision: Long) = tasks.find { it.taskId == taskId && it.revision == revision }
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
plugins {
|
||||
id("com.android.application") version "8.2.0" apply false
|
||||
id("org.jetbrains.kotlin.android") version "1.9.22" apply false
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
org.gradle.jvmargs=-Xmx2048m -Dfile.encoding=UTF-8
|
||||
android.useAndroidX=true
|
||||
kotlin.code.style=official
|
||||
BIN
Binary file not shown.
@@ -0,0 +1,8 @@
|
||||
distributionBase=GRADLE_USER_HOME
|
||||
distributionPath=wrapper/dists
|
||||
distributionUrl=https\://services.gradle.org/distributions/gradle-8.2-bin.zip
|
||||
distributionSha256Sum=38f66cd6eef217b4c35855bb11ea4e9fbc53594ccccb5fb82dfd317ef8c2c5a3
|
||||
networkTimeout=10000
|
||||
validateDistributionUrl=true
|
||||
zipStoreBase=GRADLE_USER_HOME
|
||||
zipStorePath=wrapper/dists
|
||||
Vendored
+248
@@ -0,0 +1,248 @@
|
||||
#!/bin/sh
|
||||
|
||||
#
|
||||
# Copyright © 2015-2021 the original authors.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
|
||||
##############################################################################
|
||||
#
|
||||
# Gradle start up script for POSIX generated by Gradle.
|
||||
#
|
||||
# Important for running:
|
||||
#
|
||||
# (1) You need a POSIX-compliant shell to run this script. If your /bin/sh is
|
||||
# noncompliant, but you have some other compliant shell such as ksh or
|
||||
# bash, then to run this script, type that shell name before the whole
|
||||
# command line, like:
|
||||
#
|
||||
# ksh Gradle
|
||||
#
|
||||
# Busybox and similar reduced shells will NOT work, because this script
|
||||
# requires all of these POSIX shell features:
|
||||
# * functions;
|
||||
# * expansions «$var», «${var}», «${var:-default}», «${var+SET}»,
|
||||
# «${var#prefix}», «${var%suffix}», and «$( cmd )»;
|
||||
# * compound commands having a testable exit status, especially «case»;
|
||||
# * various built-in commands including «command», «set», and «ulimit».
|
||||
#
|
||||
# Important for patching:
|
||||
#
|
||||
# (2) This script targets any POSIX shell, so it avoids extensions provided
|
||||
# by Bash, Ksh, etc; in particular arrays are avoided.
|
||||
#
|
||||
# The "traditional" practice of packing multiple parameters into a
|
||||
# space-separated string is a well documented source of bugs and security
|
||||
# problems, so this is (mostly) avoided, by progressively accumulating
|
||||
# options in "$@", and eventually passing that to Java.
|
||||
#
|
||||
# Where the inherited environment variables (DEFAULT_JVM_OPTS, JAVA_OPTS,
|
||||
# and GRADLE_OPTS) rely on word-splitting, this is performed explicitly;
|
||||
# see the in-line comments for details.
|
||||
#
|
||||
# There are tweaks for specific operating systems such as AIX, CygWin,
|
||||
# Darwin, MinGW, and NonStop.
|
||||
#
|
||||
# (3) This script is generated from the Groovy template
|
||||
# https://github.com/gradle/gradle/blob/HEAD/subprojects/plugins/src/main/resources/org/gradle/api/internal/plugins/unixStartScript.txt
|
||||
# within the Gradle project.
|
||||
#
|
||||
# You can find Gradle at https://github.com/gradle/gradle/.
|
||||
#
|
||||
##############################################################################
|
||||
|
||||
# Attempt to set APP_HOME
|
||||
|
||||
# Resolve links: $0 may be a link
|
||||
app_path=$0
|
||||
|
||||
# Need this for daisy-chained symlinks.
|
||||
while
|
||||
APP_HOME=${app_path%"${app_path##*/}"} # leaves a trailing /; empty if no leading path
|
||||
[ -h "$app_path" ]
|
||||
do
|
||||
ls=$( ls -ld "$app_path" )
|
||||
link=${ls#*' -> '}
|
||||
case $link in #(
|
||||
/*) app_path=$link ;; #(
|
||||
*) app_path=$APP_HOME$link ;;
|
||||
esac
|
||||
done
|
||||
|
||||
# This is normally unused
|
||||
# shellcheck disable=SC2034
|
||||
APP_BASE_NAME=${0##*/}
|
||||
APP_HOME=$( cd "${APP_HOME:-./}" && pwd -P ) || exit
|
||||
|
||||
# Use the maximum available, or set MAX_FD != -1 to use that value.
|
||||
MAX_FD=maximum
|
||||
|
||||
warn () {
|
||||
echo "$*"
|
||||
} >&2
|
||||
|
||||
die () {
|
||||
echo
|
||||
echo "$*"
|
||||
echo
|
||||
exit 1
|
||||
} >&2
|
||||
|
||||
# OS specific support (must be 'true' or 'false').
|
||||
cygwin=false
|
||||
msys=false
|
||||
darwin=false
|
||||
nonstop=false
|
||||
case "$( uname )" in #(
|
||||
CYGWIN* ) cygwin=true ;; #(
|
||||
Darwin* ) darwin=true ;; #(
|
||||
MSYS* | MINGW* ) msys=true ;; #(
|
||||
NONSTOP* ) nonstop=true ;;
|
||||
esac
|
||||
|
||||
CLASSPATH=$APP_HOME/gradle/wrapper/gradle-wrapper.jar
|
||||
|
||||
|
||||
# Determine the Java command to use to start the JVM.
|
||||
if [ -n "$JAVA_HOME" ] ; then
|
||||
if [ -x "$JAVA_HOME/jre/sh/java" ] ; then
|
||||
# IBM's JDK on AIX uses strange locations for the executables
|
||||
JAVACMD=$JAVA_HOME/jre/sh/java
|
||||
else
|
||||
JAVACMD=$JAVA_HOME/bin/java
|
||||
fi
|
||||
if [ ! -x "$JAVACMD" ] ; then
|
||||
die "ERROR: JAVA_HOME is set to an invalid directory: $JAVA_HOME
|
||||
|
||||
Please set the JAVA_HOME variable in your environment to match the
|
||||
location of your Java installation."
|
||||
fi
|
||||
else
|
||||
JAVACMD=java
|
||||
if ! command -v java >/dev/null 2>&1
|
||||
then
|
||||
die "ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH.
|
||||
|
||||
Please set the JAVA_HOME variable in your environment to match the
|
||||
location of your Java installation."
|
||||
fi
|
||||
fi
|
||||
|
||||
# Increase the maximum file descriptors if we can.
|
||||
if ! "$cygwin" && ! "$darwin" && ! "$nonstop" ; then
|
||||
case $MAX_FD in #(
|
||||
max*)
|
||||
# In POSIX sh, ulimit -H is undefined. That's why the result is checked to see if it worked.
|
||||
# shellcheck disable=SC3045
|
||||
MAX_FD=$( ulimit -H -n ) ||
|
||||
warn "Could not query maximum file descriptor limit"
|
||||
esac
|
||||
case $MAX_FD in #(
|
||||
'' | soft) :;; #(
|
||||
*)
|
||||
# In POSIX sh, ulimit -n is undefined. That's why the result is checked to see if it worked.
|
||||
# shellcheck disable=SC3045
|
||||
ulimit -n "$MAX_FD" ||
|
||||
warn "Could not set maximum file descriptor limit to $MAX_FD"
|
||||
esac
|
||||
fi
|
||||
|
||||
# Collect all arguments for the java command, stacking in reverse order:
|
||||
# * args from the command line
|
||||
# * the main class name
|
||||
# * -classpath
|
||||
# * -D...appname settings
|
||||
# * --module-path (only if needed)
|
||||
# * DEFAULT_JVM_OPTS, JAVA_OPTS, and GRADLE_OPTS environment variables.
|
||||
|
||||
# For Cygwin or MSYS, switch paths to Windows format before running java
|
||||
if "$cygwin" || "$msys" ; then
|
||||
APP_HOME=$( cygpath --path --mixed "$APP_HOME" )
|
||||
CLASSPATH=$( cygpath --path --mixed "$CLASSPATH" )
|
||||
|
||||
JAVACMD=$( cygpath --unix "$JAVACMD" )
|
||||
|
||||
# Now convert the arguments - kludge to limit ourselves to /bin/sh
|
||||
for arg do
|
||||
if
|
||||
case $arg in #(
|
||||
-*) false ;; # don't mess with options #(
|
||||
/?*) t=${arg#/} t=/${t%%/*} # looks like a POSIX filepath
|
||||
[ -e "$t" ] ;; #(
|
||||
*) false ;;
|
||||
esac
|
||||
then
|
||||
arg=$( cygpath --path --ignore --mixed "$arg" )
|
||||
fi
|
||||
# Roll the args list around exactly as many times as the number of
|
||||
# args, so each arg winds up back in the position where it started, but
|
||||
# possibly modified.
|
||||
#
|
||||
# NB: a `for` loop captures its iteration list before it begins, so
|
||||
# changing the positional parameters here affects neither the number of
|
||||
# iterations, nor the values presented in `arg`.
|
||||
shift # remove old arg
|
||||
set -- "$@" "$arg" # push replacement arg
|
||||
done
|
||||
fi
|
||||
|
||||
|
||||
# Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
|
||||
DEFAULT_JVM_OPTS='"-Xmx64m" "-Xms64m" "-Xss512k" "-XX:+UseSerialGC" "-XX:MaxMetaspaceSize=64m" "-XX:CompressedClassSpaceSize=32m" "-XX:ReservedCodeCacheSize=32m" "-XX:CICompilerCount=2"'
|
||||
|
||||
# Collect all arguments for the java command;
|
||||
# * $DEFAULT_JVM_OPTS, $JAVA_OPTS, and $GRADLE_OPTS can contain fragments of
|
||||
# shell script including quotes and variable substitutions, so put them in
|
||||
# double quotes to make sure that they get re-expanded; and
|
||||
# * put everything else in single quotes, so that it's not re-expanded.
|
||||
|
||||
set -- \
|
||||
"-Dorg.gradle.appname=$APP_BASE_NAME" \
|
||||
-classpath "$CLASSPATH" \
|
||||
org.gradle.wrapper.GradleWrapperMain \
|
||||
"$@"
|
||||
|
||||
# Stop when "xargs" is not available.
|
||||
if ! command -v xargs >/dev/null 2>&1
|
||||
then
|
||||
die "xargs is not available"
|
||||
fi
|
||||
|
||||
# Use "xargs" to parse quoted args.
|
||||
#
|
||||
# With -n1 it outputs one arg per line, with the quotes and backslashes removed.
|
||||
#
|
||||
# In Bash we could simply go:
|
||||
#
|
||||
# readarray ARGS < <( xargs -n1 <<<"$var" ) &&
|
||||
# set -- "${ARGS[@]}" "$@"
|
||||
#
|
||||
# but POSIX shell has neither arrays nor command substitution, so instead we
|
||||
# post-process each arg (as a line of input to sed) to backslash-escape any
|
||||
# character that might be a shell metacharacter, then use eval to reverse
|
||||
# that process (while maintaining the separation between arguments), and wrap
|
||||
# the whole thing up as a single "set" statement.
|
||||
#
|
||||
# This will of course break if any of these variables contains a newline or
|
||||
# an unmatched quote.
|
||||
#
|
||||
|
||||
eval "set -- $(
|
||||
printf '%s\n' "$DEFAULT_JVM_OPTS $JAVA_OPTS $GRADLE_OPTS" |
|
||||
xargs -n1 |
|
||||
sed ' s~[^-[:alnum:]+,./:=@_]~\\&~g; ' |
|
||||
tr '\n' ' '
|
||||
)" '"$@"'
|
||||
|
||||
exec "$JAVACMD" "$@"
|
||||
Vendored
+92
@@ -0,0 +1,92 @@
|
||||
@rem
|
||||
@rem Copyright 2015 the original author or authors.
|
||||
@rem
|
||||
@rem Licensed under the Apache License, Version 2.0 (the "License");
|
||||
@rem you may not use this file except in compliance with the License.
|
||||
@rem You may obtain a copy of the License at
|
||||
@rem
|
||||
@rem https://www.apache.org/licenses/LICENSE-2.0
|
||||
@rem
|
||||
@rem Unless required by applicable law or agreed to in writing, software
|
||||
@rem distributed under the License is distributed on an "AS IS" BASIS,
|
||||
@rem WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
@rem See the License for the specific language governing permissions and
|
||||
@rem limitations under the License.
|
||||
@rem
|
||||
|
||||
@if "%DEBUG%"=="" @echo off
|
||||
@rem ##########################################################################
|
||||
@rem
|
||||
@rem Gradle startup script for Windows
|
||||
@rem
|
||||
@rem ##########################################################################
|
||||
|
||||
@rem Set local scope for the variables with windows NT shell
|
||||
if "%OS%"=="Windows_NT" setlocal
|
||||
|
||||
set DIRNAME=%~dp0
|
||||
if "%DIRNAME%"=="" set DIRNAME=.
|
||||
@rem This is normally unused
|
||||
set APP_BASE_NAME=%~n0
|
||||
set APP_HOME=%DIRNAME%
|
||||
|
||||
@rem Resolve any "." and ".." in APP_HOME to make it shorter.
|
||||
for %%i in ("%APP_HOME%") do set APP_HOME=%%~fi
|
||||
|
||||
@rem Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
|
||||
set DEFAULT_JVM_OPTS="-Xmx64m" "-Xms64m" "-Xss512k" "-XX:+UseSerialGC" "-XX:MaxMetaspaceSize=64m" "-XX:CompressedClassSpaceSize=32m" "-XX:ReservedCodeCacheSize=32m" "-XX:CICompilerCount=2"
|
||||
|
||||
@rem Find java.exe
|
||||
if defined JAVA_HOME goto findJavaFromJavaHome
|
||||
|
||||
set JAVA_EXE=java.exe
|
||||
%JAVA_EXE% -version >NUL 2>&1
|
||||
if %ERRORLEVEL% equ 0 goto execute
|
||||
|
||||
echo.
|
||||
echo ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH.
|
||||
echo.
|
||||
echo Please set the JAVA_HOME variable in your environment to match the
|
||||
echo location of your Java installation.
|
||||
|
||||
goto fail
|
||||
|
||||
:findJavaFromJavaHome
|
||||
set JAVA_HOME=%JAVA_HOME:"=%
|
||||
set JAVA_EXE=%JAVA_HOME%/bin/java.exe
|
||||
|
||||
if exist "%JAVA_EXE%" goto execute
|
||||
|
||||
echo.
|
||||
echo ERROR: JAVA_HOME is set to an invalid directory: %JAVA_HOME%
|
||||
echo.
|
||||
echo Please set the JAVA_HOME variable in your environment to match the
|
||||
echo location of your Java installation.
|
||||
|
||||
goto fail
|
||||
|
||||
:execute
|
||||
@rem Setup the command line
|
||||
|
||||
set CLASSPATH=%APP_HOME%\gradle\wrapper\gradle-wrapper.jar
|
||||
|
||||
|
||||
@rem Execute Gradle
|
||||
"%JAVA_EXE%" %DEFAULT_JVM_OPTS% %JAVA_OPTS% %GRADLE_OPTS% "-Dorg.gradle.appname=%APP_BASE_NAME%" -classpath "%CLASSPATH%" org.gradle.wrapper.GradleWrapperMain %*
|
||||
|
||||
:end
|
||||
@rem End local scope for the variables with windows NT shell
|
||||
if %ERRORLEVEL% equ 0 goto mainEnd
|
||||
|
||||
:fail
|
||||
rem Set variable GRADLE_EXIT_CONSOLE if you need the _script_ return code instead of
|
||||
rem the _cmd.exe /c_ return code!
|
||||
set EXIT_CODE=%ERRORLEVEL%
|
||||
if %EXIT_CODE% equ 0 set EXIT_CODE=1
|
||||
if not ""=="%GRADLE_EXIT_CONSOLE%" exit %EXIT_CODE%
|
||||
exit /b %EXIT_CODE%
|
||||
|
||||
:mainEnd
|
||||
if "%OS%"=="Windows_NT" endlocal
|
||||
|
||||
:omega
|
||||
@@ -0,0 +1,18 @@
|
||||
pluginManagement {
|
||||
repositories {
|
||||
google()
|
||||
mavenCentral()
|
||||
gradlePluginPortal()
|
||||
}
|
||||
}
|
||||
|
||||
dependencyResolutionManagement {
|
||||
repositoriesMode.set(RepositoriesMode.FAIL_ON_PROJECT_REPOS)
|
||||
repositories {
|
||||
google()
|
||||
mavenCentral()
|
||||
}
|
||||
}
|
||||
|
||||
rootProject.name = "AutoAgent"
|
||||
include(":app")
|
||||
@@ -0,0 +1,37 @@
|
||||
# Android Agent 文档中心
|
||||
|
||||
本目录集中维护 Android Agent 的任务协议、通用动作和实现建议。根目录的 [`agent_common_action.md`](../agent_common_action.md) 是兼容入口。
|
||||
|
||||
## 文档状态
|
||||
|
||||
| 标记 | 含义 |
|
||||
|---|---|
|
||||
| **[已实现]** | 当前协议和代码已支持 |
|
||||
| **[部分实现]** | 动作可用,但部分检查或扩展尚未实现 |
|
||||
| **[提案]** | 后续设计,当前任务不要使用 |
|
||||
|
||||
当前解析器会拒绝未知动作,但仍会忽略部分未知字段。任务编写应只使用“已实现”字段,解析器后续应补充未知字段校验,避免参数被静默忽略。
|
||||
|
||||
## 文档目录
|
||||
|
||||
| 文档 | 内容 |
|
||||
|---|---|
|
||||
| [任务协议](protocol.md) | v1 任务结构、步骤字段和当前差距 |
|
||||
| [动作目录](actions/README.md) | 动作状态、统一规则和各动作入口 |
|
||||
| [统一选择器](selector.md) | 控件定位、状态和候选数量 |
|
||||
| [值与输出](values-and-outputs.md) | 输入值、简单引用和步骤输出 |
|
||||
| [执行与结果](execution.md) | 执行顺序、结果码、重试和幂等 |
|
||||
| [流程控制](flow-control.md) | 条件执行、有限分支和失败处理 |
|
||||
| [基本运行约定](security.md) | 包名核对、日志和截图的基本边界 |
|
||||
| [后续工作建议](roadmap.md) | 推荐实施顺序和测试方式 |
|
||||
| [参考文档](references.md) | Android API 与能力边界 |
|
||||
|
||||
## 事实来源
|
||||
|
||||
当前能力以 [`ActionType`](../../android/app/src/main/java/cn/auto/agent/core/model/TaskModels.kt)、[`TaskParser`](../../android/app/src/main/java/cn/auto/agent/core/protocol/TaskParser.kt)、[`TaskExecutor`](../../android/app/src/main/java/cn/auto/agent/core/executor/TaskExecutor.kt) 和 `ActionDriver` 的实际代码为准。
|
||||
|
||||
- 编写任务:先读任务协议,再读对应动作的“当前行为”。
|
||||
- 扩展动作:同时更新模型、解析器、执行器、驱动、测试和文档。
|
||||
- 共享语义只在共享文档定义,动作文件直接引用,不重复设计。
|
||||
|
||||
文档修改后检查相对链接、JSON 示例和动作状态即可,不设置额外审批或文档门禁。
|
||||
@@ -0,0 +1,43 @@
|
||||
# 通用动作目录
|
||||
|
||||
> [返回文档中心](../README.md) · [任务协议](../protocol.md) · [后续工作建议](../roadmap.md)
|
||||
|
||||
## 动作状态
|
||||
|
||||
| 动作 | 当前状态 | 文档 |
|
||||
|---|---|---|
|
||||
| `WAIT` | **[已实现]**,条件扩展为 **[提案]** | [wait.md](wait.md) |
|
||||
| `CLICK` | **[部分实现]** | [click.md](click.md) |
|
||||
| `INPUT` | **[部分实现]** | [input.md](input.md) |
|
||||
| `BACK` | **[已实现]**,单次分发和验证为 **[提案]** | [back.md](back.md) |
|
||||
| `EXTRACT_TEXT` | **[已实现]** | [extract.md](extract.md) |
|
||||
| `EXTRACT` | **[提案]** | [extract.md](extract.md) |
|
||||
| `OPEN` | **[提案]** | [open.md](open.md) |
|
||||
| `SCROLL` | **[提案]** | [scroll.md](scroll.md) |
|
||||
| `SELECT_OPTION` | **[提案]** | [select-option.md](select-option.md) |
|
||||
| `SCREENSHOT` | **[提案]** | [screenshot.md](screenshot.md) |
|
||||
| `BRANCH` | **[提案]** | [流程控制](../flow-control.md#4-branch-提案) |
|
||||
|
||||
当前代码中的动作只有 `WAIT`、`CLICK`、`INPUT`、`BACK` 和 `EXTRACT_TEXT`。
|
||||
|
||||
## 统一规则
|
||||
|
||||
每个 UI 动作遵循同一条简单流程:
|
||||
|
||||
1. 校验动作字段、`packageName` 和可选 `activityName`。
|
||||
2. 在步骤 `timeoutMs` 内等待上下文和目标控件。
|
||||
3. 默认要求目标唯一;多个目标明确失败。
|
||||
4. 执行动作;有副作用的动作默认只分发一次。
|
||||
5. 如果任务提供 `after`,重新读取页面并验证结果。
|
||||
6. 返回 `success`、`code`、`message`、`output` 和 `elapsedMs`。
|
||||
|
||||
动作文件只描述动作特有参数。选择器、条件、输出和结果结构分别引用共享文档。
|
||||
|
||||
## 实现检查清单
|
||||
|
||||
- 模型包含动作和参数。
|
||||
- 解析器校验必要字段、类型和范围。
|
||||
- 执行器有总超时,不无限等待。
|
||||
- 驱动只提供 Android 基础能力,不包含目标 App 业务规则。
|
||||
- 测试覆盖成功、未找到、多个目标、超时和底层失败。
|
||||
- 文档状态与 `ActionType` 一致。
|
||||
@@ -0,0 +1,71 @@
|
||||
# BACK
|
||||
|
||||
> [返回动作目录](README.md) · [任务协议](../protocol.md) · [执行与结果](../execution.md)
|
||||
|
||||
## 1. 当前行为 **[已实现]**
|
||||
|
||||
~~~json
|
||||
{
|
||||
"id": "go-back",
|
||||
"action": "BACK",
|
||||
"packageName": "com.example.target",
|
||||
"timeoutMs": 3000
|
||||
}
|
||||
~~~
|
||||
|
||||
当前 BACK:
|
||||
|
||||
- 不需要 selector。
|
||||
- 等待前台包和可选 Activity 匹配。
|
||||
- 调用 `GLOBAL_ACTION_BACK`。
|
||||
- 不验证返回后的页面。
|
||||
- 调用返回 `false` 时会在超时内重复调用,这是需要修正的现有问题。
|
||||
|
||||
## 2. 简单扩展 **[提案]**
|
||||
|
||||
BACK 保持“单次系统返回”,只增加可选前后条件:
|
||||
|
||||
~~~json
|
||||
{
|
||||
"id": "return-to-list",
|
||||
"action": "BACK",
|
||||
"packageName": "com.example.target",
|
||||
"before": {
|
||||
"state": "VISIBLE",
|
||||
"selector": {
|
||||
"text": "详情页面"
|
||||
}
|
||||
},
|
||||
"after": {
|
||||
"state": "VISIBLE",
|
||||
"selector": {
|
||||
"text": "列表页面"
|
||||
}
|
||||
},
|
||||
"timeoutMs": 5000
|
||||
}
|
||||
~~~
|
||||
|
||||
关闭页面内按钮使用 CLICK;HOME、RECENTS 等其他系统导航不并入 BACK。关闭输入法如果需要,可以在以后增加明确参数,不在当前动作中猜测意图。
|
||||
|
||||
## 3. 执行规则
|
||||
|
||||
1. 在总超时内满足包名、Activity 和 before。
|
||||
2. 最多调用一次 `GLOBAL_ACTION_BACK`。
|
||||
3. 调用返回 false 时立即失败。
|
||||
4. 丢弃旧页面节点。
|
||||
5. 提供 after 时等待新页面满足条件。
|
||||
|
||||
after 超时不会自动再次返回。连续返回需要编写多个 BACK 步骤,每一步都可以声明自己的页面条件。
|
||||
|
||||
## 4. 结果码
|
||||
|
||||
- `BACK_SENT`:系统接受返回请求,未声明 after。
|
||||
- `OK`:返回后 after 满足。
|
||||
- `PRECONDITION_FAILED`
|
||||
- `ACTION_FAILED`
|
||||
- `VERIFICATION_FAILED`
|
||||
|
||||
## 5. 测试
|
||||
|
||||
覆盖前置页面不符、调用成功、调用失败、单次分发、after 成功/超时和多个显式 BACK 步骤。
|
||||
@@ -0,0 +1,88 @@
|
||||
# CLICK
|
||||
|
||||
> [返回动作目录](README.md) · [任务协议](../protocol.md) · [统一选择器](../selector.md) · [执行与结果](../execution.md)
|
||||
|
||||
## 1. 当前行为 **[部分实现]**
|
||||
|
||||
~~~json
|
||||
{
|
||||
"id": "click-continue",
|
||||
"action": "CLICK",
|
||||
"packageName": "com.example.target",
|
||||
"selector": {
|
||||
"resourceId": "com.example.target:id/continue_button"
|
||||
},
|
||||
"timeoutMs": 5000
|
||||
}
|
||||
~~~
|
||||
|
||||
当前实现:
|
||||
|
||||
- 等待 selector 匹配唯一节点。
|
||||
- 节点自身不可点击时,向上查找最近的可点击祖先。
|
||||
- 调用 `ACTION_CLICK`。
|
||||
- 不检查可见、启用和有效边界。
|
||||
- 不验证点击后的页面。
|
||||
- 驱动返回 `false` 时会再次查询并重复点击,这是需要修正的现有问题。
|
||||
|
||||
## 2. 简单扩展 **[提案]**
|
||||
|
||||
~~~json
|
||||
{
|
||||
"id": "click-submit",
|
||||
"action": "CLICK",
|
||||
"packageName": "com.example.target",
|
||||
"selector": {
|
||||
"text": {
|
||||
"mode": "CONTAINS_ANY",
|
||||
"values": ["提交", "确认"]
|
||||
},
|
||||
"state": {
|
||||
"visible": true,
|
||||
"enabled": true
|
||||
}
|
||||
},
|
||||
"click": {
|
||||
"type": "SINGLE",
|
||||
"method": "AUTO"
|
||||
},
|
||||
"after": {
|
||||
"state": "VISIBLE",
|
||||
"selector": {
|
||||
"text": "操作完成"
|
||||
}
|
||||
},
|
||||
"timeoutMs": 5000
|
||||
}
|
||||
~~~
|
||||
|
||||
首批只需要:
|
||||
|
||||
- `type`:`SINGLE`、`LONG`。
|
||||
- `method`:`ACCESSIBILITY`、`CENTER_GESTURE`、`AUTO`。
|
||||
- 默认目标必须唯一。
|
||||
- `AUTO` 先使用节点动作;只有目标已唯一确认且边界有效时,才使用中心手势兜底。
|
||||
|
||||
位置排序、重叠候选归并等复杂策略按实际页面需要再增加。
|
||||
|
||||
## 3. 执行规则
|
||||
|
||||
1. 等待包名、可选 Activity 和唯一可交互目标。
|
||||
2. 解析最近的可点击祖先。
|
||||
3. 最多分发一次点击。
|
||||
4. 丢弃旧节点。
|
||||
5. 提供 after 时读取新页面并验证;未提供时只表示点击请求成功。
|
||||
|
||||
## 4. 结果码
|
||||
|
||||
- `CLICKED`:点击已分发,未声明 after。
|
||||
- `OK`:点击已分发且 after 满足。
|
||||
- `TARGET_NOT_FOUND`
|
||||
- `TARGET_AMBIGUOUS`
|
||||
- `TARGET_NOT_INTERACTABLE`
|
||||
- `ACTION_FAILED`
|
||||
- `VERIFICATION_FAILED`
|
||||
|
||||
## 5. 测试
|
||||
|
||||
覆盖节点点击、可点击祖先、多个目标、不可见/禁用、手势兜底、单次分发和 after 成功/超时。
|
||||
@@ -0,0 +1,96 @@
|
||||
# EXTRACT_TEXT 与 EXTRACT
|
||||
|
||||
> [返回动作目录](README.md) · [统一选择器](../selector.md) · [值与输出](../values-and-outputs.md)
|
||||
|
||||
## 1. EXTRACT_TEXT **[已实现]**
|
||||
|
||||
~~~json
|
||||
{
|
||||
"id": "read-status",
|
||||
"action": "EXTRACT_TEXT",
|
||||
"packageName": "com.example.target",
|
||||
"selector": {
|
||||
"resourceId": "com.example.target:id/status"
|
||||
},
|
||||
"outputField": "status",
|
||||
"timeoutMs": 3000
|
||||
}
|
||||
~~~
|
||||
|
||||
当前行为:
|
||||
|
||||
- 等待 selector 匹配唯一节点。
|
||||
- 优先读取 text,空时读取 contentDescription。
|
||||
- trim 后为空则继续等待。
|
||||
- 把非空文字追加到 `outputField` 对应的字符串列表。
|
||||
- 不自动滚动。
|
||||
|
||||
## 2. EXTRACT **[提案]**
|
||||
|
||||
EXTRACT 用于读取一个或多个节点的明确字段:
|
||||
|
||||
~~~json
|
||||
{
|
||||
"id": "read-items",
|
||||
"action": "EXTRACT",
|
||||
"packageName": "com.example.target",
|
||||
"selector": {
|
||||
"className": "android.widget.TextView",
|
||||
"state": {
|
||||
"visible": true
|
||||
}
|
||||
},
|
||||
"cardinality": "MANY",
|
||||
"fields": [
|
||||
{
|
||||
"name": "text",
|
||||
"source": "TEXT",
|
||||
"type": "TEXT",
|
||||
"trim": true
|
||||
}
|
||||
],
|
||||
"outputField": "items",
|
||||
"timeoutMs": 5000
|
||||
}
|
||||
~~~
|
||||
|
||||
首批字段来源:
|
||||
|
||||
- `TEXT`
|
||||
- `CONTENT_DESCRIPTION`
|
||||
- `RESOURCE_ID`
|
||||
- `CLASS_NAME`
|
||||
- `BOUNDS`
|
||||
- `CHECKED`、`SELECTED`、`ENABLED`
|
||||
|
||||
首批类型只需 `TEXT`、`INTEGER`、`DECIMAL` 和 `BOOLEAN`。转换失败明确返回错误,不使用默认值代替。
|
||||
|
||||
`cardinality` 支持 `ONE` 和 `MANY`。MANY 按控件树顺序输出;排序、去重和复杂对象分组等到实际需求出现时再增加。
|
||||
|
||||
## 3. 执行规则
|
||||
|
||||
1. 等待包名、Activity 和 selector。
|
||||
2. 按 cardinality 校验数量。
|
||||
3. 从同一页面快照读取声明字段。
|
||||
4. 完成 trim 和简单类型转换。
|
||||
5. 所有值都成功后一次性发布 output。
|
||||
|
||||
EXTRACT 不点击、不滚动、不读取未声明字段,也不保存完整控件树。
|
||||
|
||||
## 4. 结果码
|
||||
|
||||
- `OK`
|
||||
- `TARGET_NOT_FOUND`
|
||||
- `TARGET_AMBIGUOUS`
|
||||
- `FIELD_NOT_AVAILABLE`
|
||||
- `CONVERSION_FAILED`
|
||||
- `OUTPUT_EMPTY`
|
||||
- `EXTRACT_TIMEOUT`
|
||||
|
||||
## 5. 兼容方式
|
||||
|
||||
EXTRACT_TEXT 保留现有字符串列表结果。实现 EXTRACT 后,可以在内部把 EXTRACT_TEXT 转换为 ONE + TEXT 读取,但对外结果结构保持不变。
|
||||
|
||||
## 6. 测试
|
||||
|
||||
覆盖 text、contentDescription 回退、空文本、多个节点、MANY、字段缺失、类型转换、总超时和不自动滚动。
|
||||
@@ -0,0 +1,89 @@
|
||||
# INPUT
|
||||
|
||||
> [返回动作目录](README.md) · [任务协议](../protocol.md) · [统一选择器](../selector.md) · [值与输出](../values-and-outputs.md)
|
||||
|
||||
## 1. 当前行为 **[部分实现]**
|
||||
|
||||
~~~json
|
||||
{
|
||||
"id": "input-keyword",
|
||||
"action": "INPUT",
|
||||
"packageName": "com.example.target",
|
||||
"selector": {
|
||||
"resourceId": "com.example.target:id/input"
|
||||
},
|
||||
"value": "测试内容",
|
||||
"timeoutMs": 5000
|
||||
}
|
||||
~~~
|
||||
|
||||
当前实现对唯一节点调用 `ACTION_SET_TEXT`:
|
||||
|
||||
- value 只能是字符串。
|
||||
- 不主动请求焦点。
|
||||
- 不检查节点是否可编辑。
|
||||
- 不回读验证输入结果。
|
||||
- 驱动返回 `false` 时会重复输入,这是需要修正的现有问题。
|
||||
|
||||
## 2. 简单扩展 **[提案]**
|
||||
|
||||
~~~json
|
||||
{
|
||||
"id": "input-keyword",
|
||||
"action": "INPUT",
|
||||
"packageName": "com.example.target",
|
||||
"selector": {
|
||||
"resourceId": "com.example.target:id/input",
|
||||
"state": {
|
||||
"visible": true,
|
||||
"enabled": true
|
||||
}
|
||||
},
|
||||
"value": {
|
||||
"source": "TASK_INPUT",
|
||||
"key": "keyword"
|
||||
},
|
||||
"mode": "REPLACE",
|
||||
"verify": "EXACT",
|
||||
"timeoutMs": 5000
|
||||
}
|
||||
~~~
|
||||
|
||||
首批模式:
|
||||
|
||||
- `REPLACE`:替换全部文本。
|
||||
- `CLEAR`:清空。
|
||||
- `APPEND`:在现有文本后追加。
|
||||
|
||||
验证:
|
||||
|
||||
- `NONE`:只检查系统是否接受输入请求。
|
||||
- `EXACT`:回读文本与目标值相同。
|
||||
- `NON_EMPTY`:只检查结果非空。
|
||||
|
||||
密码节点不回读原文,可使用 NONE 或 NON_EMPTY。复杂格式化、剪贴板和模板输入等到实际需要时再增加。
|
||||
|
||||
## 3. 执行规则
|
||||
|
||||
1. 等待唯一、可见、启用且可编辑的目标。
|
||||
2. 需要时请求焦点。
|
||||
3. 计算最终输入文本。
|
||||
4. 最多调用一次输入动作。
|
||||
5. 按 verify 回读;提供 after 时继续验证页面条件。
|
||||
|
||||
`timeoutMs` 包含等待、焦点、输入和验证。
|
||||
|
||||
## 4. 结果码
|
||||
|
||||
- `INPUT_SENT`:输入已分发但未回读验证。
|
||||
- `OK`:输入验证通过。
|
||||
- `INPUT_VALUE_INVALID`
|
||||
- `TARGET_NOT_FOUND`
|
||||
- `TARGET_AMBIGUOUS`
|
||||
- `TARGET_NOT_EDITABLE`
|
||||
- `ACTION_FAILED`
|
||||
- `VERIFICATION_FAILED`
|
||||
|
||||
## 5. 测试
|
||||
|
||||
覆盖 REPLACE、CLEAR、APPEND、目标不可编辑、输入失败、回读不一致、密码节点和单次分发。WebView、自定义输入框与常用输入法需要真机验证。
|
||||
@@ -0,0 +1,77 @@
|
||||
# OPEN **[提案]**
|
||||
|
||||
> [返回动作目录](README.md) · [任务协议](../protocol.md) · [Condition](../flow-control.md#2-condition-提案)
|
||||
|
||||
OPEN 用于启动 App、把 App 带到前台或打开 URI。当前代码尚未实现。
|
||||
|
||||
## 1. 打开 App
|
||||
|
||||
~~~json
|
||||
{
|
||||
"id": "open-target",
|
||||
"action": "OPEN",
|
||||
"target": {
|
||||
"type": "APP",
|
||||
"packageName": "com.example.target"
|
||||
},
|
||||
"expectedPackageName": "com.example.target",
|
||||
"after": {
|
||||
"state": "PAGE_STABLE",
|
||||
"stableForMs": 300
|
||||
},
|
||||
"timeoutMs": 10000
|
||||
}
|
||||
~~~
|
||||
|
||||
APP 已经在前台时不重复启动,直接执行 after。否则通过 PackageManager 获取启动 Intent,并从 Service 上下文使用 `FLAG_ACTIVITY_NEW_TASK`。
|
||||
|
||||
## 2. 打开 URI
|
||||
|
||||
~~~json
|
||||
{
|
||||
"id": "open-link",
|
||||
"action": "OPEN",
|
||||
"target": {
|
||||
"type": "URI",
|
||||
"uri": "exampleapp://items/123",
|
||||
"handler": "PACKAGE",
|
||||
"packageName": "com.example.target"
|
||||
},
|
||||
"expectedPackageName": "com.example.target",
|
||||
"timeoutMs": 10000
|
||||
}
|
||||
~~~
|
||||
|
||||
`handler` 首批支持:
|
||||
|
||||
- `PACKAGE`:由指定包处理,推荐用于已知深链。
|
||||
- `DEFAULT`:使用系统默认处理程序。
|
||||
|
||||
如果需要限制 URI 的 scheme、host 或来源页面,可以作为部署配置或任务字段增加,不作为首版通用动作的固定门禁。
|
||||
|
||||
## 3. 执行规则
|
||||
|
||||
1. 解析目标并确认 App 已安装或 URI 有处理程序。
|
||||
2. 目标已经在前台时跳过 Intent 分发。
|
||||
3. 最多分发一次 Intent。
|
||||
4. 等待 `expectedPackageName` 成为前台。
|
||||
5. 提供 after 时继续验证页面条件。
|
||||
|
||||
`timeoutMs` 覆盖解析、启动、等待前台和 after。Intent 返回成功不等于目标页面已经准备完成。
|
||||
|
||||
OPEN 不写入任何具体 App 的页面文案、深链或恢复流程;这些内容由任务或目标 App 适配层提供。
|
||||
|
||||
## 4. 结果码
|
||||
|
||||
- `OPEN_ALREADY_FOREGROUND`
|
||||
- `OPEN_SENT`:目标包已进入前台,未声明 after。
|
||||
- `OK`:目标包和 after 都满足。
|
||||
- `TARGET_NOT_INSTALLED`
|
||||
- `NO_HANDLER`
|
||||
- `ACTION_FAILED`
|
||||
- `PACKAGE_MISMATCH`
|
||||
- `VERIFICATION_FAILED`
|
||||
|
||||
## 5. 测试
|
||||
|
||||
单元测试覆盖已在前台、冷启动、无处理程序、单次分发、目标包不符和 after 超时。任务栈恢复、后台启动限制和厂商系统差异需要真机验证。
|
||||
@@ -0,0 +1,82 @@
|
||||
# SCREENSHOT **[提案]**
|
||||
|
||||
> [返回动作目录](README.md) · [统一选择器](../selector.md) · [基本运行约定](../security.md)
|
||||
|
||||
SCREENSHOT 用于诊断或人工复核。当前代码尚未实现。
|
||||
|
||||
## 1. 任务结构
|
||||
|
||||
截取控件区域:
|
||||
|
||||
~~~json
|
||||
{
|
||||
"id": "capture-result",
|
||||
"action": "SCREENSHOT",
|
||||
"packageName": "com.example.target",
|
||||
"scope": {
|
||||
"type": "ELEMENT",
|
||||
"selector": {
|
||||
"resourceId": "com.example.target:id/result"
|
||||
},
|
||||
"paddingDp": 8
|
||||
},
|
||||
"format": "PNG",
|
||||
"outputField": "resultImage",
|
||||
"maxBytes": 5242880,
|
||||
"timeoutMs": 5000
|
||||
}
|
||||
~~~
|
||||
|
||||
首批范围:
|
||||
|
||||
- `SCREEN`:当前显示屏。
|
||||
- `ELEMENT`:唯一控件边界。
|
||||
- `REGION`:显式屏幕比例区域。
|
||||
|
||||
首批格式支持 PNG;需要控制体积时再增加 JPEG 和质量参数。
|
||||
|
||||
## 2. 执行规则
|
||||
|
||||
1. 确认系统版本和无障碍服务具备截图能力。
|
||||
2. 等待 packageName 和可选元素 selector。
|
||||
3. 调用系统截图接口。
|
||||
4. 在内存中按 scope 裁剪。
|
||||
5. 编码并检查 `maxBytes`。
|
||||
6. 保存到应用私有目录,返回生成的 artifactId。
|
||||
|
||||
任务不能提供任意文件系统路径。是否遮盖、上传或定期清理按实际部署需求增加,不作为首版动作的固定流程。
|
||||
|
||||
Android 无障碍截图从 API 30 开始可用;服务配置还需要声明对应能力。API 不支持时返回明确结果。
|
||||
|
||||
## 3. 输出
|
||||
|
||||
~~~json
|
||||
{
|
||||
"output": {
|
||||
"resultImage": {
|
||||
"artifactId": "generated-id",
|
||||
"format": "PNG",
|
||||
"width": 1080,
|
||||
"height": 720,
|
||||
"bytes": 245000
|
||||
}
|
||||
}
|
||||
}
|
||||
~~~
|
||||
|
||||
结果只返回产物标识和基本元数据,不把图片字节放入普通任务日志。
|
||||
|
||||
## 4. 结果码
|
||||
|
||||
- `OK`
|
||||
- `UNSUPPORTED_ON_DEVICE`
|
||||
- `TARGET_NOT_FOUND`
|
||||
- `TARGET_AMBIGUOUS`
|
||||
- `CAPTURE_FAILED`
|
||||
- `CROP_INVALID`
|
||||
- `IMAGE_TOO_LARGE`
|
||||
- `STORE_FAILED`
|
||||
|
||||
## 5. 测试
|
||||
|
||||
单元测试覆盖 scope 校验、裁剪、大小限制和输出元数据。系统回调、不同 API、屏幕方向、FLAG_SECURE 和厂商差异需要真机验证。
|
||||
@@ -0,0 +1,78 @@
|
||||
# SCROLL **[提案]**
|
||||
|
||||
> [返回动作目录](README.md) · [统一选择器](../selector.md) · [执行与结果](../execution.md)
|
||||
|
||||
SCROLL 是统一滚动动作。当前代码尚未实现。
|
||||
|
||||
## 1. 滚动到目标
|
||||
|
||||
~~~json
|
||||
{
|
||||
"id": "scroll-to-target",
|
||||
"action": "SCROLL",
|
||||
"packageName": "com.example.target",
|
||||
"mode": "TO_TARGET",
|
||||
"target": {
|
||||
"text": "目标选项"
|
||||
},
|
||||
"container": {
|
||||
"resourceId": "com.example.target:id/list"
|
||||
},
|
||||
"direction": "DOWN",
|
||||
"maxAttempts": 8,
|
||||
"timeoutMs": 10000
|
||||
}
|
||||
~~~
|
||||
|
||||
目标出现且可见时成功;每次滚动后重新读取页面。
|
||||
|
||||
## 2. 按方向滚动
|
||||
|
||||
~~~json
|
||||
{
|
||||
"id": "scroll-down",
|
||||
"action": "SCROLL",
|
||||
"packageName": "com.example.target",
|
||||
"mode": "DIRECTION",
|
||||
"container": {
|
||||
"resourceId": "com.example.target:id/list"
|
||||
},
|
||||
"direction": "DOWN",
|
||||
"amount": "PAGE",
|
||||
"maxAttempts": 1,
|
||||
"timeoutMs": 3000
|
||||
}
|
||||
~~~
|
||||
|
||||
首批字段:
|
||||
|
||||
- `mode`:`TO_TARGET`、`DIRECTION`、`TO_EDGE`。
|
||||
- `direction`:`UP`、`DOWN`、`LEFT`、`RIGHT`。
|
||||
- `amount`:`SMALL`、`PAGE`。
|
||||
- `maxAttempts`:有限正整数。
|
||||
- `container`:可选;省略时要求页面中只有一个可滚动容器。
|
||||
|
||||
## 3. 执行规则
|
||||
|
||||
1. 找到唯一、可见且可滚动的容器。
|
||||
2. TO_TARGET 先检查目标是否已经可见。
|
||||
3. 优先调用节点滚动动作;不支持时可使用容器内手势。
|
||||
4. 每次滚动后重新读取容器和目标。
|
||||
5. 达到目标、页面边缘、`maxAttempts` 或总超时时停止。
|
||||
|
||||
容器内容连续两次不变可以视为到达边缘。滚动次数和持续时间始终受步骤 `timeoutMs` 限制。
|
||||
|
||||
## 4. 结果码
|
||||
|
||||
- `OK`
|
||||
- `SCROLL_SENT`:方向滚动成功。
|
||||
- `TARGET_NOT_FOUND`
|
||||
- `CONTAINER_NOT_FOUND`
|
||||
- `CONTAINER_AMBIGUOUS`
|
||||
- `EDGE_REACHED`
|
||||
- `ACTION_FAILED`
|
||||
- `SCROLL_TIMEOUT`
|
||||
|
||||
## 5. 测试
|
||||
|
||||
覆盖目标已可见、滚动后出现、唯一容器、多个容器、边缘检测、最大次数、总超时和节点动作/手势两种实现。真实 RecyclerView、WebView 和横向列表需要真机验证。
|
||||
@@ -0,0 +1,63 @@
|
||||
# SELECT_OPTION **[提案]**
|
||||
|
||||
> [返回动作目录](README.md) · [统一选择器](../selector.md) · [SCROLL](scroll.md) · [CLICK](click.md)
|
||||
|
||||
SELECT_OPTION 是“查找选项、必要时滚动、点击并验证选中”的复合动作。当前代码尚未实现。
|
||||
|
||||
## 1. 任务结构
|
||||
|
||||
~~~json
|
||||
{
|
||||
"id": "select-value",
|
||||
"action": "SELECT_OPTION",
|
||||
"packageName": "com.example.target",
|
||||
"target": {
|
||||
"text": {
|
||||
"mode": "EXACT",
|
||||
"value": "选项 A"
|
||||
}
|
||||
},
|
||||
"container": {
|
||||
"resourceId": "com.example.target:id/options"
|
||||
},
|
||||
"search": {
|
||||
"direction": "DOWN",
|
||||
"maxAttempts": 6
|
||||
},
|
||||
"selectedWhen": {
|
||||
"state": "SELECTED",
|
||||
"selector": {
|
||||
"text": "选项 A"
|
||||
}
|
||||
},
|
||||
"timeoutMs": 10000
|
||||
}
|
||||
~~~
|
||||
|
||||
`target` 可以直接使用 selector,也可以在值引用实现后使用任务输入生成文字条件。
|
||||
|
||||
## 2. 执行规则
|
||||
|
||||
1. 先检查 selectedWhen;已经选中时直接成功。
|
||||
2. 在当前容器中查找唯一目标。
|
||||
3. 未找到且配置 search 时,按 SCROLL 规则有限滚动。
|
||||
4. 找到后按 CLICK 规则最多点击一次。
|
||||
5. 重新读取页面并验证 selectedWhen。
|
||||
6. 达到最大次数、边缘或总超时时停止。
|
||||
|
||||
没有可观察的选中状态时,不应使用 SELECT_OPTION;改用 CLICK,并由页面 after 验证。
|
||||
|
||||
## 3. 结果码
|
||||
|
||||
- `ALREADY_SELECTED`
|
||||
- `OK`
|
||||
- `OPTION_NOT_FOUND`
|
||||
- `OPTION_AMBIGUOUS`
|
||||
- `CONTAINER_NOT_FOUND`
|
||||
- `ACTION_FAILED`
|
||||
- `VERIFICATION_FAILED`
|
||||
- `SELECT_TIMEOUT`
|
||||
|
||||
## 4. 测试
|
||||
|
||||
覆盖已选中、当前页找到、滚动后找到、没有目标、多个目标、不可用选项、点击失败、验证失败和边缘停止。
|
||||
@@ -0,0 +1,80 @@
|
||||
# WAIT
|
||||
|
||||
> [返回动作目录](README.md) · [任务协议](../protocol.md) · [Condition](../flow-control.md#2-condition-提案) · [统一选择器](../selector.md)
|
||||
|
||||
## 1. 当前行为 **[已实现]**
|
||||
|
||||
~~~json
|
||||
{
|
||||
"id": "wait-ready",
|
||||
"action": "WAIT",
|
||||
"packageName": "com.example.target",
|
||||
"selector": {
|
||||
"text": "准备完成"
|
||||
},
|
||||
"timeoutMs": 5000,
|
||||
"optional": false
|
||||
}
|
||||
~~~
|
||||
|
||||
当前 WAIT:
|
||||
|
||||
- 约每 100 毫秒查询一次。
|
||||
- selector 匹配一个节点时成功。
|
||||
- 没有匹配时等待到超时。
|
||||
- 多个匹配时立即返回 `TASK_AMBIGUOUS`。
|
||||
- 超时返回 `TASK_NOT_MATCHED`;optional=true 时继续下一步。
|
||||
|
||||
## 2. Condition 扩展 **[提案]**
|
||||
|
||||
~~~json
|
||||
{
|
||||
"id": "wait-result",
|
||||
"action": "WAIT",
|
||||
"packageName": "com.example.target",
|
||||
"condition": {
|
||||
"state": "VISIBLE",
|
||||
"selector": {
|
||||
"text": {
|
||||
"mode": "CONTAINS_ANY",
|
||||
"values": ["操作完成", "处理成功"]
|
||||
}
|
||||
},
|
||||
"cardinality": "ANY"
|
||||
},
|
||||
"pollIntervalMs": 200,
|
||||
"timeoutMs": 10000
|
||||
}
|
||||
~~~
|
||||
|
||||
首批状态:
|
||||
|
||||
- `PRESENT`:存在匹配节点。
|
||||
- `GONE`:不存在匹配节点。
|
||||
- `VISIBLE`、`ENABLED`、`SELECTED`、`CHECKED`:节点满足对应状态。
|
||||
- `PAGE_STABLE`:页面结构在指定时间内保持不变。
|
||||
|
||||
`cardinality` 支持 `ONE` 和 `ANY`。v1 兼容格式仍保持 ONE,不能改变现有任务语义。
|
||||
|
||||
## 3. 执行规则
|
||||
|
||||
1. 在总 `timeoutMs` 内等待包名和可选 Activity。
|
||||
2. 每轮重新读取当前页面。
|
||||
3. 求值 Condition。
|
||||
4. 条件满足时成功,否则按 `pollIntervalMs` 继续。
|
||||
5. 读取或条件错误立即失败,不无限重试。
|
||||
|
||||
`timeoutMs` 是总时间,不因轮询或稳定性判断重新计时。
|
||||
|
||||
## 4. 结果码
|
||||
|
||||
- `OK`:条件满足。
|
||||
- `WAIT_TIMEOUT`:超时未满足。
|
||||
- `TARGET_AMBIGUOUS`:要求 ONE 但匹配多个。
|
||||
- `CONDITION_ERROR`:条件无法正常求值。
|
||||
|
||||
v1 对外仍可映射为现有 `OK`、`TASK_AMBIGUOUS` 和 `TASK_NOT_MATCHED`。
|
||||
|
||||
## 5. 测试
|
||||
|
||||
覆盖立即满足、等待后满足、超时、多目标、GONE、状态变化、页面稳定和 optional 兼容行为。
|
||||
@@ -0,0 +1,103 @@
|
||||
# 执行与结果
|
||||
|
||||
> [返回文档中心](README.md) · [动作目录](actions/README.md) · [任务协议](protocol.md)
|
||||
|
||||
## 1. 当前执行流程 **[已实现]**
|
||||
|
||||
当前 `TaskExecutor` 按以下方式执行每一步:
|
||||
|
||||
1. 计算步骤截止时间。
|
||||
2. 等待 `packageName` 和可选 `activityName` 匹配。
|
||||
3. BACK 直接调用全局返回;其他动作查询 selector。
|
||||
4. 没有目标时继续等待,多个目标立即失败。
|
||||
5. 执行动作;成功后进入下一步。
|
||||
6. 到达超时时,optional 步骤跳过,其他步骤停止任务。
|
||||
|
||||
当前结果:
|
||||
|
||||
~~~json
|
||||
{
|
||||
"successful": true,
|
||||
"code": "OK",
|
||||
"message": "任务执行完成",
|
||||
"extracted": {
|
||||
"status": ["准备完成"]
|
||||
}
|
||||
}
|
||||
~~~
|
||||
|
||||
当前结果码只有:
|
||||
|
||||
| code | 含义 |
|
||||
|---|---|
|
||||
| `OK` | 任务执行完成 |
|
||||
| `TASK_AMBIGUOUS` | selector 匹配多个节点 |
|
||||
| `TASK_NOT_MATCHED` | 包名、Activity、目标或动作在超时内未成功 |
|
||||
|
||||
## 2. 简单步骤结果 **[提案]**
|
||||
|
||||
为了便于排查,可将每一步记录为简单结构:
|
||||
|
||||
~~~json
|
||||
{
|
||||
"stepId": "click-submit",
|
||||
"action": "CLICK",
|
||||
"success": false,
|
||||
"code": "ACTION_FAILED",
|
||||
"message": "点击未成功",
|
||||
"output": {},
|
||||
"elapsedMs": 1200
|
||||
}
|
||||
~~~
|
||||
|
||||
建议只保留:
|
||||
|
||||
- `stepId`、`action`:定位步骤。
|
||||
- `success`、`code`:供程序判断。
|
||||
- `message`:供排障,不参与流程判断。
|
||||
- `output`:动作输出,没有输出时为空对象。
|
||||
- `elapsedMs`:步骤耗时。
|
||||
|
||||
暂不引入 phase、effect、transition 和独立结果协议版本。以后确实需要断点恢复或复杂流程审计时再扩展。
|
||||
|
||||
建议逐步细分以下 code:
|
||||
|
||||
- `TASK_INVALID`
|
||||
- `PACKAGE_MISMATCH`
|
||||
- `ACTIVITY_MISMATCH`
|
||||
- `TARGET_NOT_FOUND`
|
||||
- `TARGET_AMBIGUOUS`
|
||||
- `ACTION_FAILED`
|
||||
- `VERIFICATION_FAILED`
|
||||
- `UNSUPPORTED_ON_DEVICE`
|
||||
|
||||
## 3. 动作执行规则
|
||||
|
||||
- `timeoutMs` 是整个步骤的总时间,包括等待和验证。
|
||||
- 每轮查询都读取当前页面,不长期保存旧节点。
|
||||
- WAIT 和只读提取可以轮询。
|
||||
- CLICK、INPUT、BACK、OPEN 等有副作用动作默认只分发一次。
|
||||
- 提供 `after` 时,动作后重新读取页面并验证。
|
||||
- 动作失败不会自动回到初始页面;需要返回或重新打开时由任务显式增加步骤。
|
||||
|
||||
## 4. 并发与幂等
|
||||
|
||||
当前 SQLite 以 `taskId + revision` 防止同一版本重复入库,但自动执行尚未实现,`TaskExecutionMutex` 也尚未接入。
|
||||
|
||||
简单目标:
|
||||
|
||||
- 同一时刻只执行一个任务。
|
||||
- 已成功执行的相同 `taskId + revision` 不自动重复。
|
||||
- 更高 revision 是否替代旧版本,等调度需求明确后决定。
|
||||
- 暂不实现抢占、租约、断点恢复和自动回滚。
|
||||
|
||||
## 5. 测试
|
||||
|
||||
至少覆盖:
|
||||
|
||||
- 每个失败原因得到稳定 code。
|
||||
- optional 只跳过约定的失败。
|
||||
- 多目标不会执行动作。
|
||||
- 有副作用动作在一个步骤内最多调用一次。
|
||||
- after 使用动作后的新页面状态。
|
||||
- 失败结果仍保留此前已经完成的提取输出。
|
||||
@@ -0,0 +1,127 @@
|
||||
# 任务流程控制
|
||||
|
||||
> [返回文档中心](README.md) · [任务协议](protocol.md) · [执行与结果](execution.md)
|
||||
|
||||
## 1. 当前状态
|
||||
|
||||
协议 v1 只支持线性 `steps` 和 `optional`。`when`、`BRANCH`、`nextStepId` 和 `onFailure` 均为 **[提案]**。
|
||||
|
||||
流程控制只在出现实际条件流程时实现,不提前引入循环、脚本表达式或复杂工作流引擎。
|
||||
|
||||
## 2. Condition **[提案]**
|
||||
|
||||
WAIT、动作 `after`、`when` 和 BRANCH 共用简单 Condition。
|
||||
|
||||
节点条件:
|
||||
|
||||
~~~json
|
||||
{
|
||||
"state": "VISIBLE",
|
||||
"selector": {
|
||||
"text": {
|
||||
"mode": "CONTAINS",
|
||||
"value": "操作完成"
|
||||
}
|
||||
}
|
||||
}
|
||||
~~~
|
||||
|
||||
组合条件:
|
||||
|
||||
~~~json
|
||||
{
|
||||
"mode": "ALL",
|
||||
"items": [
|
||||
{
|
||||
"type": "PACKAGE_IS",
|
||||
"value": "com.example.target"
|
||||
},
|
||||
{
|
||||
"state": "VISIBLE",
|
||||
"selector": {
|
||||
"resourceId": "com.example.target:id/result"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
~~~
|
||||
|
||||
首批只需要:
|
||||
|
||||
- `ALL`、`ANY`。
|
||||
- 节点状态:`PRESENT`、`GONE`、`VISIBLE`、`ENABLED`、`SELECTED`、`CHECKED`。
|
||||
- 上下文:`PACKAGE_IS`、`ACTIVITY_IS`。
|
||||
- 结果:`STEP_SUCCESS_IS`、`STEP_CODE_IS`。
|
||||
- 输出比较:`EQ`、`NE`,需要时再增加数值比较。
|
||||
|
||||
Condition 返回 `MATCHED`、`NOT_MATCHED` 或 `ERROR`。引用缺失和读取失败属于 ERROR,不能当成条件不成立。
|
||||
|
||||
## 3. when **[提案]**
|
||||
|
||||
~~~json
|
||||
{
|
||||
"id": "click-confirm",
|
||||
"action": "CLICK",
|
||||
"packageName": "com.example.target",
|
||||
"when": {
|
||||
"type": "STEP_SUCCESS_IS",
|
||||
"stepId": "prepare",
|
||||
"value": true
|
||||
},
|
||||
"selector": {
|
||||
"text": "确认"
|
||||
}
|
||||
}
|
||||
~~~
|
||||
|
||||
- MATCHED:执行动作。
|
||||
- NOT_MATCHED:跳过步骤。
|
||||
- ERROR:步骤失败。
|
||||
- when 只判断当前状态,不等待;需要等待时使用 WAIT。
|
||||
|
||||
## 4. BRANCH **[提案]**
|
||||
|
||||
~~~json
|
||||
{
|
||||
"id": "route-result",
|
||||
"action": "BRANCH",
|
||||
"cases": [
|
||||
{
|
||||
"condition": {
|
||||
"state": "VISIBLE",
|
||||
"selector": {
|
||||
"text": "操作完成"
|
||||
}
|
||||
},
|
||||
"targetStepId": "read-result"
|
||||
}
|
||||
],
|
||||
"defaultStepId": "handle-unknown"
|
||||
}
|
||||
~~~
|
||||
|
||||
BRANCH 按顺序选择第一个匹配项。目标步骤必须存在并位于当前步骤之后,避免循环和无限执行。没有匹配项且没有 default 时返回 `BRANCH_NO_MATCH`。
|
||||
|
||||
## 5. onFailure **[提案]**
|
||||
|
||||
首版只需要三种处理:
|
||||
|
||||
- `STOP`:停止任务,默认值。
|
||||
- `CONTINUE`:继续下一步。
|
||||
- `GOTO`:跳到后续指定步骤。
|
||||
|
||||
~~~json
|
||||
{
|
||||
"onFailure": {
|
||||
"codes": ["TARGET_NOT_FOUND"],
|
||||
"action": "GOTO",
|
||||
"targetStepId": "handle-missing"
|
||||
}
|
||||
}
|
||||
~~~
|
||||
|
||||
不提供自动重试动作或整个任务。需要再次尝试时,在任务中写一个明确的后续步骤,并继续受总超时限制。
|
||||
|
||||
## 6. 最小校验
|
||||
|
||||
解析时确认步骤 id 唯一、跳转目标存在且只向前。跨分支输出和复杂汇合规则等到实际需要时再设计。
|
||||
@@ -0,0 +1,101 @@
|
||||
# 任务协议与设计原则
|
||||
|
||||
> [返回文档中心](README.md) · [动作目录](actions/README.md) · [后续工作建议](roadmap.md)
|
||||
|
||||
## 1. 状态约定
|
||||
|
||||
- **[已实现]**:当前协议 v1 可用。
|
||||
- **[部分实现]**:动作存在,但部分检查或扩展未实现。
|
||||
- **[提案]**:后续设计,当前任务不要使用。
|
||||
|
||||
当前解析器会拒绝未知动作,但会忽略部分未知字段。后续应增加字段校验;这属于协议正确性,不需要额外审批流程。
|
||||
|
||||
## 2. 协议 v1 **[已实现]**
|
||||
|
||||
~~~json
|
||||
{
|
||||
"schemaVersion": 1,
|
||||
"taskId": "task-1001",
|
||||
"revision": 1,
|
||||
"assignedAgentName": "phone-01",
|
||||
"steps": [
|
||||
{
|
||||
"id": "wait-ready",
|
||||
"action": "WAIT",
|
||||
"packageName": "com.example.target",
|
||||
"selector": {
|
||||
"text": "准备完成"
|
||||
},
|
||||
"timeoutMs": 5000,
|
||||
"optional": false
|
||||
}
|
||||
]
|
||||
}
|
||||
~~~
|
||||
|
||||
任务字段:
|
||||
|
||||
| 字段 | 规则 |
|
||||
|---|---|
|
||||
| `schemaVersion` | 当前必须为 `1` |
|
||||
| `taskId` | 非空字符串 |
|
||||
| `revision` | 正整数,默认 `1` |
|
||||
| `assignedAgentName` | 与本机设备名区分大小写、精确匹配 |
|
||||
| `steps` | 1 至 200 个,按数组顺序执行 |
|
||||
|
||||
步骤公共字段:
|
||||
|
||||
| 字段 | 规则 |
|
||||
|---|---|
|
||||
| `id` | 任务内唯一 |
|
||||
| `action` | 当前只支持五个已实现动作 |
|
||||
| `packageName` | 在超时内等待前台包精确匹配 |
|
||||
| `activityName` | 可选;只作辅助判断 |
|
||||
| `selector` | BACK 以外动作必填 |
|
||||
| `timeoutMs` | 默认 5000,范围 100..30000 毫秒 |
|
||||
| `optional` | 超时后是否继续下一步 |
|
||||
|
||||
动作专属字段:
|
||||
|
||||
- INPUT 需要字符串 `value`。
|
||||
- EXTRACT_TEXT 需要 `outputField`。
|
||||
- BACK 不需要 selector。
|
||||
- 其他字段见对应动作文档。
|
||||
|
||||
## 3. 当前执行语义
|
||||
|
||||
- 步骤线性执行。
|
||||
- 包名、Activity 或目标控件不匹配时,在 `timeoutMs` 内继续等待。
|
||||
- 默认要求 selector 只匹配一个节点;多个节点立即返回 `TASK_AMBIGUOUS`。
|
||||
- 普通失败当前统一返回 `TASK_NOT_MATCHED`。
|
||||
- 非 optional 步骤失败后停止任务;optional 步骤超时后继续。
|
||||
- 当前只搜索 `rootInActiveWindow`。
|
||||
|
||||
已知差距:
|
||||
|
||||
- 未知字段会被忽略。
|
||||
- CLICK、INPUT 和 BACK 在驱动返回 `false` 后可能重复执行。
|
||||
- 节点匹配没有统一检查可见、启用和有效边界。
|
||||
- 节点生命周期管理不完整。
|
||||
- 动作没有统一的 after 验证。
|
||||
|
||||
## 4. 设计原则
|
||||
|
||||
- 通用协议只描述跨 App 动作,不包含具体 App 页面和业务流程。
|
||||
- 选择器只负责找控件,动作负责使用控件。
|
||||
- 默认唯一匹配,不自动选择第一个。
|
||||
- 所有等待、滚动和重试都有总超时或最大次数。
|
||||
- 动作结果使用结构化 code,不依赖 message 文本判断流程。
|
||||
- 页面文字、资源 ID 和成功条件保留在任务或目标 App 适配层。
|
||||
|
||||
## 5. 后续扩展
|
||||
|
||||
后续可以按需求增加:
|
||||
|
||||
- `inputs` 和简单值引用。
|
||||
- WAIT 的 Condition。
|
||||
- 动作的 `after`。
|
||||
- `when`、前向 `BRANCH` 和简单 `onFailure`。
|
||||
- OPEN、SCROLL、SELECT_OPTION、EXTRACT 和 SCREENSHOT。
|
||||
|
||||
新增字段只有在模型、解析器、执行器和测试同时支持后,才把状态改为“已实现”。暂不引入单独的能力协商协议;出现多版本设备同时运行的实际需求时再设计。
|
||||
@@ -0,0 +1,38 @@
|
||||
# 能力边界与参考文档
|
||||
|
||||
> [返回文档中心](README.md) · [动作目录](actions/README.md)
|
||||
|
||||
## 1. 不迁移的浏览器专属动作
|
||||
|
||||
以下能力依赖浏览器 DOM、JavaScript 或 Chrome DevTools Protocol,不适合作为原生 Android Agent 的通用动作:
|
||||
|
||||
- CSS 选择器、浏览器 XPath、iframe 和 Shadow DOM 切换。
|
||||
- JavaScript Evaluate、修改 DOM 属性、注入脚本。
|
||||
- 等待网络空闲、拦截请求、Cookie、下载、上传文件控件。
|
||||
- 生成 PDF、浏览器标签页和弹窗事件。
|
||||
- Hover、鼠标右键和多次鼠标点击。
|
||||
|
||||
若目标 App 内嵌 WebView 且确实需要这些能力,应在独立 WebView 适配器中实现,不能让核心执行器依赖浏览器语义。
|
||||
|
||||
|
||||
## 2. 参考文档
|
||||
|
||||
Android 侧(规范性):
|
||||
|
||||
- Android AccessibilityService:https://developer.android.com/reference/android/accessibilityservice/AccessibilityService
|
||||
- Android AccessibilityService.takeScreenshot:https://developer.android.com/reference/android/accessibilityservice/AccessibilityService#takeScreenshot(int,%20java.util.concurrent.Executor,%20android.accessibilityservice.AccessibilityService.TakeScreenshotCallback)
|
||||
- Android AccessibilityServiceInfo 截图能力:https://developer.android.com/reference/android/accessibilityservice/AccessibilityServiceInfo#CAPABILITY_CAN_TAKE_SCREENSHOT
|
||||
- Android canTakeScreenshot 属性:https://developer.android.com/reference/android/R.attr#canTakeScreenshot
|
||||
- Android AccessibilityNodeInfo:https://developer.android.com/reference/android/view/accessibility/AccessibilityNodeInfo
|
||||
- Android Intent 与 FLAG_ACTIVITY_NEW_TASK:https://developer.android.com/reference/android/content/Intent#FLAG_ACTIVITY_NEW_TASK
|
||||
- Android PackageManager 启动入口:https://developer.android.com/reference/android/content/pm/PackageManager#getLaunchIntentForPackage(java.lang.String)
|
||||
- Android 后台 Activity 启动限制:https://developer.android.com/guide/components/activities/secure-bal
|
||||
|
||||
浏览器自动化(仅作动作命名与参数形态的横向参考,语义不适用于本协议,见本文件的[不迁移能力](#1-不迁移的浏览器专属动作)):
|
||||
|
||||
- chromedp 包文档:https://pkg.go.dev/github.com/chromedp/chromedp
|
||||
- chromedp 官方仓库:https://github.com/chromedp/chromedp
|
||||
- chromedp 官方示例:https://github.com/chromedp/examples
|
||||
- Rod 官方包文档:https://pkg.go.dev/github.com/go-rod/rod
|
||||
- Rod 官方仓库:https://github.com/go-rod/rod
|
||||
- Rod 输入动作说明:https://github.com/go-rod/go-rod.github.io/blob/main/input.md
|
||||
@@ -0,0 +1,66 @@
|
||||
# Android Agent 后续工作建议
|
||||
|
||||
> [返回文档中心](README.md) · [动作目录](actions/README.md)
|
||||
|
||||
本文档只记录推荐顺序,不是实施门禁。具体做哪一项仍以当前需求为准。
|
||||
|
||||
## 当前情况
|
||||
|
||||
- 协议 v1 支持 `WAIT`、`CLICK`、`INPUT`、`BACK` 和 `EXTRACT_TEXT`。
|
||||
- 任务可以按设备名过滤并以 `taskId + revision` 保存到 SQLite。
|
||||
- `TaskGateway` 还是接口,尚未接入具体服务端。
|
||||
- 当前没有自动轮询、自动执行、结果提交和取消流程。
|
||||
- Gradle Wrapper 位于 `android/`,仓库中的构建命令应统一从该目录执行,或以后把 Wrapper 移到仓库根目录。
|
||||
|
||||
## 推荐实现顺序
|
||||
|
||||
### 1. 修正现有 v1
|
||||
|
||||
- 未知或不适用字段明确报错,不再静默忽略。
|
||||
- 将“等待目标”和“执行动作”分开;`CLICK`、`INPUT`、`BACK` 不因驱动返回 `false` 而重复分发。
|
||||
- 将 `TASK_NOT_MATCHED` 逐步细分为包名不符、Activity 不符、目标未找到和动作失败。
|
||||
- 明确 `optional` 只跳过哪些失败。
|
||||
- 查询控件时补充可见、启用和有效边界判断,并正确释放节点。
|
||||
- 让任务保存返回“新增、重复或失败”,同步统计不再混淆。
|
||||
|
||||
### 2. 完善公共能力
|
||||
|
||||
- 先扩展统一选择器,再让 WAIT、CLICK、INPUT 和 EXTRACT 共用。
|
||||
- 增加简单 Condition,供 WAIT、`when` 和动作 `after` 使用。
|
||||
- 统一简单步骤结果和任务结果。
|
||||
- 真正接入单任务串行执行;是否持久化执行进度按实际运行需求决定。
|
||||
|
||||
### 3. 按需求增加动作
|
||||
|
||||
建议顺序:
|
||||
|
||||
1. 完善 `WAIT`、`CLICK`、`INPUT` 和 `BACK`。
|
||||
2. 将 `EXTRACT_TEXT` 扩展为 `EXTRACT`。
|
||||
3. 实现 `SCROLL`,再实现复合动作 `SELECT_OPTION`。
|
||||
4. 需要启动或切换 App 时实现 `OPEN`。
|
||||
5. 有诊断取证需求时实现 `SCREENSHOT`。
|
||||
6. 出现条件流程需求时实现 `when` 和前向 `BRANCH`。
|
||||
|
||||
没有实际需求时,不提前实现复杂正则、多窗口、任意循环、任务租约、能力协商或崩溃恢复。
|
||||
|
||||
## 简单开发流程
|
||||
|
||||
1. 确认一个动作或公共能力的输入、成功条件、超时和结果码。
|
||||
2. 同一个改动完成模型、解析、执行、驱动和测试。
|
||||
3. 运行受影响测试;Android 资源或应用代码变化时再运行 `assembleDebug`。
|
||||
4. 更新动作状态和示例,说明未做的真机验证。
|
||||
|
||||
## 测试建议
|
||||
|
||||
- 解析测试:合法字段、缺失字段、未知字段和范围错误。
|
||||
- 执行测试:成功、超时、目标不存在、目标不唯一和驱动失败。
|
||||
- 副作用测试:同一步骤不会意外点击、输入或返回多次。
|
||||
- 真机测试:只覆盖本次涉及的 Android 版本、目标 App 页面和输入法。
|
||||
- 文档测试:相对链接有效,JSON 示例能解析。
|
||||
|
||||
## 等外部信息明确后再设计
|
||||
|
||||
- 服务端任务接口与结果提交格式。
|
||||
- 是否需要不可变 `agentId`。
|
||||
- 同一任务多 revision 的保留和执行规则。
|
||||
- 是否需要自动执行、取消、断点恢复或截图上传。
|
||||
@@ -0,0 +1,29 @@
|
||||
# 基本运行约定
|
||||
|
||||
> [返回文档中心](README.md) · [动作目录](actions/README.md)
|
||||
|
||||
本项目是内部系统,不设置额外的安全审批、操作白名单门禁或文档门禁。以下只保留动作正常运行所需的基本约定。
|
||||
|
||||
## 1. 操作上下文
|
||||
|
||||
- 每个 UI 步骤在执行前核对任务声明的 `packageName`。
|
||||
- 提供 `activityName` 时同时核对,但它只作为辅助信息。
|
||||
- 节点来源包与当前前台包不一致时不执行动作。
|
||||
- 通用动作不使用 shell、root 或脚本注入。
|
||||
|
||||
是否增加额外包名范围、URI 范围或系统页面限制,按具体部署需求决定,不作为当前通用协议的前置条件。
|
||||
|
||||
## 2. 日志
|
||||
|
||||
- 日志记录任务 id、步骤 id、动作、结果码和耗时。
|
||||
- INPUT 原值、完整控件树和截图内容默认不写入日志。
|
||||
- message 用于排障,不参与程序分支判断。
|
||||
- 需要更细的数据处理规则时,在具体服务端和部署方案明确后补充。
|
||||
|
||||
## 3. 截图
|
||||
|
||||
SCREENSHOT 如果实现,默认保存到应用私有目录,任务结果返回标识而不是任意文件路径。是否遮盖、上传和定期清理按实际使用场景配置。
|
||||
|
||||
## 4. 页面稳定判断
|
||||
|
||||
PAGE_STABLE 可以比较页面结构、节点数量和稳定属性;不需要保存完整页面文本。首版只用于等待页面不再变化,不作为长期指纹或审计数据。
|
||||
@@ -0,0 +1,91 @@
|
||||
# 统一选择器
|
||||
|
||||
> [返回文档中心](README.md) · [动作目录](actions/README.md)
|
||||
|
||||
选择器只负责从当前页面中找出匹配节点,不等待、不点击,也不默认选择第一个。
|
||||
|
||||
## 1. 当前字段 **[已实现]**
|
||||
|
||||
~~~json
|
||||
{
|
||||
"resourceId": "com.example.target:id/continue_button",
|
||||
"text": "继续",
|
||||
"contentDescription": "继续操作",
|
||||
"className": "android.widget.Button",
|
||||
"clickable": true
|
||||
}
|
||||
~~~
|
||||
|
||||
规则:
|
||||
|
||||
- 已提供字段按 AND 组合。
|
||||
- `resourceId`、`text`、`contentDescription` 和 `className` 精确匹配。
|
||||
- `clickable` 匹配节点自身状态。
|
||||
- 至少提供一个字段。
|
||||
- 当前只遍历 `rootInActiveWindow`。
|
||||
|
||||
## 2. 简单扩展 **[提案]**
|
||||
|
||||
~~~json
|
||||
{
|
||||
"resourceId": "com.example.target:id/continue_button",
|
||||
"text": {
|
||||
"source": "TEXT_OR_DESCRIPTION",
|
||||
"mode": "CONTAINS_ANY",
|
||||
"values": ["继续", "下一步"]
|
||||
},
|
||||
"state": {
|
||||
"visible": true,
|
||||
"enabled": true
|
||||
},
|
||||
"region": {
|
||||
"minXRatio": 0.0,
|
||||
"maxXRatio": 1.0,
|
||||
"minYRatio": 0.5,
|
||||
"maxYRatio": 1.0
|
||||
}
|
||||
}
|
||||
~~~
|
||||
|
||||
首批扩展建议只支持:
|
||||
|
||||
- 文字来源:`TEXT`、`CONTENT_DESCRIPTION`、`TEXT_OR_DESCRIPTION`。
|
||||
- 文字模式:`EXACT`、`CONTAINS`、`CONTAINS_ANY`。
|
||||
- 状态:`visible`、`enabled`、`clickable`、`selected`、`checked`、`focused`、`scrollable`。
|
||||
- 区域:屏幕比例矩形,节点中心点落在区域内即匹配。
|
||||
|
||||
`visible` 统一定义为 `isVisibleToUser == true`,且屏幕边界非空并与屏幕相交。
|
||||
|
||||
正则、祖先/后代关系、逻辑嵌套和多窗口查询等到出现明确场景时再增加。
|
||||
|
||||
## 3. 候选数量
|
||||
|
||||
不同动作决定如何使用匹配结果:
|
||||
|
||||
- WAIT v1 要求恰好一个。
|
||||
- CLICK、INPUT 和 BACK 以外的单目标动作默认要求一个。
|
||||
- EXTRACT 可以显式声明 `ONE` 或 `MANY`。
|
||||
- 多个候选时不默认选第一个。
|
||||
|
||||
CLICK 可以把文字节点向上解析为最近的可点击祖先;这是 CLICK 的规则,不改变基础选择器结果。
|
||||
|
||||
## 4. 查询接口
|
||||
|
||||
建议公共接口保持简单:
|
||||
|
||||
~~~kotlin
|
||||
fun findAll(
|
||||
root: AccessibilityNodeInfo,
|
||||
selector: NodeSelector,
|
||||
): List<UiNodeRef>
|
||||
~~~
|
||||
|
||||
每次轮询读取新 root。同一轮查询结束后释放不再使用的节点;动作完成后不保留旧节点引用。
|
||||
|
||||
## 5. 实现顺序
|
||||
|
||||
1. 保留 v1 精确匹配。
|
||||
2. 增加可见、启用和边界检查。
|
||||
3. 增加 `TEXT_OR_DESCRIPTION` 和 `CONTAINS_ANY`。
|
||||
4. 增加屏幕比例区域。
|
||||
5. 只有实际任务需要时再扩展复杂选择器。
|
||||
@@ -0,0 +1,102 @@
|
||||
# 值与步骤输出
|
||||
|
||||
> [返回文档中心](README.md) · [任务协议](protocol.md)
|
||||
|
||||
## 1. 当前行为 **[已实现]**
|
||||
|
||||
- INPUT 的 `value` 是步骤内字符串。
|
||||
- EXTRACT_TEXT 把文本追加到 `outputField`。
|
||||
- 最终结果中的 `extracted` 类型是 `Map<String, List<String>>`。
|
||||
- 当前没有任务输入区、跨步骤引用或类型化输出。
|
||||
|
||||
## 2. 任务输入 **[提案]**
|
||||
|
||||
变化频繁的值可以集中放在任务 `inputs`:
|
||||
|
||||
~~~json
|
||||
{
|
||||
"inputs": {
|
||||
"keyword": "测试内容",
|
||||
"quantity": 2
|
||||
}
|
||||
}
|
||||
~~~
|
||||
|
||||
首版只支持 `TEXT`、`INTEGER`、`DECIMAL` 和 `BOOLEAN`。手机号、邮编和外部编号等需要保留格式的值使用 TEXT。
|
||||
|
||||
## 3. 值引用 **[提案]**
|
||||
|
||||
字面量:
|
||||
|
||||
~~~json
|
||||
{
|
||||
"value": {
|
||||
"source": "LITERAL",
|
||||
"value": "固定内容"
|
||||
}
|
||||
}
|
||||
~~~
|
||||
|
||||
任务输入:
|
||||
|
||||
~~~json
|
||||
{
|
||||
"value": {
|
||||
"source": "TASK_INPUT",
|
||||
"key": "keyword"
|
||||
}
|
||||
}
|
||||
~~~
|
||||
|
||||
步骤输出:
|
||||
|
||||
~~~json
|
||||
{
|
||||
"value": {
|
||||
"source": "STEP_OUTPUT",
|
||||
"stepId": "read-code",
|
||||
"field": "code"
|
||||
}
|
||||
}
|
||||
~~~
|
||||
|
||||
首版不提供模板语言或表达式执行。引用不存在时返回 `VALUE_NOT_FOUND`。
|
||||
|
||||
## 4. 步骤输出 **[提案]**
|
||||
|
||||
~~~json
|
||||
{
|
||||
"output": {
|
||||
"code": {
|
||||
"type": "TEXT",
|
||||
"value": "A-100"
|
||||
}
|
||||
}
|
||||
}
|
||||
~~~
|
||||
|
||||
一个步骤可以输出多个命名值,但只有整个步骤成功时输出才生效。后续步骤只能引用已经完成的前置步骤。
|
||||
|
||||
复杂对象、列表合并和跨分支数据流等到实际需要时再设计。
|
||||
|
||||
## 5. 建议模型
|
||||
|
||||
~~~kotlin
|
||||
enum class ValueType {
|
||||
TEXT,
|
||||
INTEGER,
|
||||
DECIMAL,
|
||||
BOOLEAN,
|
||||
}
|
||||
|
||||
data class StepValue(
|
||||
val type: ValueType,
|
||||
val value: Any,
|
||||
)
|
||||
|
||||
data class StepOutput(
|
||||
val values: Map<String, StepValue>,
|
||||
)
|
||||
~~~
|
||||
|
||||
输入约束先支持必要的长度、数值范围和枚举即可,不提前加入复杂格式系统。
|
||||
@@ -0,0 +1,33 @@
|
||||
# Android Agent 通用动作定义
|
||||
|
||||
本文档保留为兼容入口。完整内容已经拆分到 [Android Agent 文档中心](agent/README.md)。
|
||||
|
||||
## 当前能力
|
||||
|
||||
当前协议 v1 支持:
|
||||
|
||||
- `WAIT`
|
||||
- `CLICK`
|
||||
- `INPUT`
|
||||
- `BACK`
|
||||
- `EXTRACT_TEXT`
|
||||
|
||||
其中 CLICK 和 INPUT 只实现了基础能力,尚无完整可交互检查和动作后验证。动作现状以代码中的 `ActionType`、`TaskParser` 和 `TaskExecutor` 为准。
|
||||
|
||||
## 文档入口
|
||||
|
||||
- [任务协议](agent/protocol.md)
|
||||
- [动作目录](agent/actions/README.md)
|
||||
- [统一选择器](agent/selector.md)
|
||||
- [值与输出](agent/values-and-outputs.md)
|
||||
- [执行与结果](agent/execution.md)
|
||||
- [流程控制](agent/flow-control.md)
|
||||
- [基本运行约定](agent/security.md)
|
||||
- [后续工作建议](agent/roadmap.md)
|
||||
- [参考文档](agent/references.md)
|
||||
|
||||
## 后续动作
|
||||
|
||||
OPEN、SCROLL、SELECT_OPTION、EXTRACT、SCREENSHOT 和 BRANCH 都是提案,当前任务不要使用。是否实现以及实现顺序以实际需求为准。
|
||||
|
||||
通用动作保持简单:有限超时、包名核对、默认唯一目标、结构化结果;不在核心协议中加入具体 App 业务流程、审批门禁或未确认的调度机制。
|
||||
Reference in New Issue
Block a user