mobby · 操作手机

让 AI 替你操作手机

在对话里加上「AI 操作手机」,说出想做的事。mobby 里的 Agent 会读屏幕、点按、输入、翻页,一步一步把它做完。它在一块独立的副屏上工作,你可以在旁边看着,也可以随时接手。

你说去应用商店下载番茄ToDo,打开添加待办的功能。

  • 无需 root
  • 不用电脑
  • 副屏模式 · Android 13+
  • Pi · Claude Code · Codex · OpenCode
手机演示:Agent 从应用商店安装番茄ToDo 并打开添加待办页面
真机录屏精剪 · 小米手机 Android 13
  1. 一次操作怎样完成
  2. 三种屏幕模式
  3. 副屏
  4. 需要你的时候
  5. 安全边界
  6. 开始使用
  7. 已知限制
01

每一步都先看清,再动手

Agent 不靠记忆盲点。它每做一步都要读一次屏幕,动作做完再读一次,确认页面真的变了。

  1. 读屏控件树告诉它页面上有哪些按钮、输入框和文字;截图补上没有文字描述的部分。
  2. 判断对照你的目标,决定下一步:该点哪里、该填什么、要不要返回。
  3. 动作打开应用、点按、输入文字、滑动、返回,一次只做一件事。
  4. 核对再读一次屏幕。目标页面没有出现,就不算成功。

循环往复,直到任务完成,或者需要你来决定。

每个动作都会作为一张卡片出现在对话里,和 Agent 的思考、回复按发生顺序排在一起。运行时卡片展开,结束后自动折叠,点标题可以重新查看当时的画面和结果。

结果说不准的时候,mobby 会如实标出来,不会显示成成功。已经发出去的点按和输入不会被自动重放,避免重复下单、重复发送这类副作用。连接断了要恢复时,Agent 也得先重新读屏,才能继续动手。

做过的流程可以记住。Agent 可以把一段通过验收的点按和输入封存成流程,下次按编号回放。回放前会先核对当前的应用、页面结构和目标控件,对不上就停下,不会盲点。流程里只保存控件名称、动作和页面结构摘要,不保存你输入的文字。

02

三种屏幕模式,按你的手机来选

在「设置 → 运行环境 → 屏幕操作模式」里,选择 Agent 在哪块屏幕上工作。

当前屏幕 · 无障碍

直接操作你眼前这块屏幕

执行时 mobby 隐藏自己的悬浮层并保持亮屏。任务进行时,手机交给它来操作。适合 Android 12 及以下,或者系统没有无线调试的机型。

需要:开启「mobby 屏幕操作」无障碍服务和任务通知。

虚拟屏幕 · root

同样的副屏,权限来自 root

默认置灰。点「检查 Root 权限」,在超级用户管理器里同意后才能选择。mobby 不会帮你 root 手机,也不改系统权限。

需要:Android 13 及以上,设备本身已经有 su。

03

副屏:它在旁边干活,你继续用手机

选好虚拟屏模式后,每个任务都有一块自己的屏幕。

mobby
帮我在购物 App 里搜一下登山鞋,按销量排序
打开购物 App
点击搜索框
输入「登山鞋」
点击「销量」

主屏 · 你留在 mobby 里,右上角是实时预览

购物 App

副屏 · Agent720×1280 独立显示,你看不见它

任务开始时,mobby 会创建一块 720×1280 的独立显示屏。Agent 先找到目标应用,把它启动到这块副屏上;之后的读屏、点按、滑动都只发生在这里,碰不到你的主屏。

副屏不是应用分身。应用在副屏里用的还是你手机上原来的登录和数据。少数应用会自己跳回主屏,或者只允许一个实例;遇到这种情况,mobby 会明确报告,而不会把任务挪到你的主屏上继续做。

04

需要你的时候,它会停下来等你

登录、验证码、滑块这些步骤,Agent 不该替你做主,很多时候也做不了。遇到时它会请你处理,你处理完,它接着做。

私密输入

Agent 指出要填的输入框,并说明原因。你在 mobby 的弹窗里填写,内容直接写进副屏的输入框。Agent 读屏时看不到原值,这段时间截图也会暂停提供给它。

适合:密码、验证码、身份信息。

接手副屏

进入全屏的接手页,直接点按、拖动、返回副屏上的应用。点到输入框时,用手机自己的键盘打字,和平时一样。

适合:滑块验证、手动获取验证码、需要你亲自选的步骤。

  1. Agent 请求帮助弹窗写明要填哪个字段、为什么需要你。它不会在聊天里罗列选项等你回复,而是直接请你接手。
  2. 你接手,它立刻停手你控制副屏期间,Agent 的读屏和点按都会被拒绝,不排队,也不会跟你抢屏幕。
  3. 你交还,它重新看一遍mobby 清掉接手期间的旧画面,要求 Agent 先重新读屏,再决定下一步。
  4. 自动继续就算 Agent 已经结束了这一轮,mobby 也会在你交还后替你发一句「我已在副屏完成操作,请继续」。不用再手动催它。

用手机键盘输入的内容,交还后 Agent 读屏能看到。需要保密的内容,请用接手页里的「输入文字」,它走的是私密输入通道。

05

权限只给屏幕,动作都有边界

能操作手机的权限很大,所以 mobby 把它关在一个很小的范围里。

只给屏幕进程提权

ADB 或 root 权限只交给一个独立的屏幕工作进程。四个 Agent、Shell 和其他功能仍然是普通应用身份,跑在 Android 沙箱里。

只开放固定动作

工作进程只监听本机 127.0.0.1,用每次随机生成的令牌认证;只提供读屏、点按、输入、滑动等固定操作,没有执行任意命令的入口。

每一步重新验明身份

内置 ADB 模式下,每次操作前都会用同一配对身份重新认证本机 ADB。无线调试一关,操作立刻失败并说明原因;配对保留,重新打开后在同一个任务里就能续上。

配对凭据加密保存

ADB 身份私钥用 Android Keystore 加密保存;六位配对码只在内存里用一次,不写入磁盘,也不进日志。

不碰密码框

Agent 的文字输入会拒绝密码框。受系统保护的画面(例如部分支付页面)在截图里是黑的,mobby 不会绕过。

按对话授权

屏幕权限跟着单个对话走。移除「AI 操作手机」标签后,下一次操作要重新确认。

06

开始使用

第一次配置大约几分钟,以后打开对话就能用。

  1. 打开屏幕设置进入设置 → 运行环境 → 屏幕操作模式。Android 13 及以上会预选「虚拟屏幕(内置 ADB)」,确认后才会保存。
  2. 跟着向导一步步来通知、Wi-Fi、开发者选项、无线调试,一次只处理一项,并附有不同品牌手机的说明。已经满足的步骤会自动跳过。
  3. 完成配对开启无障碍后可以点「自动配对」,mobby 会读取系统弹窗里的六位配对码;也可以在通知里手动输入。
  4. 在对话里加上「AI 操作手机」空对话页就有这个入口,平板上叫「AI 操作平板」。然后像平时聊天一样,说出要做的事。

配对只需要一次。只要无线调试开着,mobby 会用保存的配对自动重连。手机重启后系统通常会关掉无线调试,重新打开即可;只有系统忘了这次配对,才需要再配一遍。

07

已知限制

这些是现在做不到或还没验证的地方。

mobby · Using your phone

Let AI use your phone

Add AI controls phone to a conversation and say what you want done. The agent inside mobby reads the screen, taps, types and moves between pages until the job is finished. It works on a separate virtual display, so you can watch it, or step in at any time.

You sayInstall Via browser from the app store and open Bookmarks.

  • No root
  • No computer
  • Virtual display · Android 13+
  • Pi · Claude Code · Codex · OpenCode
Phone demo: the agent installs Via browser and opens Bookmarks
Edited screen recording · Xiaomi phone, Android 13
  1. How a step works
  2. Three screen modes
  3. The virtual display
  4. When it needs you
  5. Safety limits
  6. Getting started
  7. Known limits
01

Look first, then act, every single step

The agent never taps from memory. It reads the screen before each action and again afterwards, to confirm the page really changed.

  1. ReadThe view tree lists the buttons, fields and text on the page; a screenshot covers anything without a description.
  2. DecideIt compares the page with your goal and picks the next move: where to tap, what to type, whether to go back.
  3. ActOpen an app, tap, type, swipe or go back: one action at a time.
  4. CheckIt reads the screen again. If the expected page did not appear, the step did not succeed.

The loop repeats until the task is done, or until a decision belongs to you.

Every action appears as a card in the conversation, in order with the agent's reasoning and replies. Cards expand while they run and fold when done; tap a title to see the screen and result from that moment.

When an outcome is uncertain, mobby says so instead of showing success. A tap or input that has already been sent is never replayed automatically, so nothing gets ordered or sent twice. After a dropped connection, the agent must read the screen again before it may act.

Finished flows can be remembered. The agent can save a verified series of taps and inputs as a workflow and replay it by ID later. Before replaying, it checks the current app, page structure and target controls, and stops if they do not match. A workflow keeps control names, actions and a structural summary of each page, never the text you typed.

02

Three screen modes, chosen for your phone

Under Settings → Runtime → Screen control mode, choose which screen the agent works on.

Current screen · Accessibility

It works on the screen in front of you

mobby hides its own overlays and keeps the screen on while it acts, so hand the phone over for the duration. Best for Android 12 and earlier, or phones without wireless debugging.

Needs: the mobby Screen Control accessibility service and task notifications.

Virtual display · Root

The same virtual display, powered by root

Disabled by default. Tap Check root permission and approve in your superuser manager to enable it. mobby never roots your phone or changes system permissions.

Needs: Android 13 or later, and a device that already has su.

03

The virtual display: it works beside you

With a virtual mode selected, every task gets a screen of its own.

mobby
Search for hiking boots in the shopping app and sort by sales
Open shopping app
Tap the search box
Type "hiking boots"
Tap "Sales"

Main screen · YouStay in mobby, with a live preview in the corner

Shopping app

Virtual display · AgentA separate 720×1280 display you never see

When a task starts, mobby creates a separate 720×1280 display. The agent finds the target app and launches it there; from then on every read, tap and swipe happens on that display and never touches your main screen.

It is not an app clone. Apps on the virtual display use the same sign-ins and data as on your phone. A few apps jump back to the main screen on their own or allow only one instance; mobby reports that plainly instead of moving the task onto your main screen.

04

When it needs you, it stops and waits

Sign-ins, verification codes and slider checks are not the agent's call, and often it cannot do them anyway. It asks you to handle them, then carries on.

Private input

The agent names the field to fill and why. You type into a dialog in mobby and the value goes straight into the field on the virtual display. The agent never sees the original value when it reads the screen, and screenshots are withheld from it meanwhile.

Use it for: passwords, verification codes, personal details.

Take over

Tap Take over display to open the full-screen takeover page and tap, drag or go back in the app on the virtual display yourself. Tap a text field and type with your phone's own keyboard, as usual.

Use it for: slider checks, requesting codes, choices only you should make.

  1. The agent asks for helpThe dialog states which field and why. Instead of listing options in chat and waiting for a reply, the agent asks you to take over right away.
  2. You take over; it stops at onceWhile you control the display, the agent's reads and taps are refused, not queued. It never fights you for the screen.
  3. You hand back; it looks againmobby discards frames from your session and makes the agent read the screen before deciding anything.
  4. It continues on its ownEven if the agent already ended its turn, mobby sends "I finished on the virtual display, please continue" for you. No need to nudge it.

Text you type with the phone keyboard is visible to the agent after you hand back. For anything secret, use Enter text on the takeover page, which goes through the private input channel.

05

Privileges stay with the screen, and actions stay narrow

Controlling a phone takes a lot of power, so mobby keeps that power in a very small box.

Only the screen process is elevated

ADB or root access goes to one separate screen worker. The four agents, the shell and everything else keep running as an ordinary app inside the Android sandbox.

Only fixed actions

The worker listens on 127.0.0.1 only and authenticates with a fresh random token. It offers reading, tapping, typing, swiping and similar fixed actions, with no way to run arbitrary commands.

Verified before every step

In built-in ADB mode, each action first re-authenticates with local ADB using the same pairing. Turn wireless debugging off and the action fails right away with the reason; the pairing is kept, and the same task resumes once you turn it back on.

Pairing keys are encrypted

The ADB identity key is encrypted with Android Keystore. The six-digit pairing code is used once in memory and never written to disk or logs.

Hands off password fields

The agent's text input refuses password fields. Screens the system protects, such as some payment pages, appear black in screenshots, and mobby does not get around that.

Granted per conversation

Screen access belongs to a single conversation. Remove the AI controls phone tag and the next action has to be approved again.

06

Getting started

The first setup takes a few minutes; after that, just open a conversation.

  1. Open the screen settingsGo to Settings → Runtime → Screen control mode. On Android 13 and later, Virtual display (built-in ADB) is preselected and saved only when you confirm.
  2. Follow the guideNotifications, Wi-Fi, Developer options and Wireless debugging, one at a time, with notes for each phone brand. Steps already done are skipped.
  3. PairWith accessibility on, tap Pair automatically and mobby reads the six-digit code from the system dialog. You can also type the code into a notification.
  4. Add “AI controls phone” to a conversationIt is right on the empty conversation page (“AI controls tablet” on tablets). Then describe what you want, as you would in any chat.

You pair only once. As long as wireless debugging is on, mobby reconnects with the saved pairing on its own. Phones usually turn wireless debugging off after a restart, so turn it back on; you only pair again if the system forgets the pairing.

07

Known limits

What it cannot do yet, or what has not been verified.