Repository navigation
Add a --system driver for system UI in the simulator - #93
Open
gabriellanata wants to merge 3 commits into
Open
gabriellanata wants to merge 3 commits into
gabriellanata wants to merge 3 commits into
Conversation
gabriellanata
marked this pull request as ready for review
October 30, 2026 00:07
The agent can't reach permission alerts and other system UI, so `describe --system` and `tap --system` run on the simulator's whole screen from the Mac, through the same SimulatorKit and accessibility APIs AXe uses. Commands now go through a small `Driver` protocol so the CLI and MCP server keep one command language whichever driver runs it.
gabriellanata
force-pushed
the
system-driver
branch
from
October 30, 2026 06:03
9e33b76 to
711e883
Compare
gabriellanata
force-pushed
the
system-driver
branch
from
October 30, 2026 07:03
711e883 to
5f51006
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The agent lives inside the app, so it can't see or tap permission alerts, the home screen or other apps.
describe --systemandtap --systemrun on the simulator's whole screen from the Mac instead, using the same SimulatorKit and accessibility APIs AXe uses. No app changes needed. MCP gets the same thing as asystemargument on those two tools.Commands now go through a small
Driverprotocol (app agent and simulator for now) so the CLI keeps one command language and just picks where it runs. Should make an Android or web driver a new folder rather than a rewrite.docs/internal/drivers.mdhas the details.These are private Xcode frameworks, so they can break with Xcode releases like AXe does. Reading the screen also needs Simulator.app running, same as AXe, so the error says to open it. Tested on an iPhone simulator with iOS 27.0, tapping Allow on a real notification alert. Stacked on
docs.