New tool Opposable lets AI agents physically tap and swipe smartphones
Developers released Opposable, a tool enabling AI agents to control Android and iPhone screens via taps and swipes. The system uses accessibility services and the GUI-Owl 1.5 2B vision model to interpret screens. It allows desktop AI clients to perform basic tasks like changing settings on connected devices, aiming to automate manual interactions for users who prefer agent-driven workflows.
Key points
- Opposable is a newly launched tool that allows AI agents to tap, swipe, and type on Android phones and iPhones.
- The Android implementation uses accessibility services to inspect screens, while the iPhone version requires a companion computer.
- Visual processing is handled by the GUI-Owl 1.5 2B vision-language model running through llama.cpp to determine tap locations.
- The tool is designed to work with any client supporting the Model Context Protocol (MCP) standard.
- Developers describe the tool as giving AI 'thumbs' to automate tasks like updating calendars or enabling dark mode.
- The system operates by decoding screenshots to figure out where to tap, sparing users from manually operating their devices.
A new tool called Opposable has been released, allowing artificial intelligence agents to physically interact with smartphone screens. The software enables AI to tap, swipe, and type on Android devices and control iPhones through a companion computer, effectively automating manual user interactions. The developers describe the tool as a way to give AI "thumbs," allowing agents to perform basic tasks such as updating calendar events, starting timers, or toggling dark mode without human intervention.
The technical implementation varies by operating system. On Android, Opposable utilizes accessibility services to inspect the screen state. For both platforms, the system relies on screenshots to provide visual input to an AI model. This visual processing is handled by the GUI-Owl 1.5 2B vision-language model, which runs through llama.cpp. The model decodes the images to determine the precise coordinates where the AI needs to tap to complete the requested task.
Opposable is designed to integrate with existing desktop AI agents or on-device agents that support the Model Context Protocol (MCP). This allows users to command a connected phone from their computer, offloading the physical interaction to the AI. The tool targets users who prefer to delegate simple device operations to automated agents, reducing the need for direct manual input on mobile devices.
Sources
The WireByte editorial team synthesises technology news from multiple primary sources, verifies the facts, and links every source. Articles are produced with AI assistance and reviewed under our editorial policy.
