Google Gemini Mac Update Adds Background Computer Use
Google has expanded its Gemini app for macOS with a new computer use feature, allowing the AI to operate desktop applications and websites in the background. Available initially in the US for Apple Silicon Macs, the tool handles multi-step tasks like form filling and batch editing while leaving the user’s cursor free. The rollout precedes upcoming macOS 27.3 security restrictions on app permissions.
Key points
- Google has updated the Gemini app for macOS to include computer use capabilities, enabling the AI to control local desktop applications and websites.
- The feature operates in an isolated background window, allowing users to retain full control of their screen and cursor while Gemini completes tasks.
- Access is restricted to Apple Silicon Macs and requires a Google AI Plus, Pro, or Ultra subscription, with initial availability limited to the United States.
- Google emphasizes safety controls, including user-selected app permissions, one-click stop functions, and required confirmations for sensitive actions like logins.
- The update arrives ahead of Apple’s macOS 27.3 release, which introduces stricter transparency indicators and permission requirements for apps seeking full disk access.
- Use cases demonstrated by Google include batch photo editing, filling shipping forms from PDFs, and generating presentations from live dashboard data.
Google has officially expanded the capabilities of its Gemini application on macOS, introducing a new computer use feature that allows the artificial intelligence assistant to interact directly with local desktop applications and websites. This update marks a significant shift from previous iterations, which were largely limited to processing local files or interacting with connected services like Gmail and Google Calendar. The new functionality enables Gemini to perform multi-step tasks by clicking, typing, and navigating menus on behalf of the user, effectively acting as an automated agent for repetitive digital chores.
The implementation is designed to minimize disruption to the user’s workflow. Unlike previous AI tools that might hijack the cursor or require constant user intervention, Gemini operates within an isolated background window. This window provides a live preview of the AI’s activity directly within the chat interface, allowing users to monitor progress in real time without losing control of their primary screen. Users retain the ability to stop the process at any moment with a single click, ensuring that the AI does not execute unintended actions.
Security and permission management are central to the new feature’s design. Google allows users to explicitly choose which applications Gemini is permitted to access, rather than granting blanket permissions. The system is programmed to pause for sensitive operations, such as entering login credentials or processing payments, requiring explicit user confirmation before proceeding. This approach aims to balance automation with user oversight, addressing potential concerns regarding AI agents executing complex tasks without human verification.
The rollout of this feature is strategically timed to precede significant changes in Apple’s operating system. Apple is preparing to release macOS 27.3, which introduces stricter security protocols for applications requesting full disk access. The new macOS version includes transparency indicators to alert users when an app is controlling another application, as well as more granular permission settings that allow users to limit access to specific apps and folders. By releasing the computer use feature now, Google allows users to adopt the technology before the new permission screens and restrictions become mandatory for all apps.
Compatibility for the new feature is limited to Apple Silicon Macs, reflecting the hardware requirements for running the underlying AI models efficiently. Additionally, access to the computer use functionality requires a paid subscription to Google AI Plus, Pro, or Ultra. The feature is currently rolling out in English and is initially available only to users in the United States. Google has stated that it will expand availability to other regions and languages in the future, though no specific timeline has been provided.
Google has highlighted several practical use cases for the new capability. These include speeding up creative workflows, such as batch color-grading photos in design software or setting up complex 3D scenes. The AI can also extract data from local PDFs, such as weights and dimensions, to automatically fill out multi-box shipping quotes. Another demonstrated use case involves pulling metrics and screenshots from live dashboards across multiple regions to build presentations in Google Slides. In a more complex demo, Google showed Gemini converting a child’s butterfly sketch into an animated 3D model using Blender and then bringing it to life through an augmented reality web app.
The introduction of computer use capabilities in Gemini reflects a broader industry trend toward AI agents that can operate across disparate software environments. As AI models become more capable of understanding and interacting with graphical user interfaces, the distinction between digital assistants and automated software workers continues to blur. However, this increased capability also raises questions about data privacy and security, particularly as AI agents gain the ability to access and manipulate sensitive information across multiple applications.
Apple’s upcoming changes to macOS permissions suggest that the company is aware of these risks. The new transparency indicators and permission configurations in macOS 27.3 are designed to give users greater visibility and control over what AI agents can see and do. Users who have already granted permissions to apps requesting full disk access will not need to re-grant them after the update, but they will be able to configure apps to request permission each time access is needed. This shift underscores the growing importance of user consent in the era of autonomous AI agents.
As Google and other tech companies continue to develop AI features that can operate on users’ devices, the balance between convenience and security will remain a critical issue. The success of Gemini’s computer use feature will likely depend on how well users trust the system to handle sensitive tasks without compromising their data or privacy. The initial rollout in the US will serve as a test case for how these technologies are received by consumers and how effectively they can integrate into daily workflows without causing disruption or security vulnerabilities.
Sources
The WireByte editorial team synthesises technology news from multiple primary sources, verifies the facts, and links every source. Articles are produced with AI assistance and reviewed under our editorial policy.