01What Is Claude Code Computer Use and How It Works

Claude Code Computer Use is a new feature that lets the Claude AI agent take control of your computer screen. It can open apps, click buttons, type text, scroll, and navigate interfaces — all from the command line. Anthropic announced it on March 31, 2026, and it is now available inside Claude Code as a built-in MCP server.

Here is how it works: when Claude needs to interact with a GUI tool, a browser tab, or a desktop app that has no API, it uses computer use as a last resort. It first tries faster methods like MCP servers, bash commands, or the Claude Chrome extension. If none of those work, it switches to screen control. Claude takes a screenshot of your desktop, analyzes what it sees, decides what to do (click, type, scroll, or act), takes another screenshot, checks the result, and repeats until the task is done.

This loop means Claude sees your screen in real time, not from memory.

02Main Features and What Is New in This Update

This update adds computer use directly inside Claude Code, from the CLI. Before, Claude Code could only work inside the terminal. It could run bash commands, edit files, and use MCP connectors, but once a task moved into a browser or a desktop app with no API, you had to take over.

That wall is now gone.

Key features include:

  • Built-in MCP server for computer use — no separate setup needed.
  • Priority order: Claude tries MCP servers first, then bash, then the Claude Chrome extension, and finally computer use as the last resort.
  • Permission system: for every new app, Claude asks in the terminal. You approve or deny, and approvals last for the session.
  • Escape key: press Escape any time to release control, unhide your apps, and take back control immediately.
  • Lock system: only one screen control session at a time. You get a macOS notification that says "Claude is using your computer" with an option to stop.
  • App category restrictions: browsers and trading platforms are view-only; terminals and IDEs are click-only; other apps get full control.

03How to Set Up Claude Code Computer Use Step by Step

Follow these steps to enable computer use inside Claude Code. You need macOS, the right version, and screen recording permissions.

  1. Update Claude Code to version 2.185 or later. Check your terminal and run the update command if needed.
  2. Check your Claude plan. You need a Pro or Max plan. Computer use is not available on free or basic plans.
  3. Enable screen recording permissions in macOS. Go to System Settings → Privacy and Security → Screen Recording. Add your terminal app (like Terminal or iTerm2) and grant permission.
  4. Open a Claude session in your terminal by running claude.
  5. Enable the computer use server by typing /mcp and then enabling the computer use MCP server from the menu.
  6. Test with a simple task: ask Claude to open TextEdit, type a test phrase, and save it to the desktop. Watch Claude take screenshots and click through. If it works, your setup is ready.

04How to Use Claude Code to Automate GUI Tasks

Once computer use is enabled, you can automate tasks that previously required a human to sit and click. Here are real examples from the episode.

Building and testing a Swift app. Ask Claude Code to write a Swift app, compile it, launch it, click through every button, screenshot the result, and report what it found — all in the same conversation where it wrote the code. Before this, you had to write code, switch to Xcode, build, launch, click manually, find bugs, switch back to the terminal, fix, and repeat. Now Claude handles the full cycle.

Testing a client portal. A freelancer can say: "Build this portal, launch it, click through every screen, screenshot anything that looks broken, and report back." Claude Code then does everything autonomously. The freelancer just reviews the report.

You can also use computer use for internal dashboards, legacy systems, or any tool that has no API. Just describe the task in plain English. Claude will decide whether to use an MCP, bash, or screen control depending on what is most efficient.

05Results and Comparisons Shown in the Episode

In the episode, Julian demonstrated how computer use closes a gap that previously slowed down development. Before this update, any task that moved into a GUI meant you had to take over. Now that wall is gone.

Speed comparison: Computer use is slower than a direct integration because it works through screenshots. It is not a live video understanding; it takes a screenshot, analyzes, acts, then takes another screenshot. So it is best used as a last resort when no API or MCP exists.

Real-world test: The Swift app example shows the full loop: write code, compile, launch, click every button, screenshot, and report — all in one conversation. That used to require multiple manual switches between the terminal and Xcode. Now it is automated.

Permission model test: Julian noted that permissions are baked into the app categories. Browsers and trading platforms are view-only (they cannot be clicked). Terminals and IDEs are click-only (they cannot be viewed as screenshots).

Everything else gets full control. Claude cannot override these restrictions.

06Tips, Limits, and Mistakes to Avoid

Computer use is a research preview. It will not always get it right on the first try. Start with low-stakes tasks to learn how it behaves.

  • It is slow because it works through screenshots. Use MCP servers or bash first when possible.
  • Your terminal window is excluded from screenshots. Claude never sees its own output, so it cannot be tricked by on-screen prompts feeding back into the session.
  • Only one session at a time. When Claude takes the screen, you get a macOS notification saying "Claude is using your computer." You can press Escape to stop.
  • Not available on Windows yet. Only macOS is supported right now. Windows is coming.
  • Non-interactive mode does not work. You cannot use the -p flag with computer use.
  • Enable screen recording permissions before starting. Without it, nothing will run. Go to System Settings → Privacy and Security → Screen Recording and add your terminal.

07Who This Update Is For and Why It Matters

This update is for developers, freelancers, and power users who regularly need to interact with GUI tools that have no API. If you spend time every week clicking through internal dashboards, legacy systems, or testing apps manually, Claude Code Computer Use can automate those loops.

It matters because it closes the final gap between AI code generation and real-world testing. Before this, Claude could write code but could not test it in a GUI. Now it can write, build, launch, click through every button, screenshot results, and report back — all autonomously.

Julian Goldie described this as a shift: instead of doing the manual loop yourself, you just review the report.

According to the episode, the people who figure out how to use this inside their workflows early will have a real edge. If you want step-by-step coaching on building these workflows, the AR Profit Boardroom community provides tutorials, roadmaps, and weekly coaching calls.