01What a Free Local Computer Use Agent Is

A free local computer use agent is an AI that controls your computer for you. It can open apps, write notes, browse the web, and do tasks without sending your data anywhere. All processing happens on your own hardware.

In this episode, Julian Goldie shows how to set this up using Olama, Claude Code or Codex, and a skill called CUA driver. You get an AI assistant that works in the background while you keep working.

02Main Tools You Need for a Local Agent

Three main pieces make a local computer use agent work. First, Olama lets you download and run local AI models like Qwen 3.6 or DeepSeek. Second, Claude Code or Codex gives the agent a command-line interface to use those models.

Third, CUA driver is the skill that gives the agent actual computer control. With CUA driver, the agent can click buttons, type text, and move the mouse. Together these tools let you build a free local agent that doesn't phone home to any cloud service.

03How to Set Up Your Local Computer Use Agent

  1. Make sure your computer has good hardware. A DGX Spark or Mac Studio works best. A Mac mini is okay but slower.
  2. Download and install Olama from its official site.
  3. Open your terminal and run the Olama command to pull a local model. For example, type olama run qwen3.6 and press Enter.
  4. Wait for the model to download. This can take a while and uses a lot of memory.
  5. Install Claude Code or Codex. If you want to use DeepSeek, run a command like claude code --model deepseek.
  6. Set up CUA driver by installing it as a skill. Follow the on-screen instructions to grant permissions.
  7. Test the agent by giving it a simple command, such as opening the Notes application using CUA driver.

04Using the Agent with CUA Driver

Once everything is installed, you can start giving commands. Julian shows two examples. First, he types "use CUA driver skill to open the Notes application".

The agent opens the app and waits for the next instruction.

Second, he asks it to write a note. The command is: "write a note about today inside Notes, using CUA driver". The agent opens Notes, types a date and a summary, and saves the note.

All of this happens without Julian touching the keyboard or mouse.

The agent works in the background. You can keep using your terminal or other apps while the agent does its job. This makes it useful for automating repetitive tasks like writing logs, creating drafts, or organizing files.

05Results and Limitations Shown in the Demo

In the live demo, the agent successfully opened Notes and wrote a full note. The note said: "May the 14th, started working with CUA driver today to automate macOS apps from the terminal" with key takeaways below it.

Julian does point out that computer use on any platform is not amazing yet. He says: "Is it smooth? Is it easy to use?

No." The agent can be slow and may require multiple attempts. But it does get the job done for simple tasks.

He also tests Hermes with CUA driver. That agent follows the same process and produces a similar result. Both agents work, but neither is polished or fast.

06Tips, Limits, and Mistakes to Avoid

Hardware matters. If your computer is not powerful enough, the local model will run very slowly. Julian says a DGX Spark or Mac Studio gives the best results.

A Mac mini is okay but not ideal. If your setup is weak, consider using a cloud API instead of a local model.

Make sure you grant all required permissions to CUA driver. Julian warns: "the one thing you have to set up first or the whole thing falls apart" is allowing the settings for CUA driver. Without that step, the agent cannot control your computer.

Do not expect a smooth experience. Computer use agents are still early-stage tools. They work best for background tasks that don't need perfect accuracy or fast speed.

07Who Should Use a Free Local Computer Use Agent

This tool fits people who want to keep all their data on their own machine. Anyone who values privacy and doesn't want their information sent to a cloud provider will like this agent. It also suits developers who want to experiment with AI agents without paying API costs.

If you do repetitive computer tasks like opening apps, writing notes, or copying data between programs, a local agent can save you time. Julian explains that the agent runs in the background so you can keep working on other things while it handles the busywork. It's a practical first step into local AI automation.

08Where to Get More Help and Training

Julian mentions there is a full training video on using Hermes with CUA driver. It's a step-by-step guide that walks through the whole process. He also invites viewers to join his community called "Aircraft for Boarding" for daily Q&A and support.

Inside that community, members can post questions, share their setup, and get answers every day. There are weekly coaching calls where you can show your screen and get live help. You can also connect with other AI agent users in your local city through the community map.