01What Is Hermes Agent Computer Use?

Hermes Agent computer use is a free, experimental feature in the open-source Hermes Agent tool by Luce Research. It lets an AI model operate your Mac computer just like a person would—clicking, typing, scrolling, and dragging—all in the background while you keep working. Instead of copying and pasting prompts, the AI reads your screen, identifies buttons, and performs actions for you.

This update makes Hermes the most powerful it has ever been. You can handle emails, notes, files, and admin tasks without lifting a finger. In this episode, Julian Goldie tests the feature live and shows how it saves time within minutes.

02Main Features of the Computer Use Update

The new computer use update adds several key capabilities to Hermes Agent:

  • Background operation: The AI works on your Mac without moving your cursor or changing your keyboard focus. You can keep using the same machine at the same time.
  • Any AI model support: You can use Claude, ChatGPT, Gemini, or even a free local model. Only text-only models are not recommended because they lack vision.
  • Permission-based safety: Multi-layer guardrails require your approval for risky actions like deleting files. It asks before doing anything that could mess up your system.
  • Two usage modes: Run it through the regular Hermes CLI or the dedicated command hermes computer use chat.
  • Self-improving: Hermes tests itself and verifies that computer use works, showing its ability to learn and improve.

These features make it a major level up from previous versions, according to Julian.

03How to Set Up Hermes Agent Computer Use

Setting up computer use is straightforward and requires no coding. Here are the steps Julian followed:

  1. Open a terminal and run hermes update to ensure you have the latest version of Hermes.
  2. Install the computer use tool by running hermes computer use install. Alternatively, you can enable the tool set interactively in Hermes.
  3. Grant OS permissions: Go to Accessibility settings and allow terminal to control your computer.
  4. Verify the installation by asking Hermes to test computer use, which it does automatically.

After these steps, you can start using it. Julian notes that this is an experimental feature, so give it access only if you're comfortable. He may even uninstall it after testing because it's very new.

04How to Use It: Real Examples from the Episode

Julian tested computer use with several tasks. Here is what he did and how it worked:

  • Open the Notes app: He asked Hermes to open Notes, and it did instantly using the macOS computer skill.
  • Create a journal note: He prompted Hermes to open Notes and create a new note journaling about ways it could save him time. Hermes generated 10 personalized ideas, like personal knowledge capture, and put them in a new note. It worked perfectly.
  • Organize Obsidian: He asked Hermes to organize his Obsidian knowledge base, add emojis, titles, and improve the graph. This task was more complex and slowed down. It asked for permissions and struggled, so Julian stopped it.
  • Talk to itself in Terminal: He asked Hermes to open Terminal and have a conversation with itself. It managed to open a new Terminal window, start Hermes, and write a greeting.

The best results came from basic, well-defined tasks like note creation.

05Results, Tests, and Comparisons

Julian's live tests showed mixed but promising results. The simplest tasks were incredibly fast and smooth. Opening Notes and creating a journal note with 10 personalized ideas took seconds and worked flawlessly.

He said he had never seen computer use work that quickly before.

However, the complex task of organizing Obsidian exposed limitations. Hermes moved the mouse, typed in the background, and asked for permissions, but it was slow and had trouble. Julian admitted it seemed a bit too slow for complex jobs.

He also noted that using Codex OAuth via an API hit usage limits quickly, so he switched to Kimi K2.6. The screenshots used a lot of tokens, which is a known issue.

Overall, for basic tasks, it's a major level up. For massive or complicated tasks, it still needs work.

06Tips, Limits, and Mistakes to Avoid

Here are some important tips and warnings from the episode:

  • Use a goal instead of one-off tasks: Give Hermes a full goal like "finalize emails, summarize what they need, draft a reply, put it in drafts folder" rather than just "write an email." This gets better results.
  • Act as CEO, not operator: Let the agent do the work while you oversee and approve big decisions. Don't micromanage clicks.
  • Pick a model with vision: Avoid text-only models because they can't see screenshots, which are essential for computer use.
  • Expect slower performance on complex tasks: Julian found complex tasks like organizing Obsidian to be slow and prone to issues. Stick to simple tasks for best results.
  • Watch token usage: Screenshots consume many tokens. Use models like Claude or free options like Qwen3 to manage costs.
  • Always grant permissions carefully: The feature asks before risky actions, but it's experimental—don't give it access to anything you're not comfortable with.

These tips help you get the most out of computer use while avoiding common pitfalls.

07Who It's For and Why It Matters

Hermes Agent computer use is for anyone who wants to automate repetitive computer tasks without technical expertise. Julian emphasizes that you don't need to be a coder or developer—if you can type a sentence, you can give an agent a task. It's especially useful for people who spend time on admin, email, note-taking, and file management.

Why it matters is that it changes how you use AI. Instead of copying and pasting back and forth, the AI does the actual clicking and typing for you. This frees you up to focus on work that grows your business or career.

Julian uses it for AISO, which stands for AI SEO, and finds it powerful for that.

If you're worried about being late to AI, note that computer use is brand new, so you're early. This gives you an advantage to learn and implement it before others.