How to Build Your Own Agent Operating System

01What an Agent Operating System Is

An agent operating system is a platform where you can run teams of AI agents that work together on tasks. In this episode, Julian Goldie explains how you can build your own system using tools like Hermes, Claude, and Obsidian. This setup lets agents handle everything from daily SEO updates to creating fully edited videos without you touching the work.

The idea is to give your agents a clear workflow and a way to check their own quality. For example, you can create a team of agents where one agent acts as a judge. That judge scores the work of the other agents and tells them to keep improving until the output is good enough.

This makes the system reliable and scalable.

02Main Features of the Agent OS Setup

Julian shows several key features of an agent operating system. Here are the most important ones:

• Scheduled tasks: You can assign a task to an agent to run every day or week, like optimizing meta tags or uploading a blog post. The manage section lets you see and delete scheduled tasks easily.
• Kanban board: Hermes has a native Kanban board for organizing agents and tasks. You can create multiple boards for different projects and keep everything separate.
• Judge agent: A separate agent that reviews the work of the team. It scores the output, lists critical issues, and makes the team iterate until the quality meets your standards.
• Obsidian memory: You can connect your agents to an Obsidian vault. Agents automatically update the vault with context and notes, so your system gets smarter over time.
• Voice activation: With Hermes Jarvis, you can talk to your agents and see the full conversation history.

03How to Set Up Your Agent Operating System

Follow these steps to get started with your own agent operating system, based on what Julian explains:

1. Choose your main tool: Use Hermes for teams of agents because it has a built-in Kanban board. If you need multiple teams across different tools like OpenClaw and Claude, consider Paperclip instead.
2. Create agent profiles: Set up profiles for each team member. For example, you might have a video creator agent, a script writer agent, and a judge agent.
3. Add a knowledge base: Connect Obsidian to your agents. This gives them context and memory. Your agents can automatically update the vault so you don't have to organize it manually.
4. Set up scheduled tasks: Go to the manage section and the cron job area. Assign a task to an agent with instructions to improve the workflow each time it runs.
5. Use the Kanban board: Assign a task to a board. The system delegates it to sub-agents automatically. Make sure one agent is a judge who scores the work and requests improvements.
6. Test and iterate: Start with a simple task like SEO optimization. Let the agents run and check the results. Adjust the workflow based on what the judge agent reports.

04How to Use the Agent OS with Examples

Julian walks through several examples of using an agent operating system. For a video creation task, you would create a team with a judge agent. The judge looks at the finished video, scores it, and lists critical issues.

If the video is not good enough, the team keeps iterating until the score improves. Julian shows a final video with music and editing that he says looked better than what his own team could produce.

For SEO optimization, you assign a scheduled task to an agent. You tell the agent to run a workflow daily, like optimizing meta tags and uploading one blog post per day. The agent also updates its own workflow based on what works and what doesn't.

You can manage all scheduled tasks in the cron job section.

Another example is comparing free vs paid agents. One user set up twin agents for every task: one free agent and one paid agent. The system compares the quality of each output.

This helps you decide if the paid agent is worth the cost and gives you a benchmark for all your tasks.

05Results, Tests and Comparisons Shown in the Episode

Julian shares several results from testing the agent operating system. He says that Fusion, a new team of agents, outperforms Opus 4.8 by a long way on benchmarks. Fusion gives you Fable 5-level intelligence without waiting for Fable 5 to come back.

Julian tested this personally and confirms the results.

For video creation, the final product looked great with music and full editing, all done by the agent team from a single prompt. Julian says it was probably better than what his own team could do. The judge agent scored the team and made them iterate until the quality was high.

Julian also compares Hermes to OpenClaw for managing teams. He says OpenClaw is not designed for teams of agents and works best as a single instance. Hermes, with its native Kanban board, is much smoother for delegation and control.

Paperclip is better if you need to combine multiple types of agents across different tools.

06Tips, Limits and Mistakes to Avoid

Julian gives several tips based on questions from the community. First, if you use Hermes for coding tasks, you might get bad results. Instead, use Claude for coding work because it handles complex code better.

Hermes is better for smaller scheduled tasks or admin work. If your agent is not following instructions, check which model you plugged into it—you can switch to minimax m3 or grok for better performance.

A common mistake is not adding a memory system. Julian says adding Obsidian memory improves everything. Agents get better context and can automatically update your second brain.

Without it, your agents lack context and repeat work.

Another limit: the infinite context engine (OMI) does not work natively on Windows. If you use Windows, you need a wearable device or an open-source alternative that works on Windows. You can still take context from that tool and plug it into Obsidian.

Finally, if you use OpenClaw with 25 agents, Julian recommends switching to Hermes for better delegation. Hermes has a migrate option on onboarding to move your setup over smoothly.

07Who the Agent Operating System Is For and Why It Matters

This agent operating system is for anyone who wants to automate complex, ongoing tasks with AI. Julian says it works for business owners who want to scale their marketing agency, like Kyle's setup with a dispatcher and sub-agents, or for personal projects like Benjamin's philosophy voice agent. Even non-technical people can build with this system because it is highly customizable.

Why it matters: you can give agents long-term responsibility for tasks like SEO or content creation. They run on schedules, check their own quality, and improve over time. This saves you hours of manual work and lets you focus on high-level decisions.

Julian's community of over 3,600 business owners and 182 pages of testimonials shows that real people get real results with this approach.