01What a Jarvis Local AI Assistant Is and How It Works

A jarvis local ai assistant is a voice assistant that runs on your own computer instead of in the cloud. In this episode, Julian Goldie shows his version, called Jarvis, answering the phone, mapping clients, writing new skills, showing a command center and taking over his Mac.

The big idea is that everything stays on your machine. Jarvis listens, thinks and talks back without sending your data to an online service. In the command center, the number for data sent to the cloud reads zero.

Under the hood, Jarvis is built from a few simple parts:

  • Ollama runs the AI model locally.
  • Qwen 2.5 is the language model that does the thinking, along with an embedding model.
  • Whisper is the helper that listens and turns speech into text.
  • Kokoro, an open source voice, speaks the answers out loud.
  • One Node server file ties it all together and runs a page in your browser.

Julian calls this version Jarvis 2.0. It is free to run because it uses open source tools, and it works offline once the models are downloaded. He runs it on a normal laptop, an M4 Mac with 16 gigabytes of memory.

02The 5 Upgrades in Jarvis 2.0

Julian gives Jarvis five upgrades in this episode. Each one is aimed at helping run a business, not just answering trivia questions. Here is the full list.

  1. Phone receptionist. Jarvis answers the phone and books calls into open time slots.
  2. Jarvis City. Jarvis turns a client list into a 3D city, where each tower is a client.
  3. Self-upgrading skills. Jarvis writes brand new skills for himself from a one-sentence request.
  4. Command center. One screen shows clients, leads, reviews, tasks, calendar and machine stats.
  5. Mac takeover. Jarvis takes control of the computer and sets up everything for a meeting.

For each upgrade, Julian shows it working, says the exact thing he asks Jarvis, and explains how it works underneath. That makes it easy to follow along and see what each piece does.

The common thread is that every upgrade uses your own data. The calendar, the client list, the notes in Jarvis's second brain and the activity feed all come from your machine. Nothing needs to be uploaded to make it work.

03How to Set Up a Jarvis Local AI Assistant

The setup is short. Everything downloads once, and then the assistant runs on your own machine. These are the steps Julian describes:

  1. Install Ollama on your computer.
  2. Use Ollama to pull the Qwen 2.5 model.
  3. Pull the embedding model as well.
  4. Set up the two small Python helpers: Whisper for listening and Kokoro for speaking.
  5. Run the one Node server file.
  6. Open the page in your browser and start talking to Jarvis.

Once those pieces are in place, you do not need an internet connection for the assistant to work. The models, the ears and the voice all live on your computer.

You also do not need a special machine. Julian runs it on an M4 Mac with 16 gigabytes of memory, which he calls a normal laptop. The command center even shows live CPU and memory use, so you can see how hard your computer is working.

If you want the exact files, prompts and a short guide for each build, Julian says they are inside the AI Profit Boardroom.

04Upgrade 1: Jarvis Answers the Phone

The episode opens with Jarvis taking a phone call. The caller picks the 3PM slot, and Jarvis books it for tomorrow at 3. The call feels like talking to a real receptionist.

Here is what happens behind the scenes:

  • Whisper listens. Each line of the call has a small tag that shows what Whisper heard.
  • Jarvis checks the calendar. It can see tomorrow's calendar, so it only offers times that are actually open.
  • Kokoro speaks. The open source Kokoro voice reads the answer back to the caller.

This matters because double bookings are a common problem when someone else handles your calls. Since Jarvis reads your real calendar first, it cannot offer a slot you already filled.

The booked call also shows up later. In the command center, tomorrow's calendar shows the call Jarvis booked on the phone, marked as new. It also appears in the activity feed, so you always know what Jarvis did while you were busy.

05Upgrade 2: Jarvis City Maps Your Client List

Julian asks Jarvis to show him Jarvis City, and a 3D city appears, and every tower is one client. The demo has 36 clients.

The city is easy to read at a glance:

  • Taller towers are more active clients.
  • Cyan towers are clients you spoke to recently.
  • Amber towers are clients that are cooling off.
  • Dark red towers are clients who have gone quiet for over a month.

Jarvis then does real work with that picture. It scanned all 36 clients, found the 10 quiet ones and flagged the three coldest. Then it wrote each of those three a personal follow-up.

Each follow-up greets the contact by name, mentions the service Julian's team does for them and offers one useful idea. The drafts are saved in Jarvis's second brain, ready for Julian to check and send. Jarvis does not send them on its own.

The clients in the demo are samples. Julian says that if you point Jarvis at your own client list, your city builds itself.

06Upgrade 3: Jarvis Writes New Skills for Himself

This is the upgrade Julian calls his favorite. He asks Jarvis to write a brand new skill: one that replies to Google reviews in a warm, personal way.

Jarvis comes back with a reply ready to post for a roofing company called Peak Roofing. The reply thanks the customer, mentions that their leak was fixed in one afternoon and thanks them for their patience during a rescheduling. Jarvis points out that the reply thanks the customer, owns the rescheduling and invites them back.

The process works for any job you repeat. You describe it in one sentence, and Jarvis:

  1. Writes the new skill.
  2. Tests it.
  3. Adds it to his toolkit.

Julian gives a few more ideas for skills you could ask for:

  • A proposal writer.
  • A meeting summarizer.
  • A cold email checker.

Because each new skill stays in the toolkit, the assistant gets more useful every week. You are not stuck with the features it shipped with.

07Upgrade 4: The Jarvis Command Center

Julian asks Jarvis to show him everything, and the command center opens. It puts everything on one screen.

Here is what it shows:

  • All 36 clients.
  • Leads from the last 30 days.
  • Reviews and open tasks, counting up from Julian's own data.
  • Top clients by leads.
  • The mix of services the business delivers.
  • Tomorrow's calendar, with the call Jarvis booked marked as new.

An activity feed lists everything Jarvis did today. In the demo, that includes the phone booking, the client follow-ups and the review skill he wrote for himself.

At the bottom are live CPU and memory numbers from the Mac, plus the models loaded right now. This helps you see that the work really is happening on your own computer.

The number Julian likes most is data sent to the cloud: zero. For a business that handles client details, that is a strong reason to run an assistant locally.

08Upgrade 5: Jarvis Takes Over Your Mac

For the last upgrade, Julian has a call coming up with Peak Roofing. Instead of opening 10 things himself, he asks Jarvis in one sentence to prep him for the Peak Roofing call.

Jarvis replies that it is taking the controls. Then it does all of this on its own:

  1. Hides every other app on the Mac.
  2. Reads Peak Roofing's file from its second brain.
  3. Writes an agenda for the call in TextEdit.
  4. Opens tomorrow's schedule and the client dossier in two clean browser windows.
  5. Moves itself to the left and arranges everything side by side.
  6. Starts a 30-minute focus timer.

When it is done, Jarvis confirms that the agenda is written, the schedule is up, the dossier is on screen and the timer is running.

This is where a local AI assistant goes past chat. It does not just tell you what to do. It sets up your screen so you can start the call prepared, from one spoken request.

09Tips, Limits and Who a Jarvis Local AI Assistant Is For

A few things to keep in mind before you build your own:

  • Check drafts before sending. Jarvis saves follow-ups as drafts in its second brain. Read them before they go out.
  • Use your own data. The demo clients are samples. The city and command center only become useful when you point them at your real client list.
  • Expect a one-time download. The models and helpers download once, then everything runs on your machine.
  • Start with repeat jobs. Skills work best for tasks you do over and over, like review replies or meeting summaries.

This kind of assistant is a good fit for small business owners who handle calls, clients and reviews themselves. It saves time on booking, follow-ups and meeting prep, and it keeps client data off the cloud.

Julian sums up Jarvis 2.0 simply: it answers the phone, maps your clients, writes its own skills, shows you everything and runs your Mac, all local. If you want every file, prompt and guide from the episode, Julian says they are inside the AI Profit Boardroom.