Jacob Wiseberg

After spending months pushing my agent workflows through hosted APIs, I finally sat down and wired up Hermes against a local Ollama instance. It’s not a full replacement yet, but it’s close enough that I think about my API bill very differently now.

Why bother running local at all

I’ve written before about treating compute as a commodity and not wanting to get locked into a single provider. That line of thinking eventually runs into its logical endpoint: why pay per token at all for the agent work that doesn’t need a frontier model? A lot of what I ask an agent to do is repetitive and low-stakes. Reformatting a config, grepping through a monorepo, summarizing a log file, drafting a commit message. None of that needs the smartest model on earth, and all of it adds up on a bill if I run it through a hosted endpoint every time.

My Beelink SER8 with 64 GB of RAM has been sitting there mostly as network storage and a document processor for my fiancée and me. It turns out 64 GB of unified-ish memory is more than enough to host a handful of quantized models comfortably, so it felt wasteful not to put it to work.

The setup

Ollama handles the model serving side. It’s genuinely the easiest way I’ve found to pull a model and get an OpenAI-compatible endpoint running without fighting with CUDA drivers or quantization formats myself.

curl -fsSL https://ollama.com/install.sh | sh
ollama pull qwen2.5-coder:32b
ollama pull llama3.1:8b

Hermes, on the agent side, just needs to be told where to point. Since Ollama exposes an OpenAI-compatible API on port 11434, this ends up being a provider config change rather than anything exotic:

providers:
  ollama:
    base_url: http://beelink.local:11434/v1
    api_key: not-needed
    models:
      - qwen2.5-coder:32b
      - llama3.1:8b

Because the Beelink is already on my Tailscale network from the WSL setup I wrote about earlier, I can point Hermes at it from my MacBook over the tailnet without opening anything to the open internet. The laptop asks, the Beelink answers, and nothing involving my actual code or config files leaves my own hardware.

Where this actually works well

I route the small, mechanical tasks to the local model and keep anything that needs real reasoning on a hosted frontier model. Hermes makes this trivial because the routing is just a model selection away, not a different tool or workflow. In practice:

• File triage, log summarization, and “what changed in this diff” questions go to the local 8B model and come back fast.
• Anything touching multiple files with real logic, or anything I’d actually review line by line, still goes to a hosted model.
• Long-running background agent tasks that just need to babysit a build or watch for a pattern match run locally all day without me thinking about cost.

The honest tradeoff is latency and ceiling. A local 32B model is not going to touch a frontier model’s reasoning on a gnarly bug. But it is free, private, and always available, and that’s a different kind of valuable than being the smartest model in the room.

What surprised me

I expected the local setup to feel like a toy. It doesn’t. The combination of Hermes’s agent loop and a reasonably capable local coder model handles a surprising amount of my day-to-day terminal busywork, and it does it without a single token leaving my network. Given how much of my current work involves AI tooling that I can’t fully audit, having a lane that is entirely under my control is worth more to me than I expected going in.

I’m still tuning which tasks go where, and I expect that list to shift as local models keep getting better on the same hardware I already own. But for the first time, “local agent work” feels like a real option and not just a privacy compromise.