Local LLMs & Voice Activity Detection: building a privacy-first agent operating system

Why local, and why now

Cloud LLM APIs are the fastest path to a working prototype, but they force a tradeoff most idea-tools shouldn't make: every thought you capture leaves the device. For anyone building an agent operating system — a persistent, personal AI that watches, remembers, and connects ideas — that tradeoff shows up everywhere. Local LLMs move inference back onto the user's machine so raw notes, half-formed songs, and confidential product ideas never touch a third-party server.

The two runtimes that matter today are Ollama and llama.cpp. Both run quantized models on CPU + GPU; they take very different approaches to developer experience.

Ollama vs llama.cpp: which to pick

Ollama is a batteries-included runtime with a local HTTP API, model registry, and one-line install. If you want to type ollama run llama3.1 and call POST /api/generate from your app five minutes later, this is it. Ideal for prototypes, desktop apps, and teams that don't want to compile anything.

llama.cpp is the C/C++ inference engine most local runtimes (including Ollama) build on. Choose it directly when you need finer control: custom quantization, embedded builds, tighter memory budgets, or shipping a self-contained binary with no separate daemon. It's also the right pick when you want to run on constrained hardware where every megabyte counts.

Wiring a local LLM into an idea capture loop

The minimum viable loop looks like this: a voice or text input arrives, an intent classifier decides what kind of idea it is (song lyric, product spec, journal entry, etc.), the model rewrites/expands it, and the result lands in a local store. With Ollama the HTTP call is trivial:

// call a local Ollama model — no data leaves the machine
const res = await fetch("http://localhost:11434/api/generate", {
  method: "POST",
  body: JSON.stringify({
    model: "llama3.1",
    prompt: "Classify this idea and suggest 3 next steps:\n" + text,
    stream: false,
  }),
});
const { response } = await res.json();

For llama.cpp the equivalent is either the built-in HTTP server llama-server or a native binding (node-llama-cpp, Python bindings, etc.). The important property either way: no outbound network call at inference time.

Voice activity detection: the missing piece

Local LLMs solve the "where does the text go" problem. Voice activity detection solves "when do we start and stop listening." Push-to-talk is fine for a chat window; it falls apart when you want an ambient capture tool that snags a lyric while you're pacing, or a research assistant that records a call in the background.

Practical VAD options in 2026:

• Silero VAD — tiny ONNX model (~1 MB), runs in the browser via onnxruntime-web or on-device via Python/Node. Great default.
• WebRTC VAD — the classic, deterministic, no ML dependency. Higher false positives, but zero footprint.
• picovoice Cobra — commercial, extremely accurate, tight license terms. Consider when accuracy is the product.

Wire the VAD upstream of your speech-to-text (Whisper.cpp is the natural pairing for a fully local stack) and downstream to the local LLM. The full path stays on-device:

microphone
  → VAD (Silero / WebRTC)
      → segment on speech-start / speech-end
      → Whisper.cpp transcribe
          → local LLM (Ollama / llama.cpp)
              → local store

Framing this as a privacy-first agent OS

Once inference, transcription, and detection are all local, you're no longer building "an app with an LLM inside." You're building a small operating system for ideas — one that watches, listens, and remembers without leaking anything. That's the framing that resonates with developers looking for an alternative to cloud agent stacks: not "we call OpenAI cheaper," but "we don't call anyone at all."

Evolve-Ů is built around that framing: a personal agent surface that treats every fragment — a voice memo, a lyric, a beat, a research note — as something to preserve, not upload. Local LLMs and VAD are what make that promise real.

Where to start today

1. Install Ollama and pull a mid-size model (Llama 3.1 8B or Mistral 7B).
2. Drop Silero VAD into your capture pipeline; log speech-start / speech-end events.
3. Add Whisper.cpp for on-device transcription.
4. Send the transcript to the local model and store the result in SQLite / IndexedDB.
5. Only reach for the cloud when a specific task (image gen, long-context research) genuinely needs it — and make it explicit in the UI.

← Try the Evolve-Ů idea OS