Custom Engines

Offline mode works out of the box with the engine Ante installs and supervises. Every layer of that is replaceable: the binary, the launch flags, or the server itself.

Use your own llama.cpp build

Ante resolves the engine binary in this order:

1. The managed install at ~/.ante/llama.cpp/current
2. Binary paths pinned in offline-config.json (below)
3. llama-server on your PATH

So if you already have llama.cpp installed (say via brew install llama.cpp) and skip the managed install, Ante picks it up from PATH. To pin specific binaries explicitly, set both paths under llama_cpp.binaries in ~/.ante/offline-config.json:

{
  "llama_cpp": {
    "binaries": {
      "server": "/opt/llama.cpp/build/bin/llama-server",
      "cli": "/opt/llama.cpp/build/bin/llama-cli"
    }
  }
}

Both files must exist for the override to take effect. The server binary must accept standard llama-server flags: Ante launches it with its own arguments (--host 127.0.0.1 --port <port> -c <context> --jinja -ngl <gpu-layers>, plus per-model flags such as --temp and --mmproj).

note

Ante normally writes these fields itself during installation. If you point them at a custom build, the version check may offer an "upgrade" back to the pinned build — decline it to keep yours.

Tune the launch command

Two knobs shape the command Ante runs:

Per-model extra flags. Set extra_args in a model's preferences to append any llama-server flags. The string is parsed shell-style (quoting works) and appended after Ante's own flags, so on conflict your value wins:

{
  "model_preferences": {
    "Qwen3.6-27B-Q4_K_M.gguf": {
      "model_id": "Qwen3.6-27B-Q4_K_M.gguf",
      "extra_args": "--threads 12 --no-mmap --cache-type-k q8_0"
    }
  }
}

GPU offload. llama_cpp.server_defaults.gpu_layers controls the -ngl value. The default -1 offloads all layers to the GPU; lower it if your VRAM is tight:

{
  "llama_cpp": {
    "server_defaults": { "gpu_layers": 24 }
  }
}

Attach to a running server

You can run llama-server entirely yourself — any build, any flags — and let Ante connect to it as a client. Ante scans local ports 8080-8179 and also supports manual host:port attach from /offline-mode with Ctrl+A.

See Existing Servers for endpoint input, Docker examples, detach behavior, and the remote-host data-flow caveat.

A different engine entirely

Anything with an OpenAI-compatible endpoint — Ollama, vLLM, LM Studio, a GPU box on your LAN — can serve Ante as a regular catalog provider. That route gives you provider mechanics only (model picker, wire protocol); the lifecycle management, memory estimation, and load progress on this page are specific to the managed llama.cpp path. See add a local provider.