Custom Engines

Offline mode works out of the box with the engine Ante installs and supervises. You can replace the binary, customize model and GPU options, or attach a server you run yourself.

Use your own llama.cpp build

Ante resolves the engine binary in this order:

1. A binary path pinned in offline-config.json (below)
2. The managed ~/.ante/llama.cpp/current/llama multiplexer

Ante does not discover an arbitrary PATH installation for managed mode. To use your own build, point llama_cpp.binaries.server at its llama multiplexer in ~/.ante/offline-config.json:

{
  "llama_cpp": {
    "binaries": {
      "server": "/opt/llama.cpp/build/bin/llama"
    }
  }
}

The file must exist and support llama serve. Older saved paths ending in llama-server automatically resolve to a sibling llama, so existing managed installs migrate without a config rewrite. If that sibling multiplexer does not exist, install the supported engine with ante offline install or update the custom path.

note

Ante normally writes these fields itself during installation. If you point them at a custom build, the version check may offer an "upgrade" back to the pinned build — decline it to keep yours.

Tune managed inference

Two knobs shape managed inference:

Per-model extra flags. Set extra_args in a model's preferences to add llama.cpp model options. The string is parsed shell-style (quoting works) and translated into that model's router preset, so a supplied value overrides Ante's model default without restarting the shared server.

Use the CLI to set these without writing the preference key by hand:

ante offline use Qwen3.5-9B --args "--threads 12 --load-mode mmap --cache-type-k q8_0"

When editing the config directly, use the model's absolute path or verified download URL as the key (reference in ante offline list --json); the path below is an example.

{
  "model_preferences": {
    "/models/Qwen3.6-27B-Q4_K_M.gguf": {
      "model_id": "/models/Qwen3.6-27B-Q4_K_M.gguf",
      "context_window": 32768,
      "thinking_enabled": true,
      "extra_args": "--threads 12 --load-mode mmap --cache-type-k q8_0"
    }
  }
}

Use options supported by your installed llama.cpp build. ante offline use checks basic syntax and rejects reserved options such as --host, --port, --model, and --alias. Other invalid options or values may only fail when loading a model.

Ante v0.2.12 pins llama.cpp to b11540. The old loading flags --mlock, --mmap, --no-mmap, --dio / --direct-io, and --ndio / --no-direct-io are rejected. Replace them in saved extra_args with --load-mode and a mode supported by your engine, such as --load-mode mmap above.

GPU offload. llama_cpp.server_defaults.gpu_layers controls the -ngl value. The default -1 offloads all layers to the GPU; lower it if your VRAM is tight:

{
  "llama_cpp": {
    "server_defaults": { "gpu_layers": 24 }
  }
}

Attach to a running server

You can run llama-server entirely yourself — any build, any flags — and let Ante connect to it as a client. Ante scans local ports 8080-8179 and also supports manual host:port attach from /offline-mode with Ctrl+A.

See Existing Servers for endpoint input, Docker examples, detach behavior, and the remote-host data-flow caveat.

A different engine entirely

Anything with an OpenAI-compatible endpoint — Ollama, vLLM, LM Studio, a GPU box on your LAN — can serve Ante as a regular catalog provider. That route gives you provider mechanics only (model picker, wire protocol); the lifecycle management, memory estimation, and load progress on this page are specific to the managed llama.cpp path. See add a local provider.