Offline Mode

Ante runs local GGUF models through a natively managed llama.cpp engine. No API key, no account: point Ante at a model file and the full agent loop runs on your hardware. Once a model is on disk, inference needs no network connection at all.

For a fully disconnected run, set ANTE_TELEMETRY=off as well. This disables application metrics and log export independently of local inference; see Telemetry.

info

Offline mode is under active development. Expect rough edges and rapid iteration.

How it works

When you select offline mode, Ante:

1. Checks for llama.cpp and offers to install or upgrade it: a pinned, checksum-verified official build matched to your hardware. macOS uses Metal; Linux chooses the strongest compatible CPU/Vulkan feature tier, prefers a CUDA build matched to the NVIDIA compute capability, and falls back to Vulkan when the pinned release has no compatible CUDA artifact
2. Discovers GGUF models on your system
3. Detects llama servers already running on local ports
4. Estimates memory requirements from model size and context window
5. Starts and supervises one shared llama.cpp router with llama serve; switching managed models loads and unloads workers without restarting that process

Prefer your own llama.cpp build, extra server flags, or a server you started yourself? See Custom Engines.

Setting up

1. Launch Ante and open offline mode — Start Ante normally and use the offline mode selector in the TUI:

   ante
   
   # In the TUI
   /offline-mode

2. Install llama.cpp — If not installed, Ante will prompt you to install it automatically to ~/.ante/llama.cpp. When the pinned version changes in a new Ante release, Ante offers an upgrade. To do the same without the TUI, run ante offline install: it installs the pinned build, replaces a mismatched one, and is a no-op when already current.

3. Select a model — Choose from:

   • Verified models — curated models downloaded from Hugging Face
   • Local models — GGUF files already on your system (auto-discovered)
   • Running servers — attach to a llama server already running on a local port or reachable endpoint

4. Or load from the command line — open the TUI on a local model without visiting the model picker:

   ante --offline-model /path/to/model.gguf

   Add -p "your prompt here" for a one-shot headless task. --offline-model [MODEL] also accepts a discovered filename or verified model name/URL. It conflicts with --model and --provider.

Configure without the TUI

ante offline install
ante offline status
ante offline list
ante offline use Qwen3.5-9B
ante --offline-model

status and list support --json. use saves the default without starting or downloading anything; verified models download on first load. Use the reference field from ante offline list --json when you need an exact model reference for a script. Bare --offline-model starts the saved default and fails with setup guidance if none is set.

Once a model is loaded, it is registered as the local provider in the catalog, so you can also switch to it with /providers or target it with --provider local. The entry exists only while the model is ready: unloading the model or stopping the server removes local from the catalog. Starting with a saved local selection and no server running gates the session with a notice (run /offline-mode to start or attach one); headless runs fail fast with the same message. See mixing local and frontier.

Scripted and server usage

Headless (single session)

Pass --offline-model to run a one-shot prompt against a local GGUF file — no TUI required:

ante --offline-model ~/.ante/models/Qwen3.5-9B-Q4_K_M.gguf -p "Explain this code"

Ante starts the llama serve router, loads the model, waits for it to become ready, runs the session, then shuts the router down automatically.

Headless runs never prompt to install the engine. On a machine that doesn't have it yet, the run fails fast and tells you to run ante offline install first — useful to bake into scripts and CI images.

Serve mode (WebSocket server)

When running ante serve, you can load a model once and share it across all connecting clients:

ANTE_SERVE_TOKEN="your-secret-token" ante serve --ws 127.0.0.1:9000 --offline-model

This example uses the default saved with ante offline use above. WebSocket clients must send the same token in the upgrade request as Authorization: Bearer <token>; see Serve mode.

The llama serve router starts and loads the model before Ante accepts connections, and each WebSocket client uses the same running process. The router shuts down cleanly on Ctrl+C or SIGTERM.

Loading progress

When a model is downloading or being loaded into memory, Ante shows a real-time progress bar in the TUI with status messages:

• Download phase — progress tracks bytes received against the total file size
• Metadata — model metadata is being read from the GGUF file
• Tensor loading — weights are being loaded into RAM/VRAM
• GPU offload — layers are being transferred to the GPU (if available)

Launches using --offline-model report download and load progress on stderr, keeping headless results and served protocol streams on stdout separate.

Model discovery

Ante automatically scans the following directories for GGUF model files:

Directory	Description
~/.ante/models	Default model directory (configurable)
~/.cache/llama.cpp	llama.cpp cache
~/.cache/huggingface/hub	Hugging Face cache
~/.llama/models	Common llama model directory

For a sharded GGUF, select the first shard (*-00001-of-*.gguf) and keep every shard in the same directory. Discovery, direct paths, and saved defaults inspect the full set and total size; missing shards or a non-first-shard reference are rejected. The model must support at least 32,768 context tokens, even though the load window can be set lower.

Model preferences

Per-model settings, saved automatically in the offline config:

Setting	Description
context_window	Load context window: 8,192 tokens up to the model maximum; default cap is 32,768
thinking	Enable/disable thinking; stored as thinking_enabled in the config
temperature	Sampling temperature (0–2); unset uses the model default
extra_args	Extra llama.cpp model options, translated into the shared router's per-model preset — see Custom Engines

Save preferences from the CLI for the next TUI, headless, or serve load:

ante offline use Qwen3.5-9B --context 16384 --thinking off --temperature 0.7
ante offline use Qwen3.5-9B --args "--threads 12"
ante offline use Qwen3.5-9B --temperature default --args ""

Each option replaces only its own value and preserves the others. --temperature default clears the temperature override; --args "" clears extra options. Every use prints Loads with:, including when no preference flags are supplied.

Before saving, --context and --temperature are range-checked. --args receives basic syntax checks and rejects reserved or removed options; other invalid options or values may only fail when loading a model.

In the config, preferences are keyed by the local file's absolute path or the verified model's download URL, available as reference in list --json.

Vision models

If a multimodal projector file (mmproj-*.gguf) sits next to the model file, Ante attaches it automatically at launch, and the model accepts images in chat.

Memory considerations

Ante estimates memory usage based on model file size, KV cache (scales with context window), and shard count.

tip

For large models, lower the load window with ante offline use <MODEL> --context 8192 to reduce memory use. 8,192 tokens is the minimum setting; 32,768 is the default cap. The memory-fit estimate in list uses a 32K context, not the per-model saved load window.

Server management

Shortcut	Action
Ctrl+E	Stop the currently connected server
Ctrl+G	Toggle the inference log panel
Ctrl+K / Ctrl+J	Scroll the log panel up/down while it is open

When exiting Ante with a server running, you'll be prompted:

• s — Stop the server and exit
• k — Keep the server running and exit (prints PID)
• Esc — Cancel and stay in Ante

A kept router is reused by the next offline launch. ante offline status shows its PID, endpoint, and current model on the Router: line. Run ante offline stop to stop this Ante home's router, including one still used by another session; all sessions sharing it are affected.

Configuration reference

All offline mode configuration is stored in ~/.ante/offline-config.json:

{
  "version": "1.0.0",
  "model_directory": "~/.ante/models",
  "port": 8080,
  "default_model": "/models/model.gguf",
  "model_preferences": {
    "/models/model.gguf": {
      "model_id": "/models/model.gguf",
      "context_window": 32768,
      "thinking_enabled": true,
      "temperature": 0.7,
      "extra_args": "--threads 12"
    }
  }
}

Field	Description	Default
model_directory	Where to look for local GGUF models	~/.ante/models
port	Starting port for the managed llama.cpp router or attached server	8080
default_model	Model started by bare --offline-model, set by ante offline use	—
model_preferences	Per-model settings (see above)	—
llama_cpp	Engine install details, binary paths, and server defaults — see Custom Engines	managed by Ante

Per-model preferences are read during discovery. After editing offline-config.json, close and reopen /offline-mode to load the new values.

Environment variables

Variable	Description
ANTE_OFFLINE_CONTEXT	Override the initial context window in tokens when a model has no saved preferences (default cap: 32K). To change an existing preference, use ante offline use <MODEL> --context <TOKENS>