# llama.cpp stack (planned)

Placeholder for a future Docker stack serving **GGUF** models from `models.catalog.yaml` (lmstudio-community Q4).

vLLM on port **8000** stays separate; this stack is intended for port **8001**.

## Why a separate host

| Format | Runtime | Notes |
|--------|---------|-------|
| AWQ / safetensors (HF) | vLLM (`stacks/vllm/`) | Interim Q4 equivalent |
| GGUF Q4_K_M / Q4_0 | **llama.cpp** (this stack) | Native K/V cache `q4_0` like LM Studio |

Standard `vllm/vllm-openai` does **not** load `.gguf` files. Links in the catalog point here.

## Disk layout (created by vLLM scripts)

```
/data/apps/gguf/
├── qwen3.6-27b/Qwen3.6-27B-Q4_K_M.gguf
└── gemma-4-12b/
    ├── gemma-4-12B-it-QAT-Q4_0.gguf
    └── mmproj-gemma-4-12B-it-QAT-BF16.gguf
```

Download on demand from vLLM stack:

```bash
cd stacks/vllm
./scripts/download-model.sh qwen3.6-27b-q4-gguf
./scripts/download-model.sh gemma-4-12b-q4-gguf
```

## Planned configuration

- **Binary:** `llama-server` (llama.cpp)
- **Router mode:** `--models-dir /data/apps/gguf` — multiple models on disk, one loaded per request via API `"model"` field
- **K/V cache:** `--cache-type-k q4_0 --cache-type-v q4_0` (LM Studio parity)
- **Port:** `8001` (vLLM remains on `8000`)
- **GPU:** RTX 3090 Ti only (`CUDA_VISIBLE_DEVICES=0`)

Example (not wired yet):

```bash
llama-server \
  --host 0.0.0.0 \
  --port 8001 \
  --models-dir /data/apps/gguf \
  --ctx-size 131072 \
  --cache-type-k q4_0 \
  --cache-type-v q4_0 \
  --n-gpu-layers 999
```

## Model switching

Unlike vLLM (one model per container restart), llama.cpp router mode can keep several GGUF files on disk and select by name in the OpenAI-compatible API without restarting the whole service.

See: [Model management in llama.cpp](https://huggingface.co/blog/ggml-org/model-management-in-llamacpp)

## Status

- Catalog entries: `stacks/vllm/models.catalog.yaml` (`runtime: llamacpp`)
- Docker compose: **not implemented** — this README only
- When ready: add `docker-compose.yml`, profile, and `scripts/start.sh` mirroring the vLLM stack pattern