tomasz-syn-grzegorza
359afb3a59
Infrastructure configs for GMKtec K11 (Docker, vLLM, LocalAI, ComfyUI, control-plane, gpu-fan agent, Server UI with CLI/file explorer/GPU fan curve). Co-authored-by: Cursor <cursoragent@cursor.com>
66 lines
2.1 KiB
Markdown
66 lines
2.1 KiB
Markdown
# llama.cpp stack (planned)
|
|
|
|
Placeholder for a future Docker stack serving **GGUF** models from `models.catalog.yaml` (lmstudio-community Q4).
|
|
|
|
vLLM on port **8000** stays separate; this stack is intended for port **8001**.
|
|
|
|
## Why a separate host
|
|
|
|
| Format | Runtime | Notes |
|
|
|--------|---------|-------|
|
|
| AWQ / safetensors (HF) | vLLM (`stacks/vllm/`) | Interim Q4 equivalent |
|
|
| GGUF Q4_K_M / Q4_0 | **llama.cpp** (this stack) | Native K/V cache `q4_0` like LM Studio |
|
|
|
|
Standard `vllm/vllm-openai` does **not** load `.gguf` files. Links in the catalog point here.
|
|
|
|
## Disk layout (created by vLLM scripts)
|
|
|
|
```
|
|
/data/apps/gguf/
|
|
├── qwen3.6-27b/Qwen3.6-27B-Q4_K_M.gguf
|
|
└── gemma-4-12b/
|
|
├── gemma-4-12B-it-QAT-Q4_0.gguf
|
|
└── mmproj-gemma-4-12B-it-QAT-BF16.gguf
|
|
```
|
|
|
|
Download on demand from vLLM stack:
|
|
|
|
```bash
|
|
cd stacks/vllm
|
|
./scripts/download-model.sh qwen3.6-27b-q4-gguf
|
|
./scripts/download-model.sh gemma-4-12b-q4-gguf
|
|
```
|
|
|
|
## Planned configuration
|
|
|
|
- **Binary:** `llama-server` (llama.cpp)
|
|
- **Router mode:** `--models-dir /data/apps/gguf` — multiple models on disk, one loaded per request via API `"model"` field
|
|
- **K/V cache:** `--cache-type-k q4_0 --cache-type-v q4_0` (LM Studio parity)
|
|
- **Port:** `8001` (vLLM remains on `8000`)
|
|
- **GPU:** RTX 3090 Ti only (`CUDA_VISIBLE_DEVICES=0`)
|
|
|
|
Example (not wired yet):
|
|
|
|
```bash
|
|
llama-server \
|
|
--host 0.0.0.0 \
|
|
--port 8001 \
|
|
--models-dir /data/apps/gguf \
|
|
--ctx-size 131072 \
|
|
--cache-type-k q4_0 \
|
|
--cache-type-v q4_0 \
|
|
--n-gpu-layers 999
|
|
```
|
|
|
|
## Model switching
|
|
|
|
Unlike vLLM (one model per container restart), llama.cpp router mode can keep several GGUF files on disk and select by name in the OpenAI-compatible API without restarting the whole service.
|
|
|
|
See: [Model management in llama.cpp](https://huggingface.co/blog/ggml-org/model-management-in-llamacpp)
|
|
|
|
## Status
|
|
|
|
- Catalog entries: `stacks/vllm/models.catalog.yaml` (`runtime: llamacpp`)
|
|
- Docker compose: **not implemented** — this README only
|
|
- When ready: add `docker-compose.yml`, profile, and `scripts/start.sh` mirroring the vLLM stack pattern
|