Initial import: bare-metal stacks, Server UI, GPU fan, tutorials.

Infrastructure configs for GMKtec K11 (Docker, vLLM, LocalAI, ComfyUI, control-plane, gpu-fan agent, Server UI with CLI/file explorer/GPU fan curve). Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-05 12:02:04 +00:00
commit 359afb3a59
153 changed files with 18169 additions and 0 deletions
@@ -0,0 +1,65 @@
+# llama.cpp stack (planned)
+
+Placeholder for a future Docker stack serving **GGUF** models from `models.catalog.yaml` (lmstudio-community Q4).
+
+vLLM on port **8000** stays separate; this stack is intended for port **8001**.
+
+## Why a separate host
+
+| Format | Runtime | Notes |
+|--------|---------|-------|
+| AWQ / safetensors (HF) | vLLM (`stacks/vllm/`) | Interim Q4 equivalent |
+| GGUF Q4_K_M / Q4_0 | **llama.cpp** (this stack) | Native K/V cache `q4_0` like LM Studio |
+
+Standard `vllm/vllm-openai` does **not** load `.gguf` files. Links in the catalog point here.
+
+## Disk layout (created by vLLM scripts)
+
+```
+/data/apps/gguf/
+├── qwen3.6-27b/Qwen3.6-27B-Q4_K_M.gguf
+└── gemma-4-12b/
+    ├── gemma-4-12B-it-QAT-Q4_0.gguf
+    └── mmproj-gemma-4-12B-it-QAT-BF16.gguf
+```
+
+Download on demand from vLLM stack:
+
+```bash
+cd stacks/vllm
+./scripts/download-model.sh qwen3.6-27b-q4-gguf
+./scripts/download-model.sh gemma-4-12b-q4-gguf
+```
+
+## Planned configuration
+
+- **Binary:** `llama-server` (llama.cpp)
+- **Router mode:** `--models-dir /data/apps/gguf` — multiple models on disk, one loaded per request via API `"model"` field
+- **K/V cache:** `--cache-type-k q4_0 --cache-type-v q4_0` (LM Studio parity)
+- **Port:** `8001` (vLLM remains on `8000`)
+- **GPU:** RTX 3090 Ti only (`CUDA_VISIBLE_DEVICES=0`)
+
+Example (not wired yet):
+
+```bash
+llama-server \
+  --host 0.0.0.0 \
+  --port 8001 \
+  --models-dir /data/apps/gguf \
+  --ctx-size 131072 \
+  --cache-type-k q4_0 \
+  --cache-type-v q4_0 \
+  --n-gpu-layers 999
+```
+
+## Model switching
+
+Unlike vLLM (one model per container restart), llama.cpp router mode can keep several GGUF files on disk and select by name in the OpenAI-compatible API without restarting the whole service.
+
+See: [Model management in llama.cpp](https://huggingface.co/blog/ggml-org/model-management-in-llamacpp)
+
+## Status
+
+- Catalog entries: `stacks/vllm/models.catalog.yaml` (`runtime: llamacpp`)
+- Docker compose: **not implemented** — this README only
+- When ready: add `docker-compose.yml`, profile, and `scripts/start.sh` mirroring the vLLM stack pattern