Files

T

History

tomasz-syn-grzegorza 359afb3a59 Initial import: bare-metal stacks, Server UI, GPU fan, tutorials.

Infrastructure configs for GMKtec K11 (Docker, vLLM, LocalAI, ComfyUI,
control-plane, gpu-fan agent, Server UI with CLI/file explorer/GPU fan curve).

Co-authored-by: Cursor <cursoragent@cursor.com>

2026-07-05 12:02:04 +00:00

README.md

Initial import: bare-metal stacks, Server UI, GPU fan, tutorials.

2026-07-05 12:02:04 +00:00

README.md

llama.cpp stack (planned)

Placeholder for a future Docker stack serving GGUF models from models.catalog.yaml (lmstudio-community Q4).

vLLM on port 8000 stays separate; this stack is intended for port 8001.

Why a separate host

Format	Runtime	Notes
AWQ / safetensors (HF)	vLLM (`stacks/vllm/`)	Interim Q4 equivalent
GGUF Q4_K_M / Q4_0	llama.cpp (this stack)	Native K/V cache `q4_0` like LM Studio

Standard vllm/vllm-openai does not load .gguf files. Links in the catalog point here.

Disk layout (created by vLLM scripts)

/data/apps/gguf/
├── qwen3.6-27b/Qwen3.6-27B-Q4_K_M.gguf
└── gemma-4-12b/
    ├── gemma-4-12B-it-QAT-Q4_0.gguf
    └── mmproj-gemma-4-12B-it-QAT-BF16.gguf

Download on demand from vLLM stack:

cd stacks/vllm
./scripts/download-model.sh qwen3.6-27b-q4-gguf
./scripts/download-model.sh gemma-4-12b-q4-gguf

Planned configuration

Binary: llama-server (llama.cpp)
Router mode: --models-dir /data/apps/gguf — multiple models on disk, one loaded per request via API "model" field
K/V cache: --cache-type-k q4_0 --cache-type-v q4_0 (LM Studio parity)
Port: 8001 (vLLM remains on 8000)
GPU: RTX 3090 Ti only (CUDA_VISIBLE_DEVICES=0)

Example (not wired yet):

llama-server \
  --host 0.0.0.0 \
  --port 8001 \
  --models-dir /data/apps/gguf \
  --ctx-size 131072 \
  --cache-type-k q4_0 \
  --cache-type-v q4_0 \
  --n-gpu-layers 999

Model switching

Unlike vLLM (one model per container restart), llama.cpp router mode can keep several GGUF files on disk and select by name in the OpenAI-compatible API without restarting the whole service.

See: Model management in llama.cpp

Status

Catalog entries: stacks/vllm/models.catalog.yaml (runtime: llamacpp)
Docker compose: not implemented — this README only
When ready: add docker-compose.yml, profile, and scripts/start.sh mirroring the vLLM stack pattern