tomasz-syn-grzegorza
359afb3a59
Infrastructure configs for GMKtec K11 (Docker, vLLM, LocalAI, ComfyUI, control-plane, gpu-fan agent, Server UI with CLI/file explorer/GPU fan curve). Co-authored-by: Cursor <cursoragent@cursor.com>
19 lines
409 B
Bash
19 lines
409 B
Bash
# Template for a new vLLM profile
|
|
# Copy: cp profiles/_template.env profiles/my-model.env
|
|
# GGUF files from lmstudio-community require stacks/llamacpp (future), not vLLM.
|
|
|
|
DATA_ROOT=/data
|
|
|
|
VLLM_MODEL=
|
|
SERVED_MODEL_NAME=my-model
|
|
QUANTIZATION=awq
|
|
VLLM_EXTRA_ARGS=--language-model-only --enforce-eager
|
|
|
|
MAX_MODEL_LEN=131072
|
|
MAX_NUM_SEQS=1
|
|
GPU_MEMORY_UTILIZATION=0.95
|
|
KV_CACHE_DTYPE=fp8
|
|
|
|
HF_TOKEN=
|
|
VLLM_PORT=8000
|