tomasz-syn-grzegorza
359afb3a59
Infrastructure configs for GMKtec K11 (Docker, vLLM, LocalAI, ComfyUI, control-plane, gpu-fan agent, Server UI with CLI/file explorer/GPU fan curve). Co-authored-by: Cursor <cursoragent@cursor.com>
153 lines
4.1 KiB
Markdown
153 lines
4.1 KiB
Markdown
# LocalAI stack
|
|
|
|
[LocalAI](https://github.com/mudler/LocalAI) — silnik inference z **wbudowanym UI** (chat) i API kompatybilnym z OpenAI. Obsługuje modele skwantyzowane (GGUF, AWQ, …) przez backendy (llama.cpp, vLLM, …).
|
|
|
|
## Porty
|
|
|
|
| Serwis | Port | URL |
|
|
|--------|------|-----|
|
|
| LocalAI UI + API | **8080** | `http://HOST:8080` |
|
|
| vLLM (osobny stack) | 8000 | tylko API, bez UI |
|
|
|
|
Jeden port — UI i API na tym samym endpoincie.
|
|
|
|
## Jak to działa
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
browser["Przeglądarka"]
|
|
api["curl / OpenAI SDK"]
|
|
localai["LocalAI :8080"]
|
|
gpu["RTX 3090 Ti"]
|
|
models["/data/apps/localai/models"]
|
|
|
|
browser --> localai
|
|
api --> localai
|
|
localai --> gpu
|
|
localai --> models
|
|
```
|
|
|
|
| Element | Opis |
|
|
|---------|------|
|
|
| Obraz | `localai/localai:v4.4.3-gpu-nvidia-cuda-13` |
|
|
| Konfiguracja | `.env` + `docker-compose.yml` |
|
|
| Modele | `/data/apps/localai/models` (puste na start) |
|
|
| Upstream repo | opcjonalnie `upstream/` przez `clone-upstream.sh` |
|
|
|
|
## Struktura
|
|
|
|
```
|
|
stacks/localai/
|
|
├── README.md
|
|
├── docker-compose.yml
|
|
├── .env.example
|
|
├── .gitignore
|
|
├── coding-agent/ # notatki dla agenta (KV cache, STATE)
|
|
├── profiles/ # szablony YAML (KV q8_0)
|
|
├── upstream/ # shallow clone (gitignored)
|
|
└── scripts/
|
|
├── clone-upstream.sh
|
|
├── pull.sh
|
|
├── start.sh
|
|
└── apply-kv-profile.sh
|
|
```
|
|
|
|
Na dysku `/data`:
|
|
|
|
```
|
|
/data/apps/localai/
|
|
├── models/ # GGUF, YAML model configs
|
|
├── backends/ # custom backends
|
|
├── configuration/ # api_keys.json, runtime settings
|
|
├── images/ # generated images
|
|
└── data/ # agents, skills, persistent app data
|
|
```
|
|
|
|
## Workflow (bez modelu)
|
|
|
|
```bash
|
|
cd stacks/localai
|
|
cp .env.example .env
|
|
|
|
# opcjonalnie — referencja YAML z GitHub
|
|
./scripts/clone-upstream.sh
|
|
|
|
# tylko obraz Docker
|
|
./scripts/pull.sh
|
|
|
|
# uruchom (pusty katalog models/)
|
|
./scripts/start.sh
|
|
```
|
|
|
|
Weryfikacja:
|
|
|
|
```bash
|
|
curl -s http://localhost:8080/readyz
|
|
# UI: http://<IP-serwera>:8080
|
|
```
|
|
|
|
## Zmienne `.env`
|
|
|
|
| Zmienna | Opis | Domyślnie |
|
|
|---------|------|-----------|
|
|
| `DATA_ROOT` | Mount dysku danych | `/data` |
|
|
| `LOCALAI_PORT` | Port na hoście | `8080` |
|
|
| `LOCALAI_IMAGE` | Obraz Docker (CUDA 13) | `localai/localai:v4.4.3-gpu-nvidia-cuda-13` |
|
|
| `CUDA_VISIBLE_DEVICES` | GPU | `0` |
|
|
| `DEBUG` | Verbose logs | `false` |
|
|
|
|
## VRAM (24 GB)
|
|
|
|
Compose ustawia `SINGLE_ACTIVE_BACKEND=true` i `PARALLEL_REQUESTS=false` — jeden aktywny backend/model naraz.
|
|
|
|
**Nie uruchamiaj** dużego modelu w vLLM i LocalAI równocześnie na tej samej karcie:
|
|
|
|
```bash
|
|
cd ../vllm && docker compose --profile vllm down
|
|
```
|
|
|
|
## Modele (później)
|
|
|
|
- UI → Model Gallery w przeglądarce
|
|
- CLI w kontenerze: `docker exec -it localai local-ai models install ...`
|
|
- Ręcznie: GGUF + YAML w `/data/apps/localai/models/`
|
|
|
|
GGUF z [`stacks/vllm/models.catalog.yaml`](../vllm/models.catalog.yaml) można skopiować lub podlinkować do `models/`.
|
|
|
|
## KV cache (skwantyzowany q8_0)
|
|
|
|
Domyślnie llama.cpp trzyma KV cache w **f16** — dużo VRAM przy długim kontekście. Ustawienia są **per model** w YAML na `/data`, nie w compose.
|
|
|
|
| Pole | Rekomendacja |
|
|
|------|--------------|
|
|
| `cache_type_k` | `q8_0` |
|
|
| `cache_type_v` | `q8_0` |
|
|
| `flash_attention` | `true` (wymagane przy q8_0 V) |
|
|
| `context_size` | `8192` (start; zwiększ po teście VRAM) |
|
|
|
|
Szablon: [`profiles/gemma-4-12b-q4-kv-q8.yaml.example`](profiles/gemma-4-12b-q4-kv-q8.yaml.example)
|
|
|
|
Zastosowanie na istniejącym modelu:
|
|
|
|
```bash
|
|
./scripts/apply-kv-profile.sh gemma-4-12b-it-qat-q4_0
|
|
docker compose --profile localai restart localai
|
|
```
|
|
|
|
Szczegóły: [`coding-agent/KV-CACHE.md`](coding-agent/KV-CACHE.md)
|
|
|
|
## Zarządzanie
|
|
|
|
```bash
|
|
docker compose --profile localai ps
|
|
docker compose --profile localai logs -f localai
|
|
docker compose --profile localai restart localai
|
|
docker compose --profile localai down
|
|
```
|
|
|
|
## Dokumentacja
|
|
|
|
Tutorial: [manual-tutorial/05-localai-stack.md](../../manual-tutorial/05-localai-stack.md)
|
|
|
|
Upstream: [github.com/mudler/LocalAI](https://github.com/mudler/LocalAI)
|