tomasz-syn-grzegorza/ubuntu-bare-metal
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial import: bare-metal stacks, Server UI, GPU fan, tutorials.

Browse Source 
Infrastructure configs for GMKtec K11 (Docker, vLLM, LocalAI, ComfyUI,
control-plane, gpu-fan agent, Server UI with CLI/file explorer/GPU fan curve).

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
tomasz-syn-grzegorza
2026-07-05 12:02:04 +00:00
commit 359afb3a59
153 changed files with 18169 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
stacks/localai/.env.example
+16
View File
@@ -0,0 +1,16 @@
# Data disk mount point
DATA_ROOT=/data
# LocalAI web UI + OpenAI-compatible API (localhost bind when behind NPMPlus)
LOCALAI_PORT=8070
# Bearer token for /v1/* API
LOCALAI_API_KEY=
# Pinned GPU image for CUDA 13 (RTX 3090 Ti)
LOCALAI_IMAGE=localai/localai:v4.4.3-gpu-nvidia-cuda-13
# Use only the discrete NVIDIA GPU
CUDA_VISIBLE_DEVICES=0
DEBUG=false
stacks/localai/.gitignore
+2
View File
@@ -0,0 +1,2 @@
.env
upstream/
stacks/localai/README.md
+152
View File
@@ -0,0 +1,152 @@
# LocalAI stack
[LocalAI](https://github.com/mudler/LocalAI) — silnik inference z **wbudowanym UI** (chat) i API kompatybilnym z OpenAI. Obsługuje modele skwantyzowane (GGUF, AWQ, …) przez backendy (llama.cpp, vLLM, …).
## Porty
| Serwis | Port | URL |
|--------|------|-----|
| LocalAI UI + API | **8080** | `http://HOST:8080` |
| vLLM (osobny stack) | 8000 | tylko API, bez UI |
Jeden port — UI i API na tym samym endpoincie.
## Jak to działa
```mermaid
flowchart LR
browser["Przeglądarka"]
api["curl / OpenAI SDK"]
localai["LocalAI :8080"]
gpu["RTX 3090 Ti"]
models["/data/apps/localai/models"]
browser --> localai
api --> localai
localai --> gpu
localai --> models
```
| Element | Opis |
|---------|------|
| Obraz | `localai/localai:v4.4.3-gpu-nvidia-cuda-13` |
| Konfiguracja | `.env` + `docker-compose.yml` |
| Modele | `/data/apps/localai/models` (puste na start) |
| Upstream repo | opcjonalnie `upstream/` przez `clone-upstream.sh` |
## Struktura
```
stacks/localai/
├── README.md
├── docker-compose.yml
├── .env.example
├── .gitignore
├── coding-agent/ # notatki dla agenta (KV cache, STATE)
├── profiles/ # szablony YAML (KV q8_0)
├── upstream/ # shallow clone (gitignored)
└── scripts/
├── clone-upstream.sh
├── pull.sh
├── start.sh
└── apply-kv-profile.sh
```
Na dysku `/data`:
```
/data/apps/localai/
├── models/ # GGUF, YAML model configs
├── backends/ # custom backends
├── configuration/ # api_keys.json, runtime settings
├── images/ # generated images
└── data/ # agents, skills, persistent app data
```
## Workflow (bez modelu)
```bash
cd stacks/localai
cp .env.example .env
# opcjonalnie — referencja YAML z GitHub
./scripts/clone-upstream.sh
# tylko obraz Docker
./scripts/pull.sh
# uruchom (pusty katalog models/)
./scripts/start.sh
```
Weryfikacja:
```bash
curl -s http://localhost:8080/readyz
# UI: http://<IP-serwera>:8080
```
## Zmienne `.env`
| Zmienna | Opis | Domyślnie |
|---------|------|-----------|
| `DATA_ROOT` | Mount dysku danych | `/data` |
| `LOCALAI_PORT` | Port na hoście | `8080` |
| `LOCALAI_IMAGE` | Obraz Docker (CUDA 13) | `localai/localai:v4.4.3-gpu-nvidia-cuda-13` |
| `CUDA_VISIBLE_DEVICES` | GPU | `0` |
| `DEBUG` | Verbose logs | `false` |
## VRAM (24 GB)
Compose ustawia `SINGLE_ACTIVE_BACKEND=true` i `PARALLEL_REQUESTS=false` — jeden aktywny backend/model naraz.
**Nie uruchamiaj** dużego modelu w vLLM i LocalAI równocześnie na tej samej karcie:
```bash
cd ../vllm && docker compose --profile vllm down
```
## Modele (później)
- UI → Model Gallery w przeglądarce
- CLI w kontenerze: `docker exec -it localai local-ai models install ...`
- Ręcznie: GGUF + YAML w `/data/apps/localai/models/`
GGUF z [`stacks/vllm/models.catalog.yaml`](../vllm/models.catalog.yaml) można skopiować lub podlinkować do `models/`.
## KV cache (skwantyzowany q8_0)
Domyślnie llama.cpp trzyma KV cache w **f16** — dużo VRAM przy długim kontekście. Ustawienia są **per model** w YAML na `/data`, nie w compose.
| Pole | Rekomendacja |
|------|--------------|
| `cache_type_k` | `q8_0` |
| `cache_type_v` | `q8_0` |
| `flash_attention` | `true` (wymagane przy q8_0 V) |
| `context_size` | `8192` (start; zwiększ po teście VRAM) |
Szablon: [`profiles/gemma-4-12b-q4-kv-q8.yaml.example`](profiles/gemma-4-12b-q4-kv-q8.yaml.example)
Zastosowanie na istniejącym modelu:
```bash
./scripts/apply-kv-profile.sh gemma-4-12b-it-qat-q4_0
docker compose --profile localai restart localai
```
Szczegóły: [`coding-agent/KV-CACHE.md`](coding-agent/KV-CACHE.md)
## Zarządzanie
```bash
docker compose --profile localai ps
docker compose --profile localai logs -f localai
docker compose --profile localai restart localai
docker compose --profile localai down
```
## Dokumentacja
Tutorial: [manual-tutorial/05-localai-stack.md](../../manual-tutorial/05-localai-stack.md)
Upstream: [github.com/mudler/LocalAI](https://github.com/mudler/LocalAI)
stacks/localai/coding-agent/BACKLOG.md
+28
View File
@@ -0,0 +1,28 @@
# BACKLOG — LocalAI stack
## P0 — KV cache (bieżąca sesja)
- [x] Audyt konfiguracji KV (compose, YAML, backendy)
- [x] Szablon `profiles/gemma-4-12b-q4-kv-q8.yaml.example`
- [x] Skrypt `scripts/apply-kv-profile.sh`
- [x] Aktualizacja YAML Gemma na `/data` (via `docker exec` — plik root-owned)
- [x] Restart `localai` — health OK
- [ ] **Test VRAM po załadowaniu Gemma GGUF** — uzupełnić STATE.md
## P1 — po pierwszym modelu chat
- [ ] Dostroić `context_size` (8192 → 16384 jeśli VRAM pozwala)
- [ ] Porównanie jakości odpowiedzi f16 vs q8_0 KV (krótki prompt)
- [ ] Profil KV dla Qwen3.6-27B GGUF (gdy dodany do LocalAI)
- [ ] Przekazać `LOCALAI_API_KEY` do `docker-compose.yml` (zsynchronizować z root BACKLOG)
## P2 — opcjonalnie
- [ ] Backend `turboquant` + `turbo3`/`turbo4` (~3–4× kompresja KV)
- [ ] Skrypt sync GGUF z `stacks/vllm/models.catalog.yaml`
- [ ] Reverse proxy + firewall (root tutorial 07)
## P3 — dokumentacja
- [ ] Zsynchronizować port 8070 vs 8080 w całym repo (root BACKLOG)
- [ ] Przykład `curl /v1/chat/completions` z auth w tutorialu
stacks/localai/coding-agent/CONVENTIONS.md
+41
View File
@@ -0,0 +1,41 @@
# Konwencje — stack LocalAI
Skrót reguł specyficznych dla tego stacku. Pełne konwencje repo: [`../../coding-agent/CONVENTIONS.md`](../../coding-agent/CONVENTIONS.md).
## Ścieżki
| Warstwa | Ścieżka |
|---------|---------|
| Repo stack | `ubuntu-bare-metal/stacks/localai/` |
| `.env` (sekrety) | `stacks/localai/.env` — **gitignore** |
| Modele runtime | `/data/apps/localai/models/` |
| YAML modeli | `/data/apps/localai/models/*.yaml` — **poza git** |
| Szablony KV | `stacks/localai/profiles/*.yaml.example` |
| Notatki agenta | `stacks/localai/coding-agent/` |
## KV cache
- Ustawienia **tylko** w sekcji `parameters:` pliku YAML modelu.
- Skwantyzowany `cache_type_v` wymaga `flash_attention: true`.
- Dozwolone na `llama-cpp`: `f16`, `f32`, `q8_0`, `q4_0`, `q4_1`, `q5_0`, `q5_1`.
- Startowa rekomendacja: `cache_type_k: q8_0`, `cache_type_v: q8_0`, `context_size: 8192`.
- Modele embedding (np. bge-m3) — **nie** dodawać KV cache.
## Docker
```bash
cd stacks/localai
docker compose --profile localai up -d
docker compose --profile localai restart localai
```
Port wewnątrz kontenera zawsze **8080**; host mapuje `LOCALAI_PORT` (użytkownik: **8070**).
## VRAM
- Jeden duży model chat na GPU naraz.
- Przed loadem dużego modelu: `cd ../vllm && docker compose --profile vllm down`.
## Sekrety
- `LOCALAI_API_KEY` — tylko w `.env` na serwerze, nie w `coding-agent/`.
stacks/localai/coding-agent/EMBEDDING-STATUS-REPORT.md
+147
View File
@@ -0,0 +1,147 @@
# RTX1 — raport naprawy embeddingu (odpowiedź dla ai-lawyer-srvr)
**Typ dokumentu:** raport po stronie **RTX1** (LocalAI na `gmktec-k11`) — odpowiedź na żądanie z hosta dev.
**Data:** 2026-06-30
**Status:** **NAPRAWIONE** — `POST /v1/embeddings` zwraca HTTP 200, wektor **1024** wymiarów.
**Adresat:** agent kodujący na hoście dev (`ai-lawyer-srvr`).
---
## 1. Podsumowanie wykonawcze
| Obszar | Stan |
|--------|------|
| Connectivity (`llm.rtx1.mobile.agency-ai.dev`) | OK (wcześniej potwierdzone przez dev) |
| Graph LLM (`gemma-4-12b-it-qat-q4_0`) | OK |
| Embedding (`bge-m3-FP16.gguf`) | **OK** po poprawce YAML |
| Zmiana `.env` dev | **Nie wymagana** |
---
## 2. Root cause
Model `bge-m3-FP16.gguf` został zaimportowany z galerii LocalAI z konfiguracją **chat**, bez flagi embedding:
```yaml
# PRZED (błędne)
known_usecases:
- chat
# brak: embeddings: true
```
Skutek: worker llama-cpp padał przy `POST /v1/embeddings` z błędem:
```
rpc error: code = Unavailable desc = error reading from server: EOF
```
**Nie było to:** zły URL, API key, brak pliku GGUF (1.1 GB OK), ani brak VRAM w stanie idle.
---
## 3. Zastosowana poprawka
Zaktualizowano `/data/apps/localai/models/bge-m3-FP16.gguf.yaml`:
```yaml
name: bge-m3-FP16.gguf
backend: llama-cpp
embeddings: true
description: BGE-M3 embedding model (1024 dims)
known_usecases:
- embedding
parameters:
model: bge-m3-FP16.gguf
context_size: 8192
```
Szablon w repo (na przyszłe reimporty): [`../profiles/bge-m3-FP16-embedding.yaml.example`](../profiles/bge-m3-FP16-embedding.yaml.example)
Restart: `docker compose --profile localai restart localai`
---
## 4. Wyniki testów akceptacyjnych (RTX1)
| Test | Endpoint | HTTP | Wynik |
|------|----------|------|-------|
| Embeddings lokalnie | `POST http://127.0.0.1:8070/v1/embeddings` | **200** | wektor **1024** dims |
| Chat → embed (swap) | gemma chat, potem bge embed | **200** / **200** | `SINGLE_ACTIVE_BACKEND` OK |
| Embeddings przez domenę z RTX1 | `POST https://llm.rtx1.mobile.agency-ai.dev/v1/embeddings` | timeout (000) | hairpin NAT z wnętrza LAN — **nie blokuje dev** |
Test lokalny (na RTX1):
```bash
curl -s http://127.0.0.1:8070/v1/embeddings \
-H "Authorization: Bearer $LOCALAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{"input":"test prawny","model":"bge-m3-FP16.gguf"}' \
| python3 -c "import sys,json; d=json.load(sys.stdin); print(len(d['data'][0]['embedding']))"
# Wynik: 1024
```
---
## 5. Wartości dla `.env` dev — bez zmian
```bash
EMBEDDING_URL=https://llm.rtx1.mobile.agency-ai.dev/v1
GRAPH_LLM_URL=https://llm.rtx1.mobile.agency-ai.dev/v1
EMBEDDING_MODEL=bge-m3-FP16.gguf
GRAPH_LLM_MODEL=gemma-4-12b-it-qat-q4_0
EMBEDDING_DIMS=1024
```
`EMBEDDING_API_KEY` / `GRAPH_LLM_API_KEY` — bez zmian (ten sam token co `LOCALAI_API_KEY` na RTX1).
---
## 6. Checklist dla agenta dev (wykonaj teraz)
1. [ ] `docker compose -f docker-compose.dev.yml exec api python scripts/discover_rtx1_models.py`
Oczekiwane: embed test **OK**, chat test **OK**
2. [ ] `docker compose -f docker-compose.dev.yml up -d api` (jeśli zmieniano `.env`)
3. [ ] Smoke hybrid chat — SSE z `"route": "hybrid"` i semantycznym retrieval
4. [ ] Jeśli ES był indeksowany bez wektorów: `python scripts/reindex_embeddings.py`
Test z hosta dev:
```bash
curl -s https://llm.rtx1.mobile.agency-ai.dev/v1/embeddings \
-H "Authorization: Bearer $EMBEDDING_API_KEY" \
-H "Content-Type: application/json" \
-d '{"input":"test prawny","model":"bge-m3-FP16.gguf"}' \
| python3 -c "import sys,json; d=json.load(sys.stdin); print('dims:', len(d['data'][0]['embedding']))"
```
Oczekiwane: `dims: 1024`
---
## 7. Architektura (bez zmian)
- Publiczny URL: `https://llm.rtx1.mobile.agency-ai.dev/v1`
- LocalAI LAN: `http://192.168.100.5:8070` (bind `0.0.0.0`)
- `SINGLE_ACTIVE_BACKEND=true` — gemma i bge przełączają się sekwencyjnie (OK dla hybrid RAG: embed RTX1 → chat RTX2)
---
## 8. Mapa dokumentacji
| Plik | Opis |
|------|------|
| Ten dokument | Status naprawy embeddingu |
| [`STATE.md`](STATE.md) | Runtime LocalAI na RTX1 |
| [`../profiles/bge-m3-FP16-embedding.yaml.example`](../profiles/bge-m3-FP16-embedding.yaml.example) | Szablon YAML embedding |
---
## 9. Odpowiedź na żądanie (tabela)
| Punkt | Odpowiedź |
|-------|-----------|
| 1. Root cause | Błędny YAML — `chat` zamiast `embedding`, brak `embeddings: true` |
| 2. Poprawka | YAML + restart LocalAI |
| 3. curl embed | HTTP **200**, **1024** dims (lokalnie na RTX1) |
| 4. Zmiana `id` / dims | **Nie** — `bge-m3-FP16.gguf`, `1024` |
stacks/localai/coding-agent/HANDOFF.md
+46
View File
@@ -0,0 +1,46 @@
# HANDOFF — LocalAI KV cache
## Cel sesji
Przeanalizować konfigurację KV cache w LocalAI i włączyć **skwantyzowany KV** (`q8_0`), aby zmieścić większe modele / dłuższy kontekst na RTX 3090 Ti (24 GB).
## Wynik audytu
1. W [`docker-compose.yml`](../docker-compose.yml) i [`.env`](../.env) **brak** ustawień KV — to prawidłowe; LocalAI konfiguruje KV w YAML modelu.
2. YAML `gemma-4-12b-it-qat-q4_0` na `/data` nie miał `cache_type_k`, `cache_type_v`, `flash_attention`, `context_size` → domyślnie **f16** KV (więcej VRAM).
3. Backend: tylko `cuda13-llama-cpp`. TurboQuant **nie** instalowany.
4. Plik GGUF Gemma **jeszcze nie pobrany** — YAML gotowy przed pierwszym loadem.
## Decyzja
| Opcja | Wybór |
|-------|-------|
| Standard `q8_0` + `flash_attention` na `llama-cpp` | **TAK** |
| Backend `turboquant` (turbo3/4) | **NIE** (odłożone) |
Uzasadnienie: ~2× mniej pamięci KV vs f16, bez nowego backendu, minimalny wpływ na jakość.
## Co zrobiono w repo
- Katalog `coding-agent/` (ten handoff + STATE, BACKLOG, KV-CACHE, CONVENTIONS)
- [`profiles/gemma-4-12b-q4-kv-q8.yaml.example`](../profiles/gemma-4-12b-q4-kv-q8.yaml.example)
- [`scripts/apply-kv-profile.sh`](../scripts/apply-kv-profile.sh)
- Sekcja KV w README stacku i tutorialu 05
## Co zrobiono na serwerze
- Zaktualizowano `/data/apps/localai/models/gemma-4-12b-it-qat-q4_0.yaml` (parametry KV)
- Restart kontenera `localai`
## Następne kroki (dla agenta / użytkownika)
1. Dokończyć pobieranie GGUF Gemma 4 12B (galeria UI lub URI z `._gallery_*.yaml`).
2. Po loadzie: `nvidia-smi` + krótki chat — zapisać VRAM w [STATE.md](STATE.md).
3. Jeśli zapas VRAM: podnieść `context_size` do 16384 w YAML.
4. Dla przyszłego Qwen3.6-27B GGUF: skopiować wzorzec KV z `profiles/`.
5. BACKLOG P0 root repo: przekazać `LOCALAI_API_KEY` do compose (osobne zadanie).
## Dokumentacja zewnętrzna
- [LocalAI model configuration — cache_type_k/v](https://localai.io/advanced/model-configuration/)
- [Text generation — llama-cpp backend](https://localai.io/features/text-generation/)
stacks/localai/coding-agent/KV-CACHE.md
+84
View File
@@ -0,0 +1,84 @@
# KV cache w LocalAI (llama-cpp)
## Problem
Przy inference LLM pamięć KV rośnie z długością kontekstu. Domyślnie LocalAI/llama.cpp używa **f16** dla K i V — pełna precyzja, najwięcej VRAM.
Na RTX 3090 Ti (24 GB) przy modelu Q4 + mmproj (Gemma 4 12B) kwantyzacja KV zwalnia miejsce na dłuższy `context_size` lub większy model.
## Gdzie konfigurować
| Miejsce | KV cache? |
|---------|-----------|
| `docker-compose.yml` | nie |
| `.env` stacku | nie |
| `models/<nazwa>.yaml` → `parameters:` | **tak** |
## Pola YAML (llama-cpp)
| Pole | Typ | Domyślnie | Opis |
|------|-----|-----------|------|
| `cache_type_k` | string | `f16` | Kwantyzacja cache kluczy (`-ctk` w llama.cpp) |
| `cache_type_v` | string | `f16` | Kwantyzacja cache wartości (`-ctv`) |
| `flash_attention` | bool/string | off | **Wymagane** przy skwantyzowanym `cache_type_v` |
| `context_size` | int | niski / auto | Maks. tokenów kontekstu (wpływa na rozmiar KV) |
### Dozwolone typy (`cuda13-llama-cpp`)
`f16`, `f32`, `q8_0`, `q4_0`, `q4_1`, `q5_0`, `q5_1`
### Rekomendacja dla tego serwera (q8_0)
```yaml
parameters:
cache_type_k: q8_0
cache_type_v: q8_0
flash_attention: true
context_size: 8192
```
Po teście VRAM można podnieść `context_size` do `16384`.
## Szacunek VRAM (Gemma 4 12B Q4_0 + mmproj)
| Składnik | Orientacyjnie |
|----------|---------------|
| Wagi + mmproj | ~8–10 GB |
| KV @ f16, ctx 8k | ~2–4 GB |
| KV @ q8_0, ctx 8k | ~1–2 GB |
## TurboQuant (odłożone)
Backend `turboquant` + typy `turbo2`/`turbo3`/`turbo4` dają większą kompresję (~3–4×), ale wymagają:
```bash
docker exec localai /local-ai backends install turboquant
```
oraz `backend: turboquant` w YAML. Nie wdrożone w bieżącej sesji.
## Zastosowanie profilu
```bash
cd stacks/localai
./scripts/apply-kv-profile.sh gemma-4-12b-it-qat-q4_0
docker compose --profile localai restart localai
```
## Weryfikacja
```bash
# modele
curl -s http://localhost:8070/v1/models -H "Authorization: Bearer $KEY"
# VRAM
nvidia-smi
# logi backendu
docker compose --profile localai logs localai 2>&1 | grep -iE 'cache|ctk|ctv|flash' | tail -20
```
## Źródła
- https://localai.io/advanced/model-configuration/
- https://localai.io/features/text-generation/
stacks/localai/coding-agent/README.md
+25
View File
@@ -0,0 +1,25 @@
# coding-agent — notatki dla agenta (stack LocalAI)
Katalog handoff dla sesji Cursor pracujących nad [`stacks/localai/`](../) na serwerze GMKtec K11.
## Kolejność czytania
1. **[HANDOFF.md](HANDOFF.md)** — decyzje (KV q8_0), audyt, następne kroki
2. **[STATE.md](STATE.md)** — stan runtime: kontener, modele, backendy, VRAM
3. **[KV-CACHE.md](KV-CACHE.md)** — referencja techniczna KV cache w YAML
4. **[BACKLOG.md](BACKLOG.md)** — priorytetyzowane zadania
5. **[CONVENTIONS.md](CONVENTIONS.md)** — ścieżki, sekrety, konwencje stacku
Wspólne konwencje repo: [`../../coding-agent/CONVENTIONS.md`](../../coding-agent/CONVENTIONS.md)
## Zasady
- Instrukcje dla użytkownika: **po polsku**. Komendy: **po angielsku**.
- **Nie commituj** ani nie zapisuj tutaj wartości `LOCALAI_API_KEY`, tokenów HF itd.
- YAML modeli na `/data/apps/localai/models/` **nie są w git** — szablony trzymaj w [`../profiles/`](../profiles/).
- Commity i push **tylko na prośbę** użytkownika.
- Nie edytuj plików planu w `.cursor/plans/`.
## Ostatnia aktualizacja
Sesja: audyt KV cache → wdrożenie `cache_type_k/v: q8_0` + `flash_attention` dla modeli chat (llama-cpp).
stacks/localai/coding-agent/STATE.md
+62
View File
@@ -0,0 +1,62 @@
# STATE — LocalAI runtime
Ostatnia aktualizacja: po wdrożeniu BGE-Reranker-v2-m3 (2026-07-01).
## Kontener
| Element | Wartość |
|---------|---------|
| Nazwa | `localai` |
| Obraz | `localai/localai:v4.4.3-gpu-nvidia-cuda-13` |
| Status | running (healthy) |
| Port hosta | **8070** → 8080 w kontenerze (`0.0.0.0` — LAN) |
| GPU | `CUDA_VISIBLE_DEVICES=0` (RTX 3090 Ti) |
| API auth | `LOCALAI_API_KEY` w compose — 401 bez Bearer |
## Publiczny endpoint (NPMPlus)
| Element | Wartość |
|---------|---------|
| Domena | `https://llm.rtx1.mobile.agency-ai.dev/v1` |
| Upstream | `http://127.0.0.1:8070` (lub LAN `192.168.100.5:8070`) |
## Backendy
| Backend | Zainstalowany |
|---------|---------------|
| `cuda13-llama-cpp` (alias `llama-cpp`) | tak |
| `turboquant` | **nie** (odłożone) |
## Modele (`/data/apps/localai/models/`)
| Model | GGUF | YAML | Status API |
|-------|------|------|------------|
| `gemma-4-12b-it-qat-q4_0` | tak (~6.5 GB) | KV `q8_0`, `flash_attention`, `context_size: 8192` | chat **OK** |
| `bge-m3-FP16.gguf` | tak (1.1 GB) | `embeddings: true`, `known_usecases: [embedding]` | embed **OK**, 1024 dims |
| `bge-reranker-v2-m3-FP16.gguf` | tak (1.1 GB) | `reranking: true`, `known_usecases: [rerank]`, `backend: llama-cpp` | rerank **OK** |
Szablony w repo:
- Gemma KV: [`../profiles/gemma-4-12b-q4-kv-q8.yaml.example`](../profiles/gemma-4-12b-q4-kv-q8.yaml.example)
- BGE embed: [`../profiles/bge-m3-FP16-embedding.yaml.example`](../profiles/bge-m3-FP16-embedding.yaml.example)
- BGE rerank: [`../profiles/bge-reranker-v2-m3-FP16-rerank.yaml.example`](../profiles/bge-reranker-v2-m3-FP16-rerank.yaml.example)
## Weryfikacja (2026-06-30)
| Test | Wynik |
|------|-------|
| `GET /readyz` | 200 |
| `GET /v1/models` (auth) | 200 — 2 modele |
| `POST /v1/chat/completions` (gemma) | 200 |
| `POST /v1/embeddings` (bge-m3) | **200**, wektor **1024** |
| `POST /v1/rerank` (bge-reranker) | **200**, indeks 2 (panda) na top |
| Chat → embed (SINGLE_ACTIVE_BACKEND) | 200 / 200 |
| Embeddings z RTX1 przez publiczną domenę | timeout (hairpin NAT) — dev powinien testować z zewnątrz |
## Ścieżki
| Co | Gdzie |
|----|-------|
| Repo stack | `/home/tomasz-syn-grzegorza/cursor/ubuntu-bare-metal/stacks/localai` |
| Modele runtime | `/data/apps/localai/models` |
| Raport dla dev | [`EMBEDDING-STATUS-REPORT.md`](EMBEDDING-STATUS-REPORT.md) |
stacks/localai/compose.yaml
Symlink
+1
View File
@@ -0,0 +1 @@
docker-compose.yml
stacks/localai/docker-compose.yml
+31
View File
@@ -0,0 +1,31 @@
services:
localai:
image: ${LOCALAI_IMAGE:-localai/localai:v4.4.3-gpu-nvidia-cuda-13}
container_name: localai
profiles:
- localai
restart: unless-stopped
init: true
ports:
- "${LOCALAI_PORT:-8080}:8080"
environment:
- LOCALAI_API_KEY=${LOCALAI_API_KEY:-}
- CUDA_VISIBLE_DEVICES=${CUDA_VISIBLE_DEVICES:-0}
- NVIDIA_DRIVER_CAPABILITIES=compute,utility
- MODELS_PATH=/models
- DEBUG=${DEBUG:-false}
- SINGLE_ACTIVE_BACKEND=true
- PARALLEL_REQUESTS=false
volumes:
- ${DATA_ROOT:-/data}/apps/localai/models:/models
- ${DATA_ROOT:-/data}/apps/localai/backends:/backends
- ${DATA_ROOT:-/data}/apps/localai/configuration:/configuration
- ${DATA_ROOT:-/data}/apps/localai/images:/tmp/generated/images
- ${DATA_ROOT:-/data}/apps/localai/data:/data
gpus: all
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8080/readyz"]
interval: 1m
timeout: 20m
retries: 5
start_period: 2m
stacks/localai/profiles/bge-m3-FP16-embedding.yaml.example
+16
View File
@@ -0,0 +1,16 @@
# BGE-M3 embedding model for LocalAI (llama-cpp backend).
# Copy to /data/apps/localai/models/bge-m3-FP16.gguf.yaml after gallery import.
#
# GGUF: bge-m3-FP16.gguf (~1.1 GB) in same directory as this YAML.
# API id: bge-m3-FP16.gguf
# Vector dimensions: 1024
name: bge-m3-FP16.gguf
backend: llama-cpp
embeddings: true
description: BGE-M3 embedding model (1024 dims)
known_usecases:
- embedding
parameters:
model: bge-m3-FP16.gguf
context_size: 8192
stacks/localai/profiles/bge-reranker-v2-m3-FP16-rerank.yaml.example
+27
View File
@@ -0,0 +1,27 @@
# BGE-Reranker-v2-m3 for LocalAI (llama-cpp / cuda13-llama-cpp backend).
# Copy to /data/apps/localai/models/bge-reranker-v2-m3-FP16.gguf.yaml after GGUF download.
#
# GGUF: bge-reranker-v2-m3-FP16.gguf (~1.1 GB) in same directory as this YAML.
# API id: bge-reranker-v2-m3-FP16.gguf
# Endpoint: POST /v1/rerank
#
# Do NOT use backend: rerankers for GGUF — that backend is for HuggingFace transformers.
# cuda13-llama-cpp is selected automatically on the cuda-13 image when backend: llama-cpp.
name: bge-reranker-v2-m3-FP16.gguf
backend: llama-cpp
reranking: true
embeddings: false
description: BGE-Reranker-v2-m3 cross-encoder (FP16 GGUF)
known_usecases:
- rerank
parameters:
model: bge-reranker-v2-m3-FP16.gguf
context_size: 8192
template:
use_tokenizer_template: true
function:
grammar:
disable: true
options:
- use_jinja:true
stacks/localai/profiles/gemma-4-12b-q4-kv-q8.yaml.example
+32
View File
@@ -0,0 +1,32 @@
# Szablon: Gemma 4 12B Q4_0 z kwantyzowanym KV cache (q8_0)
#
# Zastosowanie na serwerze:
# ./scripts/apply-kv-profile.sh gemma-4-12b-it-qat-q4_0
# lub skopiuj sekcję parameters do /data/apps/localai/models/<nazwa>.yaml
#
# Dokumentacja: coding-agent/KV-CACHE.md
name: gemma-4-12b-it-qat-q4_0
backend: llama-cpp
mmproj: llama-cpp/mmproj/gemma-4-12B-it-qat-q4_0-gguf/mmproj-gemma-4-12b-it-qat-q4_0.gguf
known_usecases:
- chat
options:
- use_jinja:true
parameters:
model: llama-cpp/models/gemma-4-12B-it-qat-q4_0-gguf/gemma-4-12b-it-qat-q4_0.gguf
cache_type_k: q8_0
cache_type_v: q8_0
flash_attention: true
context_size: 8192
temperature: 1
top_p: 0.95
top_k: 64
repeat_penalty: 1
min_p: 0
template:
use_tokenizer_template: true
function:
automatic_tool_parsing_fallback: true
grammar:
disable: true
stacks/localai/scripts/apply-kv-profile.sh
Executable
+95
View File
@@ -0,0 +1,95 @@
#!/usr/bin/env bash
# Merge KV cache settings (q8_0) into a model YAML on /data.
# Usage: ./scripts/apply-kv-profile.sh <model-name-without-.yaml>
# Example: ./scripts/apply-kv-profile.sh gemma-4-12b-it-qat-q4_0
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
STACK_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)"
if [[ ! -f "${STACK_DIR}/.env" ]]; then
echo "ERROR: ${STACK_DIR}/.env not found"
exit 1
fi
set -a
# shellcheck disable=SC1091
source "${STACK_DIR}/.env"
set +a
DATA_ROOT="${DATA_ROOT:-/data}"
MODELS_DIR="${DATA_ROOT}/apps/localai/models"
if [[ $# -lt 1 ]]; then
echo "Usage: $0 <model-name>"
echo "Example: $0 gemma-4-12b-it-qat-q4_0"
exit 1
fi
MODEL_NAME="$1"
TARGET="${MODELS_DIR}/${MODEL_NAME}.yaml"
if [[ ! -f "${TARGET}" ]]; then
echo "ERROR: ${TARGET} not found"
exit 1
fi
BACKUP="${TARGET}.bak.$(date +%Y%m%d%H%M%S)"
cp "${TARGET}" "${BACKUP}"
echo "Backup: ${BACKUP}"
run_python_merge() {
python3 - "${1}" <<'PY'
import sys
from pathlib import Path
try:
import yaml
except ImportError:
sys.exit("ERROR: python3-yaml required (sudo apt install python3-yaml)")
path = Path(sys.argv[1])
data = yaml.safe_load(path.read_text(encoding="utf-8")) or {}
params = data.setdefault("parameters", {})
kv = {
"cache_type_k": "q8_0",
"cache_type_v": "q8_0",
"flash_attention": True,
}
if "context_size" not in params:
kv["context_size"] = 8192
params.update(kv)
path.write_text(
yaml.dump(data, default_flow_style=False, allow_unicode=True, sort_keys=False),
encoding="utf-8",
)
print(f"Updated KV settings in {path}")
for k, v in kv.items():
print(f" {k}: {v}")
if "context_size" in params and "context_size" not in kv:
print(f" context_size: {params['context_size']} (unchanged)")
PY
}
if run_python_merge "${TARGET}" 2>/dev/null; then
:
elif docker ps --format '{{.Names}}' 2>/dev/null | grep -qx localai; then
echo "Host write failed — applying via docker exec localai (models volume is root-owned)"
CONTAINER_PATH="/models/${MODEL_NAME}.yaml"
docker exec localai sh -c "grep -q cache_type_k '${CONTAINER_PATH}' || sed -i '/^ model:/a\\
cache_type_k: q8_0\\
cache_type_v: q8_0\\
flash_attention: true\\
context_size: 8192' '${CONTAINER_PATH}'"
grep -E 'cache_type_k|cache_type_v|flash_attention|context_size' "${TARGET}" || true
else
echo "ERROR: cannot write ${TARGET} (permission denied) and container localai not running"
echo "Run: sudo $0 ${MODEL_NAME}"
exit 1
fi
echo ""
echo "Restart LocalAI to apply:"
echo " cd ${STACK_DIR} && docker compose --profile localai restart localai"
stacks/localai/scripts/clone-upstream.sh
Executable
+28
View File
@@ -0,0 +1,28 @@
#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
STACK_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)"
UPSTREAM_DIR="${STACK_DIR}/upstream"
REPO_URL="https://github.com/mudler/LocalAI.git"
TAG="v4.4.3"
cd "${STACK_DIR}"
if [[ -d "${UPSTREAM_DIR}/.git" ]]; then
echo "Upstream already cloned: ${UPSTREAM_DIR}"
echo "Remove it first to re-clone: rm -rf upstream"
exit 0
fi
echo "=== Cloning LocalAI upstream (reference only) ==="
echo "Repo: ${REPO_URL}"
echo "Tag: ${TAG}"
echo "Dest: ${UPSTREAM_DIR}"
echo ""
git clone --depth 1 --branch "${TAG}" "${REPO_URL}" "${UPSTREAM_DIR}"
echo ""
echo "Done. Use upstream/ for example model YAML and compose reference."
echo "Runtime uses the official Docker image from .env — not a local build."
stacks/localai/scripts/download-reranker.sh
Executable
+66
View File
@@ -0,0 +1,66 @@
#!/usr/bin/env bash
# Download BGE-Reranker-v2-m3 FP16 GGUF and apply YAML profile.
# Usage: ./scripts/download-reranker.sh
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
STACK_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)"
cd "${STACK_DIR}"
if [[ -f .env ]]; then
set -a
# shellcheck disable=SC1091
source .env
set +a
fi
DATA_ROOT="${DATA_ROOT:-/data}"
MODELS_DIR="${DATA_ROOT}/apps/localai/models"
GGUF_NAME="bge-reranker-v2-m3-FP16.gguf"
GGUF_URL="https://huggingface.co/gpustack/bge-reranker-v2-m3-GGUF/resolve/main/${GGUF_NAME}"
YAML_SRC="${STACK_DIR}/profiles/bge-reranker-v2-m3-FP16-rerank.yaml.example"
YAML_DST="${MODELS_DIR}/${GGUF_NAME}.yaml"
"${SCRIPT_DIR}/ensure-dirs.sh" "${DATA_ROOT}"
echo "=== BGE-Reranker-v2-m3 download ==="
echo "Target: ${MODELS_DIR}/${GGUF_NAME}"
echo ""
if [[ -f "${MODELS_DIR}/${GGUF_NAME}" ]]; then
echo "GGUF already exists — skipping download"
else
if command -v wget &>/dev/null; then
wget -c -O "${MODELS_DIR}/${GGUF_NAME}.partial" "${GGUF_URL}"
mv "${MODELS_DIR}/${GGUF_NAME}.partial" "${MODELS_DIR}/${GGUF_NAME}"
else
curl -fL -C - -o "${MODELS_DIR}/${GGUF_NAME}.partial" "${GGUF_URL}"
mv "${MODELS_DIR}/${GGUF_NAME}.partial" "${MODELS_DIR}/${GGUF_NAME}"
fi
fi
echo ""
echo "=== Applying YAML profile ==="
if cp "${YAML_SRC}" "${YAML_DST}" 2>/dev/null; then
chmod 644 "${YAML_DST}" "${MODELS_DIR}/${GGUF_NAME}" 2>/dev/null || true
elif docker ps --format '{{.Names}}' | grep -qx localai; then
echo "Host copy failed (permissions) — writing via docker exec localai"
docker exec -i localai sh -c "cat > /models/${GGUF_NAME}.yaml" < "${YAML_SRC}"
else
echo "ERROR: cannot write ${YAML_DST} (permission denied) and localai container not running"
exit 1
fi
ls -lh "${MODELS_DIR}/${GGUF_NAME}" "${YAML_DST}"
echo ""
echo "=== Done ==="
echo "Restart LocalAI to load the model:"
echo " cd ${STACK_DIR} && docker compose --profile localai restart localai"
echo ""
echo "Smoke test:"
echo ' curl -s http://127.0.0.1:${LOCALAI_PORT:-8070}/v1/rerank \'
echo ' -H "Authorization: Bearer <LOCALAI_API_KEY>" \'
echo ' -H "Content-Type: application/json" \'
echo ' -d '"'"'{"model":"bge-reranker-v2-m3-FP16.gguf","query":"panda","documents":["hi","it is a bear","The giant panda is a bear species endemic to China."],"top_n":2}'"'"
stacks/localai/scripts/ensure-dirs.sh
Executable
+16
View File
@@ -0,0 +1,16 @@
#!/usr/bin/env bash
# Create LocalAI data directories on the data disk.
ensure_localai_dirs() {
local data_root="${1:-/data}"
mkdir -p \
"${data_root}/apps/localai/models" \
"${data_root}/apps/localai/backends" \
"${data_root}/apps/localai/configuration" \
"${data_root}/apps/localai/images" \
"${data_root}/apps/localai/data"
}
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
ensure_localai_dirs "${1:-/data}"
fi
stacks/localai/scripts/pull.sh
Executable
+31
View File
@@ -0,0 +1,31 @@
#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
STACK_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)"
cd "${STACK_DIR}"
if [[ ! -f .env ]]; then
echo "ERROR: .env not found. Run: cp .env.example .env"
exit 1
fi
set -a
# shellcheck disable=SC1091
source .env
set +a
if ! docker info &>/dev/null; then
echo "ERROR: Docker is not running"
exit 1
fi
echo "=== LocalAI — pull image only (no start) ==="
echo "Image: ${LOCALAI_IMAGE:-localai/localai:v4.4.3-gpu-nvidia-cuda-13}"
echo ""
docker compose --profile localai pull
echo ""
echo "Done. Start with: ./scripts/start.sh"
stacks/localai/scripts/start.sh
Executable
+55
View File
@@ -0,0 +1,55 @@
#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
STACK_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)"
# shellcheck disable=SC1091
source "${SCRIPT_DIR}/ensure-dirs.sh"
cd "${STACK_DIR}"
if [[ ! -f .env ]]; then
echo "ERROR: .env not found. Run: cp .env.example .env"
exit 1
fi
set -a
# shellcheck disable=SC1091
source .env
set +a
DATA_ROOT="${DATA_ROOT:-/data}"
if ! mountpoint -q "${DATA_ROOT}" 2>/dev/null; then
echo "ERROR: ${DATA_ROOT} is not mounted"
echo " Complete disk setup (tutorial 04 part A) first"
exit 1
fi
ensure_localai_dirs "${DATA_ROOT}"
if ! docker info &>/dev/null; then
echo "ERROR: Docker is not running"
exit 1
fi
echo "=== LocalAI stack ==="
echo "Image: ${LOCALAI_IMAGE:-localai/localai:v4.4.3-gpu-nvidia-cuda-13}"
echo "Port: ${LOCALAI_PORT:-8080}"
echo "Models: ${DATA_ROOT}/apps/localai/models (empty OK)"
echo "GPU: CUDA_VISIBLE_DEVICES=${CUDA_VISIBLE_DEVICES:-0}"
echo ""
docker compose --profile localai pull
docker compose --profile localai up -d
echo ""
echo "Started. Follow logs:"
echo " docker compose --profile localai logs -f localai"
echo ""
echo "Health check:"
echo " curl -s http://localhost:${LOCALAI_PORT:-8080}/readyz"
echo ""
echo "Web UI:"
echo " http://localhost:${LOCALAI_PORT:-8080}"