tomasz-syn-grzegorza
359afb3a59
Infrastructure configs for GMKtec K11 (Docker, vLLM, LocalAI, ComfyUI, control-plane, gpu-fan agent, Server UI with CLI/file explorer/GPU fan curve). Co-authored-by: Cursor <cursoragent@cursor.com>
63 lines
2.3 KiB
Markdown
63 lines
2.3 KiB
Markdown
# STATE — LocalAI runtime
|
|
|
|
Ostatnia aktualizacja: po wdrożeniu BGE-Reranker-v2-m3 (2026-07-01).
|
|
|
|
## Kontener
|
|
|
|
| Element | Wartość |
|
|
|---------|---------|
|
|
| Nazwa | `localai` |
|
|
| Obraz | `localai/localai:v4.4.3-gpu-nvidia-cuda-13` |
|
|
| Status | running (healthy) |
|
|
| Port hosta | **8070** → 8080 w kontenerze (`0.0.0.0` — LAN) |
|
|
| GPU | `CUDA_VISIBLE_DEVICES=0` (RTX 3090 Ti) |
|
|
| API auth | `LOCALAI_API_KEY` w compose — 401 bez Bearer |
|
|
|
|
## Publiczny endpoint (NPMPlus)
|
|
|
|
| Element | Wartość |
|
|
|---------|---------|
|
|
| Domena | `https://llm.rtx1.mobile.agency-ai.dev/v1` |
|
|
| Upstream | `http://127.0.0.1:8070` (lub LAN `192.168.100.5:8070`) |
|
|
|
|
## Backendy
|
|
|
|
| Backend | Zainstalowany |
|
|
|---------|---------------|
|
|
| `cuda13-llama-cpp` (alias `llama-cpp`) | tak |
|
|
| `turboquant` | **nie** (odłożone) |
|
|
|
|
## Modele (`/data/apps/localai/models/`)
|
|
|
|
| Model | GGUF | YAML | Status API |
|
|
|-------|------|------|------------|
|
|
| `gemma-4-12b-it-qat-q4_0` | tak (~6.5 GB) | KV `q8_0`, `flash_attention`, `context_size: 8192` | chat **OK** |
|
|
| `bge-m3-FP16.gguf` | tak (1.1 GB) | `embeddings: true`, `known_usecases: [embedding]` | embed **OK**, 1024 dims |
|
|
| `bge-reranker-v2-m3-FP16.gguf` | tak (1.1 GB) | `reranking: true`, `known_usecases: [rerank]`, `backend: llama-cpp` | rerank **OK** |
|
|
|
|
Szablony w repo:
|
|
|
|
- Gemma KV: [`../profiles/gemma-4-12b-q4-kv-q8.yaml.example`](../profiles/gemma-4-12b-q4-kv-q8.yaml.example)
|
|
- BGE embed: [`../profiles/bge-m3-FP16-embedding.yaml.example`](../profiles/bge-m3-FP16-embedding.yaml.example)
|
|
- BGE rerank: [`../profiles/bge-reranker-v2-m3-FP16-rerank.yaml.example`](../profiles/bge-reranker-v2-m3-FP16-rerank.yaml.example)
|
|
|
|
## Weryfikacja (2026-06-30)
|
|
|
|
| Test | Wynik |
|
|
|------|-------|
|
|
| `GET /readyz` | 200 |
|
|
| `GET /v1/models` (auth) | 200 — 2 modele |
|
|
| `POST /v1/chat/completions` (gemma) | 200 |
|
|
| `POST /v1/embeddings` (bge-m3) | **200**, wektor **1024** |
|
|
| `POST /v1/rerank` (bge-reranker) | **200**, indeks 2 (panda) na top |
|
|
| Chat → embed (SINGLE_ACTIVE_BACKEND) | 200 / 200 |
|
|
| Embeddings z RTX1 przez publiczną domenę | timeout (hairpin NAT) — dev powinien testować z zewnątrz |
|
|
|
|
## Ścieżki
|
|
|
|
| Co | Gdzie |
|
|
|----|-------|
|
|
| Repo stack | `/home/tomasz-syn-grzegorza/cursor/ubuntu-bare-metal/stacks/localai` |
|
|
| Modele runtime | `/data/apps/localai/models` |
|
|
| Raport dla dev | [`EMBEDDING-STATUS-REPORT.md`](EMBEDDING-STATUS-REPORT.md) |
|