tomasz-syn-grzegorza
359afb3a59
Infrastructure configs for GMKtec K11 (Docker, vLLM, LocalAI, ComfyUI, control-plane, gpu-fan agent, Server UI with CLI/file explorer/GPU fan curve). Co-authored-by: Cursor <cursoragent@cursor.com>
2.3 KiB
2.3 KiB
STATE — LocalAI runtime
Ostatnia aktualizacja: po wdrożeniu BGE-Reranker-v2-m3 (2026-07-01).
Kontener
| Element | Wartość |
|---|---|
| Nazwa | localai |
| Obraz | localai/localai:v4.4.3-gpu-nvidia-cuda-13 |
| Status | running (healthy) |
| Port hosta | 8070 → 8080 w kontenerze (0.0.0.0 — LAN) |
| GPU | CUDA_VISIBLE_DEVICES=0 (RTX 3090 Ti) |
| API auth | LOCALAI_API_KEY w compose — 401 bez Bearer |
Publiczny endpoint (NPMPlus)
| Element | Wartość |
|---|---|
| Domena | https://llm.rtx1.mobile.agency-ai.dev/v1 |
| Upstream | http://127.0.0.1:8070 (lub LAN 192.168.100.5:8070) |
Backendy
| Backend | Zainstalowany |
|---|---|
cuda13-llama-cpp (alias llama-cpp) |
tak |
turboquant |
nie (odłożone) |
Modele (/data/apps/localai/models/)
| Model | GGUF | YAML | Status API |
|---|---|---|---|
gemma-4-12b-it-qat-q4_0 |
tak (~6.5 GB) | KV q8_0, flash_attention, context_size: 8192 |
chat OK |
bge-m3-FP16.gguf |
tak (1.1 GB) | embeddings: true, known_usecases: [embedding] |
embed OK, 1024 dims |
bge-reranker-v2-m3-FP16.gguf |
tak (1.1 GB) | reranking: true, known_usecases: [rerank], backend: llama-cpp |
rerank OK |
Szablony w repo:
- Gemma KV:
../profiles/gemma-4-12b-q4-kv-q8.yaml.example - BGE embed:
../profiles/bge-m3-FP16-embedding.yaml.example - BGE rerank:
../profiles/bge-reranker-v2-m3-FP16-rerank.yaml.example
Weryfikacja (2026-06-30)
| Test | Wynik |
|---|---|
GET /readyz |
200 |
GET /v1/models (auth) |
200 — 2 modele |
POST /v1/chat/completions (gemma) |
200 |
POST /v1/embeddings (bge-m3) |
200, wektor 1024 |
POST /v1/rerank (bge-reranker) |
200, indeks 2 (panda) na top |
| Chat → embed (SINGLE_ACTIVE_BACKEND) | 200 / 200 |
| Embeddings z RTX1 przez publiczną domenę | timeout (hairpin NAT) — dev powinien testować z zewnątrz |
Ścieżki
| Co | Gdzie |
|---|---|
| Repo stack | /home/tomasz-syn-grzegorza/cursor/ubuntu-bare-metal/stacks/localai |
| Modele runtime | /data/apps/localai/models |
| Raport dla dev | EMBEDDING-STATUS-REPORT.md |