New Grimoire

Gremlin-Grimoire/Runbooks/Deploy.md

@@ -0,0 +1,73 @@
+---
+title: Deploy Gremlin Stack
+description: How to deploy and redeploy the Gremlin AI stack
+published: true
+date: 2026-04-12T00:00:00.000Z
+tags: gremlin, deploy, runbook
+editor: markdown
+dateCreated: 2026-04-12T00:00:00.000Z
+---
+
+# Deploy Gremlin Stack
+
+All Gremlin services run on `docker4` (hermes), pinned via `node.hostname == docker4`.
+
+---
+
+## Prerequisites
+
+```bash
+# On docker4 — create volume directories
+mkdir -p /DockerVol/ollama
+mkdir -p /DockerVol/open-webui
+mkdir -p /DockerVol/qdrant
+
+# n8n requires specific ownership
+mkdir -p /DockerVol/n8n
+chown -R 1000:1000 /DockerVol/n8n
+```
+
+---
+
+## Deploy
+
+```bash
+cd ~/services && git pull
+cd swarm/stack/Gremlin
+set -a && source .env && set +a
+docker stack config --compose-file gremlin-stack.yml > resolved.yml
+docker stack deploy --compose-file resolved.yml gremlin
+rm resolved.yml
+docker stack services gremlin
+```
+
+---
+
+## Pull Models After Deploy
+
+Models must be pulled before n8n workflows run. Ollama returns a silent model-not-found error if workflows fire first.
+
+```bash
+docker exec $(docker ps -qf name=gremlin_ollama) ollama pull llama3.2:3b
+docker exec $(docker ps -qf name=gremlin_ollama) ollama pull qwen2.5-coder:7b
+
+# Verify
+docker exec $(docker ps -qf name=gremlin_ollama) ollama list
+```
+
+---
+
+## Verify Open WebUI Secret Key
+
+Check that `WEBUI_SECRET_KEY` in `.env` on docker4 is set to a real secret, not the placeholder `change-this-secret-key`.
+
+---
+
+## Service URLs After Deploy
+
+| Service | Internal | External |
+|---------|----------|---------|
+| Ollama | `http://ollama:11434` | `http://ollama.netgrimoire.com:11434` |
+| Open WebUI | `http://open-webui:8080` | `https://ai.netgrimoire.com` |
+| Qdrant | `http://qdrant:6333` | `http://qdrant.netgrimoire.com:6333/dashboard` |
+| n8n | `http://n8n:5678` | `https://n8n.netgrimoire.com` |

Gremlin-Grimoire/Runbooks/Model-Management.md

@@ -0,0 +1,41 @@
+---
+title: Ollama Model Management
+description: Pulling, verifying, and managing models on the Gremlin stack
+published: true
+date: 2026-04-12T00:00:00.000Z
+tags: gremlin, ollama, models, runbook
+editor: markdown
+dateCreated: 2026-04-12T00:00:00.000Z
+---
+
+# Ollama Model Management
+
+## Pull Required Models
+
+Run on docker4 after any fresh deploy or after the Ollama container is recreated:
+
+```bash
+docker exec $(docker ps -qf name=gremlin_ollama) ollama pull llama3.2:3b
+docker exec $(docker ps -qf name=gremlin_ollama) ollama pull qwen2.5-coder:7b
+```
+
+## Verify Models Loaded
+
+```bash
+docker exec $(docker ps -qf name=gremlin_ollama) ollama list
+```
+
+## Model Reference
+
+| Model | Size | Pull Time (CPU) | Used By |
+|-------|------|----------------|---------|
+| `llama3.2:3b` | ~2 GB | ~5 min | Kuma triage, Open WebUI |
+| `qwen2.5-coder:7b` | ~5 GB | ~15 min | Forgejo audit, Open WebUI |
+
+## Models Storage Path
+
+`/DockerVol/ollama` — survives container restarts and redeployments.
+
+## ⚠ Pull Before Workflows Run
+
+n8n workflows fail silently if models aren't present. Ollama returns a model-not-found response but n8n may not surface this as an obvious error. Always pull models immediately after deploy before enabling workflows.

Gremlin-Grimoire/Runbooks/Troubleshooting.md

@@ -0,0 +1,64 @@
+---
+title: Gremlin Troubleshooting
+description: Common Gremlin stack problems and fixes
+published: true
+date: 2026-04-12T00:00:00.000Z
+tags: gremlin, troubleshooting, runbook
+editor: markdown
+dateCreated: 2026-04-12T00:00:00.000Z
+---
+
+# Gremlin Troubleshooting
+
+## n8n Won't Start / Permission Error
+
+```bash
+# On docker4
+chown -R 1000:1000 /DockerVol/n8n
+docker service update --force gremlin_n8n
+```
+
+## Workflow Fails Silently on Ollama Call
+
+Model not pulled. Ollama returns model-not-found but n8n may not surface it clearly.
+
+```bash
+docker exec $(docker ps -qf name=gremlin_ollama) ollama list
+# If model missing:
+docker exec $(docker ps -qf name=gremlin_ollama) ollama pull llama3.2:3b
+docker exec $(docker ps -qf name=gremlin_ollama) ollama pull qwen2.5-coder:7b
+```
+
+## Forgejo Webhook Not Reaching n8n
+
+Add to Forgejo `app.ini`:
+```ini
+[webhook]
+ALLOWED_HOST_LIST = *
+```
+Restart Forgejo. Required when `OFFLINE_MODE = true`.
+
+## Caddy Routes to Wrong Container IP
+
+Ensure all Gremlin services include in labels:
+```yaml
+caddy_ingress_network: netgrimoire
+```
+
+Never use `{{upstreams PORT}}` — breaks during `docker stack config` preprocessing. Use `caddy.reverse_proxy: servicename:PORT`.
+
+## Audit Workflow Times Out
+
+Check `N8N_RUNNERS_TASK_TIMEOUT` is set to `3600` in n8n environment. Default timeout is too short for 67-file audit runs.
+
+## n8n Code Node Can't Access Env Vars
+
+Set `N8N_BLOCK_ENV_ACCESS_IN_NODE=false` in n8n environment.
+
+## Open WebUI Can't Connect to Qdrant
+
+Verify both services are on the `netgrimoire` overlay and pinned to `docker4`. Qdrant gRPC port is 6334, REST is 6333.
+
+## Audit Reports Not Committing to Forgejo
+
+Check write token is set in n8n credentials. The read and write tokens are separate — confirm the workflow is using the write token for commit operations (POST new files, PUT+SHA for updates).