traveler/Netgrimoire
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Compare commits

base: traveler:8a6320e9ca49f89bf84b6398384f4da2d5dd6140
Branches Tags
traveler:main
..
compare: traveler:b69dcf275b7e801d09a4dd2b811077ce3abe8266
Branches Tags
traveler:main

No commits in common. "8a6320e9ca49f89bf84b6398384f4da2d5dd6140" and "b69dcf275b7e801d09a4dd2b811077ce3abe8266" have entirely different histories.
8a6320e9ca ... b69dcf275b

2 changed files with 0 additions and 626 deletions
Download patch file Download diff file Expand all files Collapse all files

219
Netgrimoire/Gremlin-Grimoire/CICD_Architecture.md
View file

@ -1,219 +0,0 @@
---
title: Gremlin CI/CD Pipepline
description: N8N with LLAMA
published: true
date: 2026-04-28T20:55:22.848Z
tags:
editor: markdown
dateCreated: 2026-04-28T20:55:22.848Z
---
# Gremlin CI/CD Pipeline
> **NetGrimoire Infrastructure Reference**
> Automated validation, auto-fix, and deployment pipeline for Docker Swarm stacks.
> Runs on n8n (docker4). Triggered by Forgejo push webhooks on `traveler/services`.
---
## Overview
The Gremlin CI/CD pipeline is an n8n workflow that intercepts every push to `traveler/services`, validates changed Swarm compose files against NetGrimoire standards, automatically fixes common issues, and deploys clean stacks to the Swarm cluster. It is the enforcement layer for stack consistency across the homelab.
The pipeline is modular — each check and fix is a discrete node. Adding a new rule means adding one checker node and one fixer node. Nothing else changes.
---
## Architecture
### File Detection
Every push to `traveler/services` fires a webhook to n8n. The pipeline detects changed files via three passes:
1. **Standard arrays** — `commit.added` and `commit.modified` from the Forgejo payload
2. **Gremlin commit messages** — extracts file path from `gremlin: auto-fix swarm/foo.yaml (N issues fixed)` messages, handling Forgejo's habit of sending empty file arrays for programmatic commits
3. **Compare API fallback** — calls Forgejo's `/api/v1/repos/traveler/services/compare/before...after` if both passes find nothing
Only files under `swarm/` with `.yml` or `.yaml` extensions are processed.
### File Classification
After fetching the file content, Build Envelope classifies it:
| Type | Detection | Route |
|---|---|---|
| **Pocket** | `pocket.include: "true"` label | Silent exit |
| **Swarm** | Any `deploy:` block present | Full checker chain |
| **Compose** | No `deploy:` block | ntfy warning, skip |
### Pipeline Flow
```
Forgejo Push
└─ Parse Push Payload
└─ Build Envelope
└─ Switch (Pocket / Swarm / Compose)
├─ Pocket → (silent exit)
├─ Swarm → Checker Chain
│ └─ Evaluate Checks
│ └─ Switch (hasFailed)
│ ├─ Failed → ntfy: Blocked
│ │ └─ Switch (canFix)
│ │ ├─ Yes → Fixer Chain → Commit → ntfy: Auto-fixed
│ │ └─ No → (stop)
│ └─ Passed → Ollama Audit
│ └─ Switch (ollamaVerdict)
│ ├─ Fail → ntfy: Blocked — Ollama
│ └─ Pass → Deploy Gate
│ └─ Deploy Enabled?
│ ├─ No → ntfy: Deploy Skipped
│ └─ Yes → Prepare Volumes
│ └─ Git Pull + Deploy
│ └─ Gatus Sync
│ └─ ntfy: Deploy Complete
└─ Compose → ntfy: Non-Swarm
```
### The Envelope
Every checker and fixer operates on a single shared object called the **envelope**. It is built once per file and passed through the entire chain, accumulating issues and fixes.
Key fields:
| Field | Description |
|---|---|
| `stackName` | Derived from file path |
| `filePath` | Relative path in repo |
| `composeRaw` | Original file content — never modified |
| `fixedRaw` | Accumulates fixer changes — null until first fixer runs |
| `issues[]` | All checker findings |
| `fixes[]` | All fixer actions taken |
| `checkResults{}` | Pass/fail per checker ID |
| `hasFailed` | True if any checker failed |
| `canFix` | True if all issues are fixable and there are issues to fix |
| `isPocket` | True if pocket.include: "true" found |
| `isSwarm` | True if any deploy: block found |
| `directives{}` | Parsed gremlin.* label values |
---
## Checker Chain
Checkers run in this order. All checkers append to `envelope.issues` and set `envelope.checkResults[id]`.
| Order | ID | What it checks |
|---|---|---|
| 1 | `swarm-syntax` | Forbidden fields: version, container_name, hostname (service-level), restart, depends_on, dnsrr |
| 2 | `identity` | PUID/PGID must be 1964, or user: "1964:1964" |
| 3 | `network` | netgrimoire overlay declared and attached |
| 4 | `placement` | ARM exclusions, DockerVol/hostname rules, restart_policy |
| 5 | `caddy` | caddy: label, reverse_proxy format, import_1/import_2 |
| 6 | `homepage` | group, name, icon, href, description labels |
| 7 | `monitor` | monitor.name, monitor.url, optional type/interval |
| 8 | `legacy-labels` | Flags any kuma.* labels for removal |
| 9 | `diun` | diun.enable: "true" present |
### Fixable vs Unfixable
Auto-fix only runs when **all** issues in the file are fixable. A single unfixable issue blocks the fix chain entirely.
| Fixable | Unfixable |
|---|---|
| version: key | dnsrr endpoint mode |
| container_name, hostname (service), restart, depends_on | Missing deploy: block |
| Wrong or missing PUID/PGID | Invalid node.hostname value |
| Missing netgrimoire network | hostname missing when DockerVol present |
| ARM exclusion issues | — |
| Hostname present without DockerVol | — |
| Missing restart_policy | — |
| caddy: protocol prefix | — |
| Missing caddy.import_1/import_2 | — |
| Missing homepage labels (derived) | — |
| Missing monitor labels (derived) | — |
| Legacy kuma.* labels (removed) | — |
| Missing diun.enable | — |
---
## Fixer Chain
Fixers run in the same order as checkers. Each fixer reads from `fixedRaw` (or `composeRaw` if first) and writes its changes back to `fixedRaw`. Changes accumulate correctly across the chain.
When all fixers complete, the pipeline commits `fixedRaw` back to Forgejo with the message:
```
gremlin: auto-fix swarm/foo.yaml (N issues fixed)
- Removed version: key
- Added PUID/PGID 1964 to "app"
- ...
```
This commit re-triggers the webhook, and the pipeline runs again on the now-fixed file.
### Smart Fix Derivation
Homepage and monitor labels are derived from existing labels rather than placeholders:
- `homepage.name` / `monitor.name` → `capitalize(serviceName)`
- `homepage.href` / `monitor.url` → `https://` + `caddy:` hostname (falls back to `https://servicename.netgrimoire.com`)
- `homepage.group` → `"New"` when missing
- `homepage.icon` → `servicename.png`
- `homepage.description` → `"Servicename service"`
---
## Ollama Audit
After all checkers pass, the file is sent to Ollama (`qwen2.5-coder:7b`) for a semantic audit. The prompt explicitly instructs Ollama to:
- **Ignore:** environment variables, volume paths, port mappings, OIDC/OAuth config, secrets, application-specific settings
- **Check only:** clearly wrong image names, structural errors preventing startup, obviously broken network config
Ollama is conservative by design — when in doubt it passes. False positives can be suppressed with `gremlin.context`.
---
## Gatus Sync
After successful deploy, Gatus Sync reads `monitor.*` labels from the deployed compose file and upserts endpoints into `/DockerVol/gatus/config/config.yaml` on znas using base64-encoded SSH writes. Gatus hot-reloads the config automatically.
Alerts from Gatus go to the `gremlin-watch` ntfy topic.
---
## Infrastructure
| Component | Value |
|---|---|
| n8n host | `docker4` (192.168.5.16) |
| Swarm manager | `znas` (192.168.5.10) |
| Service account | `gremlin` |
| SSH key | `/home/gremlin/.ssh/id_ed25519` |
| Repo path on znas | `/home/gremlin/services` |
| Webhook path | `gremlin-cicd` |
| ntfy pipeline alerts | `gremlin-alerts` |
| ntfy monitoring alerts | `gremlin-watch` |
| Gatus config | `/DockerVol/gatus/config/config.yaml` |
---
## ntfy Notifications
| Event | Topic | Priority |
|---|---|---|
| Schema blocked | `gremlin-alerts` | 4 (high) |
| Ollama blocked | `gremlin-alerts` | 4 (high) |
| Auto-fixed | `gremlin-alerts` | 3 (default) |
| Deploy complete | `gremlin-alerts` | 3 (default) |
| Deploy skipped | `gremlin-alerts` | 2 (low) |
| Non-Swarm file | `gremlin-alerts` | 2 (low) |
| Service down/up | `gremlin-watch` | 3 (default) |
---
## Related
- [Gremlin CI/CD — Operator Guide](gremlin-cicd-guide.md)
- [NetGrimoire Stack Standards](stack-standards.md)
- [Gatus](gatus.md)
- [n8n](n8n.md)

407
Netgrimoire/Gremlin-Grimoire/CICD_UserGuide.md
View file

@ -1,407 +0,0 @@
---
title: Gremlin CI/CD User Guide
description:
published: true
date: 2026-04-28T20:56:45.863Z
tags:
editor: markdown
dateCreated: 2026-04-28T20:56:45.863Z
---
# Gremlin CI/CD — Operator Guide
> **NetGrimoire Infrastructure Reference**
> How to write, structure, and manage Swarm stacks for the Gremlin CI/CD pipeline.
> For pipeline architecture, see [Gremlin CI/CD Pipeline](gremlin-cicd-wiki.md).
---
## How It Works
Push any `.yml` or `.yaml` file under `swarm/` to `traveler/services` and Gremlin takes over:
1. Fetches the file and classifies it (Swarm, Pocket, or plain Compose)
2. Runs all schema checkers
3. If issues found and all are fixable — auto-fixes and recommits
4. If issues found and unfixable — sends ntfy alert, stops
5. If all checks pass — runs Ollama audit, then deploys
6. After deploy — updates Gatus monitoring config
You get ntfy notifications at every stage. A clean push produces one notification: ✅ Deploy Complete.
---
## Required Stack Structure
Every Swarm service must have these elements. Missing any will block deployment.
```yaml
services:
myservice:
image: vendor/image:tag
environment:
PUID: "1964"
PGID: "1964"
TZ: America/Chicago
volumes:
- /DockerVol/myservice:/data # pinned — requires node.hostname
# or
- /data/nfs/znas/Docker/myservice:/data # floating — no hostname needed
networks:
- netgrimoire
deploy:
restart_policy:
condition: any
delay: 5s
max_attempts: 3
window: 120s
placement:
constraints:
- node.platform.arch != aarch64
- node.platform.arch != arm
- node.hostname == znas # required when using /DockerVol/
labels:
caddy: myservice.netgrimoire.com
caddy.reverse_proxy: myservice:8080
caddy.import_1: crowdsec
caddy.import_2: authentik
monitor.name: MyService
monitor.url: https://myservice.netgrimoire.com
homepage.group: NetGrimoire
homepage.name: MyService
homepage.icon: myservice.png
homepage.href: https://myservice.netgrimoire.com
homepage.description: My service description
diun.enable: "true"
networks:
netgrimoire:
external: true
```
---
## Volume Path Rules
| Path type | Example | Placement constraint |
|---|---|---|
| `/DockerVol/` | `/DockerVol/myservice:/data` | `node.hostname` **required** |
| `/data/nfs/znas/` | `/data/nfs/znas/Docker/myservice:/data` | `node.hostname` **forbidden** |
Valid hostnames for `node.hostname`: `docker3`, `docker4`, `docker5`, `znas`, `dockerpi1`
---
## Identity Rules
**Method 1** — LinuxServer.io and homelab images (preferred):
```yaml
environment:
PUID: "1964"
PGID: "1964"
```
**Method 2** — Official Docker Hub images:
```yaml
user: "1964:1964"
```
**Exemption** — Images that manage their own users (Authentik, MailCow):
```yaml
labels:
gremlin.uid.exempt: "true"
gremlin.uid.reason: "Authentik manages its own internal user context"
```
---
## Caddy Label Rules
```yaml
caddy: myservice.netgrimoire.com # hostname only — no https:// prefix
caddy.reverse_proxy: myservice:8080 # service name and port — no IP addresses
caddy.import_1: crowdsec # mandatory
caddy.import_2: authentik # mandatory
```
Services without a public URL (internal sidecars, databases):
```yaml
gremlin.caddy.skip: "true"
```
---
## Monitor Labels
Gremlin writes monitor endpoints to Gatus after each successful deploy.
```yaml
monitor.name: MyService # display name in Gatus
monitor.url: https://myservice.netgrimoire.com
monitor.type: http # optional: http | tcp | ping | dns (default: http)
monitor.interval: "60" # optional: seconds, minimum 20 (default: 60)
```
Services that should not be monitored:
```yaml
gremlin.monitor.skip: "true"
```
TCP example (for non-HTTP services):
```yaml
monitor.type: tcp
monitor.url: myservice:5432
```
---
## Homepage Labels
```yaml
homepage.group: Media # dashboard group
homepage.name: MyService # display name
homepage.icon: myservice.png # icon filename
homepage.href: https://myservice.netgrimoire.com
homepage.description: Brief description
```
Services that should not appear on Homepage:
```yaml
gremlin.homepage.skip: "true"
```
> **Auto-fix note:** If homepage labels are missing, Gremlin derives them from the caddy: label and service name. Group defaults to "New", icon defaults to "servicename.png". Review and correct after auto-fix.
---
## Gremlin Directives
All directives go inside `deploy.labels`. All are opt-out — a stack with no `gremlin.*` labels gets full treatment.
### Pipeline Control
```yaml
gremlin.enable: "true"
# Set false to have Gremlin ignore this file entirely on push.
# Default: true
gremlin.checks: "all"
# Comma-separated checker IDs to run, or "all".
# Example: "swarm-syntax,identity,caddy"
# Default: all
gremlin.checks.skip: ""
# Comma-separated checker IDs to skip.
# Example: "homepage,monitor"
# Default: (none)
```
### Auto-fix Control
```yaml
gremlin.autofix: "true"
# Set false to disable all auto-fixing.
# Default: true
gremlin.autofix.skip: "false"
# Set true to notify but never attempt to fix.
# Default: false
gremlin.autofix.skip_fields: ""
# Comma-separated fields to skip during fix.
# Example: "hostname,uid"
# Default: (none)
```
### Deploy Control
```yaml
gremlin.deploy: "true"
# Set false to run checks and fixes but never deploy.
# Use for test stacks or stacks managed manually.
# Default: true
gremlin.deploy.strategy: "stack"
# Deployment method. Values: stack | helm | kubectl
# Default: stack
```
### Identity Exemptions
```yaml
gremlin.uid.exempt: "false"
# Set true to skip PUID/PGID/user checks.
# Use for images that manage their own users.
# Default: false
gremlin.uid.reason: ""
# Documents why uid.exempt is set.
# Required when uid.exempt is true.
```
### Placement Control
```yaml
gremlin.arm.allow: "false"
# Set true to allow ARM/Pi deployment.
# Removes ARM exclusion constraints.
# Default: false
```
### Service-level Skip Labels
```yaml
gremlin.caddy.skip: "false" # skip Caddy label validation
gremlin.homepage.skip: "false" # skip Homepage label validation
gremlin.monitor.skip: "false" # skip monitor label validation
gremlin.diun.skip: "false" # skip Diun label validation
gremlin.network.skip: "false" # skip network validation (whole stack)
```
### Ollama Context
```yaml
gremlin.context: ""
# Free text passed to Ollama audit as ground truth.
# Ollama will not flag anything the context explains.
# Example: "OIDC_CLIENT_SECRET in plain text is intentional — no secrets manager in use"
```
### Notification Control
```yaml
gremlin.notify: "true" # false = suppress all ntfy for this stack
gremlin.notify.level: "all" # all | failures | none
```
---
## Checker IDs
Use these IDs with `gremlin.checks` and `gremlin.checks.skip`:
| ID | What it checks |
|---|---|
| `swarm-syntax` | Forbidden fields: version, container_name, hostname, restart, depends_on, dnsrr |
| `identity` | PUID/PGID 1964 or user: "1964:1964" |
| `network` | netgrimoire overlay network |
| `placement` | ARM exclusions, DockerVol/hostname rules, restart_policy |
| `caddy` | caddy: label, reverse_proxy, import_1/import_2 |
| `homepage` | group, name, icon, href, description |
| `monitor` | monitor.name, monitor.url, optional type/interval |
| `legacy-labels` | Flags kuma.* labels for removal |
| `diun` | diun.enable: "true" |
---
## Common Patterns
### Internal sidecar (database, cache)
```yaml
postgres:
image: postgres:15
environment:
POSTGRES_USER: myapp
POSTGRES_PASSWORD: secret
volumes:
- /DockerVol/myapp/postgres:/var/lib/postgresql/data
networks:
- netgrimoire
deploy:
restart_policy:
condition: any
delay: 5s
max_attempts: 3
window: 120s
placement:
constraints:
- node.platform.arch != aarch64
- node.platform.arch != arm
- node.hostname == docker4
labels:
gremlin.caddy.skip: "true"
gremlin.homepage.skip: "true"
gremlin.monitor.skip: "true"
diun.enable: "true"
```
### Test stack (never deployed)
```yaml
labels:
gremlin.deploy: "false"
# ... other labels
```
### ARM/Pi service
```yaml
labels:
gremlin.arm.allow: "true"
# ... other labels
placement:
constraints:
- node.hostname == dockerpi1
```
### Image requiring root
```yaml
labels:
gremlin.uid.exempt: "true"
gremlin.uid.reason: "Image requires root — does not support PUID/PGID"
# ... other labels
```
---
## Forbidden Fields
These fields are automatically removed by Gremlin:
| Field | Reason |
|---|---|
| `version:` (top-level) | Obsolete in Compose v3 |
| `container_name:` | Conflicts with Swarm service naming |
| `hostname:` (service-level) | Conflicts with Swarm DNS |
| `restart:` (service-level) | Use `deploy.restart_policy` instead |
| `depends_on:` | Not supported in Swarm mode |
| `links:` | Not supported in Swarm mode |
These fields cause an **unfixable** block:
| Field | Reason |
|---|---|
| `endpoint_mode: dnsrr` | Breaks internal DNS resolution |
| Missing `deploy:` block | File treated as plain Compose, not Swarm |
---
## Troubleshooting
**"Missing deploy: block" — file skipped as non-Swarm**
Your compose file has no `deploy:` section. Add a `deploy:` block to each service for Swarm compatibility.
**"uses /DockerVol/ but has no node.hostname constraint" — unfixable**
Add a `node.hostname` constraint to your `deploy.placement.constraints`. Gremlin cannot guess which node to pin it to.
**Ollama keeps blocking on legitimate config**
Add `gremlin.context` to explain the situation. Ollama treats context as ground truth and will not flag it.
**Auto-fix loop — fixes applied but same issues keep appearing**
The fixer is finding the labels but the checker isn't recognizing them after insertion. Check label indentation — labels inside `deploy.labels` must be indented 8 spaces.
**Deploy skipped every time**
Check `gremlin.deploy` — if set to `"false"` the pipeline validates and fixes but never deploys.
---
## Related
- [Gremlin CI/CD Pipeline](gremlin-cicd-wiki.md)
- [NetGrimoire Stack Standards](stack-standards.md)
- [Gatus](gatus.md)