---
title: Gremlin CI/CD Pipepline
description: N8N with LLAMA
published: true
date: 2026-05-06T21:08:52.500Z
tags: 
editor: markdown
dateCreated: 2026-05-03T04:16:15.155Z
---

---
title: Gremlin CI/CD — Operations Guide
description: Complete operations reference for the Gremlin CI/CD pipeline. Use this to rebuild or onboard into a new project.
published: true
date: 2026-05-06
tags: gremlin, cicd, n8n, docker, swarm, compose, netgrimoire
editor: markdown
dateCreated: 2026-05-06
---

# Gremlin CI/CD — Operations Guide

> **NetGrimoire Infrastructure Reference**
> Complete reference for rebuilding, reconfiguring, or onboarding the Gremlin CI/CD pipeline in a new project. Covers infrastructure requirements, n8n setup, Forgejo configuration, service account setup, and the full node chain.

---

## Overview

Gremlin CI/CD is an n8n workflow that watches the `traveler/services` Forgejo repository for pushes, validates Docker Swarm stacks and Compose files against NetGrimoire standards, auto-fixes violations, prepares volumes, deploys via SSH, and syncs Caddy and Gatus configuration. All automated operations are attributed to the `gremlin` service account.

**Pipeline triggers:** any push to `swarm/**/*.yaml` or `compose/**/*.yaml` in `traveler/services`.

**Pipeline outputs:** validated and deployed stacks, updated Caddy static file, updated Gatus config, deploy log row in `traveler/Netgrimoire/Logs/<stack>.md`, ntfy notifications.

---

## Infrastructure Requirements

### Nodes

| Node | Role | IP |
|---|---|---|
| `znas` | Swarm manager, Caddy host, primary deploy target | `192.168.5.10` |
| `docker3` | Swarm worker | `192.168.5.15` |
| `docker4` | Swarm worker, n8n host, Ollama host | `192.168.5.16` |
| `docker5` | Swarm worker | `192.168.5.18` |
| `dockerpi1` | Swarm worker (ARM) | `192.168.5.8` |

### Required Services

| Service | Location | Purpose |
|---|---|---|
| n8n | `docker4` | Pipeline execution |
| Forgejo | `git.netgrimoire.com` | Git hosting, webhook source |
| Ollama | `docker4` | Audit model inference |
| ntfy | `ntfy.netgrimoire.com` | Notifications |
| Gatus | `znas` | Service monitoring |
| Caddy | `znas` | Reverse proxy |

---

## Service Account Setup

The `gremlin` service account must be created and configured on every node before the pipeline will function.

### Create the account

On each node:

```bash
sudo useradd -m -s /bin/bash gremlin
sudo usermod -aG docker gremlin
```

### SSH key

Generate once, distribute everywhere:

```bash
# On docker4 (n8n host) or your workstation
ssh-keygen -t ed25519 -C "gremlin@netgrimoire" -f ~/.ssh/gremlin_ed25519 -N ""
```

Copy the public key to every node:

```bash
for NODE in znas docker3 docker4 docker5 dockerpi1; do
  ssh-copy-id -i ~/.ssh/gremlin_ed25519.pub gremlin@$NODE
done
```

The private key must be placed at the path configured in `gremlin/config.yaml` (default `/home/gremlin/.ssh/id_ed25519`) inside the n8n container. Mount it via the n8n stack:

```yaml
volumes:
  - /DockerVol/n8n/keys:/home/gremlin/.ssh:ro
```

### Passwordless sudo

On each node, create `/etc/sudoers.d/gremlin-deploy`:

```
gremlin ALL=(ALL) NOPASSWD: /bin/mkdir, /bin/chown, /bin/chmod
```

On `znas` additionally:

```
gremlin ALL=(ALL) NOPASSWD: /usr/bin/docker
```

### Forgejo account

Create a `gremlin` local account in Forgejo. Grant collaborator write access to:
- `traveler/services` — stack files
- `traveler/Netgrimoire` — deploy logs

Generate two tokens:
- **Read token** — for fetching files (`FORGEJO_TOKEN`)
- **Write token** — for committing fixes and logs (`FORGEJO_WRITE_TOKEN`)

---

## n8n Stack Configuration

### Required environment variables

Add to the n8n stack compose `environment:` block:

```yaml
environment:
  # Forgejo
  FORGEJO_TOKEN: <gremlin read token>
  FORGEJO_WRITE_TOKEN: <gremlin write token>
  FORGEJO_URL: https://forgejo.netgrimoire.com

  # ntfy
  NTFY_URL: https://ntfy.netgrimoire.com

  # Node IPs (overlay DNS gives overlay IPs — store host IPs explicitly)
  ZNAS_IP: 192.168.5.10
  DOCKER3_IP: 192.168.5.15
  DOCKER4_IP: 192.168.5.16
  DOCKER5_IP: 192.168.5.18
  DOCKERPI1_IP: 192.168.5.8

  # Required for SSH and child_process in Code nodes
  NODE_FUNCTION_ALLOW_BUILTIN: child_process
```

### Volume mounts

```yaml
volumes:
  - /DockerVol/n8n:/home/node/.n8n
  - /DockerVol/n8n/keys:/home/gremlin/.ssh:ro
```

---

## Gremlin Config File

The pipeline reads runtime configuration from `gremlin/config.yaml` in the `traveler/services` repo. This file is fetched on every push and overrides hardcoded defaults.

```yaml
# gremlin/config.yaml
version: "2026-04-1"
deploy: true
autofix: true
ollama_model: "qwen2.5-coder:14b"
ollama_audit_model: "gemma3:4b"
ntfy_alerts_topic: "gremlin-alerts"
ntfy_monitor_topic: "gremlin-watch"
checks_skip: ""
maintenance: false
maintenance_message: "Gremlin is in maintenance mode"
ssh_key_path: "/home/gremlin/.ssh/id_ed25519"
repo_path: "/home/gremlin/services"
forgejo_owner: "traveler"
forgejo_repo: "services"
gatus_config_path: "/DockerVol/gatus/config/config.yaml"
caddy_file_path: "swarm/stack/caddy/Caddyfile"
caddy_static_path: "/export/Docker/caddy/Caddyfile"
```

Set `maintenance: true` to silently drop all pushes and send a single ntfy notification.

---

## Repository Structure

```
traveler/services/
  swarm/
    stack/
      caddy/
        Caddyfile               ← static Caddy config (managed by Gremlin Caddy Sync)
      <stackname>/
        <stackname>.yaml        ← Swarm stack file
  compose/
    <node>/
      <stackname>.yaml          ← Compose file for that node
      <stackname>-override.yaml ← Auto-generated by Gremlin (override mode)
  gremlin/
    config.yaml                 ← Pipeline runtime config
    logs/                       ← (unused — logs now in traveler/Netgrimoire)

traveler/Netgrimoire/
  Logs/
    <stackname>.md              ← Deploy log per stack (table + detail)

gatus/
  config.yaml                   ← Gatus monitor config (managed by Gremlin Gatus Sync)
```

---

## Forgejo Webhook

In Forgejo → `traveler/services` → Settings → Webhooks → Add Webhook:

| Field | Value |
|---|---|
| Target URL | `https://n8n.netgrimoire.com/webhook/gremlin-cicd` |
| Content type | `application/json` |
| Secret | _(optional, not currently validated)_ |
| Trigger | Push events only |

---

## Pipeline Node Chain

### Swarm path

```
Forgejo Push Webhook
  → Parse Push Payload          extract files, branch, SHA, pusher, commit directives
  → Build Envelope              fetch file from Forgejo, detect type, parse directives
  → Switch2                     route: Compose / Swarm / Pocket / fallback
  → Check: Swarm Syntax         forbidden keys (version, container_name, restart, depends_on)
  → Check: Identity             PUID/PGID or user: 1964:1964 per service
  → Check: Network              netgrimoire network on every service
  → Check: Placement            ARM exclusions, node.hostname constraints, DockerVol placement
  → Check: Caddy                caddy:, import_1/2, reverse_proxy per service
  → Check: Homepage             all 5 homepage.* labels per service
  → Check: Monitor              monitor.name, monitor.url (http://) per service
  → Check: Legacy Labels        kuma.* labels (flagged for removal)
  → Check: Version              gremlin.version label per service
  → Check: Diun                 diun.enable: "true" per service
  → Evaluate Checks             hasFailed, canFix flags
  → Switch (hasFailed)
      true  → ntfy: Blocked — Schema
              → Switch1 (canFix)
                  true  → Fix chain (Syntax→Identity→Network→Placement→Caddy→
                                     Homepage→Monitor→Legacy→Version→Diun)
                          → Commit Fixed File
                          → ntfy: Auto-fixed
                  false → (dead end — manual fix required)
      false → Ollama Audit
              → Ollama Failed?
                  true  → ntfy: Blocked — Ollama
                  false → Deploy Gate (deploy directive)
                              enabled  → Prepare Volumes
                                         → Git Pull + Deploy
                                         → Gatus Sync
                                         → ntfy: Deploy Complete (+ log write)
                              disabled → ntfy: Deploy Skipped
```

### Compose path

```
  → Check: Compose Syntax       forbidden keys (version, container_name, depends_on)
  → Check: Compose Identity     PUID/PGID per service (no restart_policy check)
  → Check: Compose Network      netgrimoire network
  → Check: Compose Caddy        caddy labels
  → Check: Compose Homepage     homepage labels
  → Check: Compose Monitor      monitor labels
  → Check: Compose Diun         diun.enable
  → Check: Compose Version      gremlin.version
  → Evaluate Compose Checks
  → Switch: Compose (hasFailed)
      true  → ntfy: Compose Blocked
              → Switch: Compose1 (canFix)
                  true  → Fix chain (writes to original or override file)
                          → Commit Compose Fixed File
                          → ntfy: Compose Auto-fixed
      false → Compose Deploy Gate
                  enabled  → Compose Prepare Volumes (SSH to composeNode)
                             → Compose Deploy (docker compose up -d on target node)
                             → Compose Caddy Sync (upsert static Caddyfile + git commit + cp)
                             → Compose Gatus Sync
                             → ntfy: Compose Deploy Complete (+ log write)
                  disabled → ntfy: Compose Deploy Skipped
```

---

## Caddy Sync

**Swarm:** caddy-docker-proxy reads labels directly from Swarm services. No Gremlin involvement needed for Swarm caddy entries — they are auto-generated by caddy-docker-proxy.

**Compose:** Gremlin reads `caddy:` and related labels from the compose file, builds a Caddyfile block, upserts it into `swarm/stack/caddy/Caddyfile` in Forgejo, commits it, then SSHes to `znas` to `git pull` and `cp` the file to `/export/Docker/caddy/Caddyfile`. Caddy reloads via inotify and the admin API.

The Caddyfile path in the repo is configured via `caddy_file_path` in `gremlin/config.yaml`.

---

## Gatus Sync

Both Swarm and Compose pipelines read `monitor.name` and `monitor.url` labels, build Gatus endpoint blocks, upsert them into `gatus/config.yaml` in `traveler/services`, commit via the Forgejo API, then SSH to `znas` to `git pull` and copy the file to the configured `gatus_config_path`.

Monitor URLs must use `http://servicename:port` — Gatus is on the `netgrimoire` overlay and reaches services directly without going through Caddy.

---

## Deploy Logs

Each stack gets a log file at `traveler/Netgrimoire/Logs/<stackname>.md`. The file is created on first deploy and a row is appended on each subsequent deploy. The ntfy message body is also written as a detail block below the table row.

**Swarm table format:**
```
| Timestamp | Commit | Branch | Pusher | Outcome | Volumes |
```

**Compose table format:**
```
| Timestamp | Commit | Branch | Pusher | Outcome | Node | Volumes |
```

Failed deploys are also logged with the error output.

---

## Volume Management

Gremlin's Prepare Volumes node SSHes to the target node and handles `/DockerVol/` and `/data/nfs/` paths found in the stack file.

| Scenario | Action |
|---|---|
| Directory does not exist | `mkdir -p` + `chown -R 1964:1964` + `chmod -R 775` |
| Directory exists, correct perms | Silent — no action |
| Directory exists, wrong perms | Added to `permWarnings` — manual fix command included in ntfy and deploy log |
| Service has `gremlin.uid.exempt: "true"` | `mkdir -p` only — no chown or chmod |

Permission warnings appear in both the ntfy Deploy Complete message and the deploy log detail block.

---

## Troubleshooting

### Pipeline not triggering

- Verify webhook is active in Forgejo → `traveler/services` → Settings → Webhooks
- Check n8n workflow is active
- Check that changed files are under `swarm/` or `compose/` paths

### SSH failures

- Verify `gremlin` has SSH key access to the target node: `ssh -i /DockerVol/n8n/keys/id_ed25519 gremlin@<node_ip>`
- Verify `NODE_FUNCTION_ALLOW_BUILTIN=child_process` is set in n8n env
- Check that the key mount is read-only: `/DockerVol/n8n/keys:/home/gremlin/.ssh:ro`

### Forgejo API 404 on file fetch

- Verify `FORGEJO_TOKEN` has read access to `traveler/services`
- Verify file path in the push payload matches actual repo path
- Check that `gremlin/config.yaml` exists in the repo

### Caddy Sync 404

- Verify `caddy_file_path` in `gremlin/config.yaml` matches the actual path in the repo (`swarm/stack/caddy/Caddyfile`)
- Verify the Caddyfile exists at that path in Forgejo

### Gatus corruption

- The upsert logic splits on `  - name:` boundaries. If the config has non-standard indentation the split may fail. Reset the file manually in Forgejo and push again.

### stat command failing in Prepare Volumes

- The stat calls use three separate SSH invocations (`stat -c %U`, `stat -c %G`, `stat -c %a`) to avoid shell quoting issues. If these fail, check that `stat` is available on the target node.

---

## Rebuilding in a New Project

1. Create service account `gremlin` on all nodes with docker group and passwordless sudo
2. Generate SSH keypair, distribute public key to all nodes
3. Place private key at `/DockerVol/n8n/keys/id_ed25519` on the n8n host
4. Create `gremlin` Forgejo account, grant write access to services and docs repos
5. Generate read and write tokens, add to n8n environment variables
6. Add all node IPs and other env vars to n8n stack compose, redeploy n8n
7. Import `gremlin-cicd-v2.json` workflow into n8n, activate it
8. Create `gremlin/config.yaml` in the services repo with your values
9. Create `gatus/config.yaml` in the services repo (empty endpoints block)
10. Create `Logs/` directory in the docs repo (`touch Logs/.gitkeep && git push`)
11. Add webhook to Forgejo pointing at `https://n8n.netgrimoire.com/webhook/gremlin-cicd`
12. Push a test stack file and verify the pipeline fires

