---
title: Gremlin CI/CD User Guide
description: 
published: true
date: 2026-05-06T21:10:05.745Z
tags: 
editor: markdown
dateCreated: 2026-05-03T04:16:19.938Z
---

---
title: Gremlin CI/CD — User Guide
description: Day-to-day usage guide for the Gremlin CI/CD pipeline. How to write stacks, use directives, interpret notifications, and manage deployments.
published: true
date: 2026-05-06
tags: gremlin, cicd, docker, swarm, compose, netgrimoire
editor: markdown
dateCreated: 2026-05-06
---

# Gremlin CI/CD — User Guide

> **NetGrimoire Infrastructure Reference**
> Day-to-day guide for working with the Gremlin CI/CD pipeline. Covers stack file conventions, directive usage, the auto-fix cycle, commit directives, notifications, and deploy logs.

---

## How the Pipeline Works

Push a `.yaml` file to `traveler/services` under `swarm/` or `compose/<node>/` and Gremlin takes over. It validates the file, fixes what it can automatically, prepares volume directories on the target node, deploys the stack, and notifies you via ntfy.

You never need to SSH to a node to deploy — push to Forgejo and Gremlin handles the rest.

---

## File Locations

| Type | Path | Example |
|---|---|---|
| Swarm stack | `swarm/stack/<name>/<name>.yaml` | `swarm/stack/wiki/wiki.yaml` |
| Compose file | `compose/<node>/<name>.yaml` | `compose/znas/namer.yaml` |
| Compose override | `compose/<node>/<name>-override.yaml` | auto-generated |
| Gremlin config | `gremlin/config.yaml` | pipeline settings |

---

## Swarm Stack Standards

Every service in a Swarm stack must comply with these standards. Gremlin will auto-fix most violations; unfixable ones block the deploy with an ntfy alert.

### Forbidden fields (auto-removed)

```yaml
version: "3.8"         # ← removed
container_name: foo    # ← removed
hostname: foo          # ← removed
restart: unless-stopped # ← removed (use restart_policy)
depends_on:            # ← removed
endpoint_mode: dnsrr   # ← removed
```

### Required per service

```yaml
services:
  myservice:
    image: myimage:latest
    networks:
      - netgrimoire
    environment:
      PUID: "1964"        # ← required (or user: "1964:1964")
      PGID: "1964"
    deploy:
      restart_policy:
        condition: any
        delay: 5s
        max_attempts: 3
        window: 120s
      placement:
        constraints:
          - node.platform.arch != aarch64   # ← required (unless arm.allow)
          - node.platform.arch != arm
      labels:
        gremlin.version: "2026-04-1"        # ← required
        caddy: myservice.netgrimoire.com    # ← required (or caddy.skip)
        caddy.reverse_proxy: myservice:8080
        caddy.import_1: crowdsec
        caddy.import_2: authentik
        homepage.group: "My Group"          # ← required (or homepage.skip)
        homepage.name: "My Service"
        homepage.icon: "myservice.png"
        homepage.href: "https://myservice.netgrimoire.com"
        homepage.description: "My service description"
        monitor.name: "My Service"          # ← required (or monitor.skip)
        monitor.url: "http://myservice:8080"
        diun.enable: "true"                 # ← required (or diun.skip)

networks:
  netgrimoire:
    external: true
```

### DockerVol placement constraint

Any service with a `/DockerVol/` volume must also have:

```yaml
placement:
  constraints:
    - node.hostname == znas   # or whichever node owns that DockerVol path
```

---

## Compose File Standards

Compose files follow the same label standards as Swarm with these differences:

- No `deploy:` block required (no restart_policy, no ARM exclusions, no placement)
- `restart:` is valid and left untouched
- Deploy is via `docker compose up -d` on the target node (derived from file path)
- Caddy entries are written to the static Caddyfile by Gremlin

```yaml
services:
  myservice:
    image: myimage:latest
    networks:
      - netgrimoire
    environment:
      PUID: "1964"
      PGID: "1964"
    restart: unless-stopped
    labels:
      gremlin.version: "2026-04-1"
      caddy: myservice.netgrimoire.com
      caddy.reverse_proxy: myservice:8080
      caddy.import_1: crowdsec
      caddy.import_2: authentik
      homepage.group: "My Group"
      homepage.name: "My Service"
      homepage.icon: "myservice.png"
      homepage.href: "https://myservice.netgrimoire.com"
      homepage.description: "My service description"
      monitor.name: "My Service"
      monitor.url: "http://myservice:8080"
      diun.enable: "true"

networks:
  netgrimoire:
    external: true
```

---

## The Auto-Fix Cycle

When Gremlin detects fixable violations it:

1. Fixes them in memory
2. Commits the corrected file back to Forgejo as the `gremlin` account
3. Sends ntfy: **Gremlin: Auto-fixed** listing what changed
4. The fix commit triggers the webhook again
5. The pipeline re-runs on the fixed file
6. If it passes, the stack deploys

You will receive two ntfy messages for an auto-fixed deploy: the fix notification and the deploy complete notification.

### What gets auto-fixed

| Violation | Fix |
|---|---|
| `version:` key present | Removed |
| `container_name:` present | Removed |
| `restart:` present (Swarm) | Removed |
| `depends_on:` present | Removed |
| Missing `PUID`/`PGID` | Added to environment block |
| Missing network | Added |
| Missing `caddy.import_1: crowdsec` | Added |
| Missing `caddy.import_2: authentik` | Added |
| Missing `caddy.reverse_proxy` | Added (derived from ports or gremlin.port) |
| Missing ARM exclusions | Added to placement constraints |
| Missing homepage labels | Added with placeholder values |
| Missing `monitor.name`/`monitor.url` | Added (derived from ports or gremlin.port) |
| Missing `diun.enable` | Added |
| Missing `gremlin.version` | Stamped |
| `kuma.*` labels present | Removed |

### What cannot be auto-fixed (blocks deploy)

| Violation | Why |
|---|---|
| Missing `caddy:` label | Gremlin cannot invent a hostname |
| Invalid `node.hostname` constraint | Must reference a real cluster node |

---

## Override Mode for Vendor Files

For downloaded vendor compose files you do not want Gremlin to modify, add to any service label:

```yaml
labels:
  gremlin.autofix.mode: "override"
```

Gremlin will:
- Never touch the original file
- Write all fixes to `<filename>-override.yaml` in the same directory
- Deploy with `docker compose -f original.yaml -f original-override.yaml up -d`
- Report unfixable issues (like `version:` key) as manual fixes needed in the original

The override file is committed to Forgejo automatically and updated on subsequent pushes.

---

## Commit Directives

Control the pipeline from the commit message without modifying any file. Use this for vendor files, one-off overrides, or emergency controls.

**Syntax:** append `[gremlin: key=value key2=value2]` anywhere in the commit message.

```bash
git commit -m "deploy stash after config change [gremlin: deploy=true autofix=false checks.skip=all]"
git commit -m "WIP refactor [gremlin: deploy=false]"
git commit -m "bump image [gremlin: notify=false]"
git commit -m "fix label [gremlin: checks.skip=caddy,homepage]"
```

**Commit directives override file directives and config defaults** — they are the highest priority in the pipeline.

### Supported commit directives

| Directive | Values | Effect |
|---|---|---|
| `deploy` | `true` / `false` | Enable or disable deploy for this push |
| `autofix` | `true` / `false` | Enable or disable auto-fix |
| `checks.skip` | comma-separated checker IDs | Skip specific checks |
| `notify` | `true` / `false` | Enable or disable ntfy notifications |
| `notify.level` | `all` / `errors` | Notification verbosity |
| `enable` | `true` / `false` | Enable or disable the entire pipeline for this push |

### Checker IDs for checks.skip

`syntax`, `identity`, `network`, `placement`, `caddy`, `homepage`, `monitor`, `legacy-labels`, `version`, `diun`

Compose checker IDs: `compose-syntax`, `compose-identity`, `compose-network`, `compose-caddy`, `compose-homepage`, `compose-monitor`, `compose-diun`, `compose-version`

---

## Notifications

All ntfy notifications go to `gremlin-alerts` (errors, blocks, fixes) or `gremlin-watch` (deploy complete, skipped).

| Message | Topic | Meaning |
|---|---|---|
| Gremlin: Blocked — Schema | alerts | Unfixable violation — manual fix required |
| Gremlin: Auto-fixed | alerts | Violations fixed and recommitted |
| Gremlin: Blocked — Ollama | alerts | Ollama audit flagged a problem |
| Gremlin: Deploy Complete | watch | Stack deployed successfully |
| Gremlin: Deploy Skipped | watch | Deploy disabled via directive |
| Gremlin: Deploy Failed | alerts | Deploy command failed — check SSH/Docker |
| Gremlin: Compose Blocked | alerts | Compose file violation |
| Gremlin: Compose Auto-fixed | alerts | Compose violations fixed |
| Gremlin: Compose Deploy | watch | Compose stack deployed |
| Gremlin: Maintenance Mode | alerts | Push received during maintenance mode |

### Permission warnings in ntfy

If a volume directory exists with wrong ownership or permissions, the Deploy Complete message will include:

```
Manual permission fixes needed on znas:
• /DockerVol/myservice (current: root:root 755)
  sudo chown -R 1964:1964 /DockerVol/myservice && sudo chmod -R 775 /DockerVol/myservice
```

Gremlin never changes permissions on existing directories — you must apply these manually.

---

## Deploy Logs

Each stack has a log at `traveler/Netgrimoire/Logs/<stackname>.md`. Every deploy (success or failure) appends a table row and a detail block containing the full ntfy message.

```markdown
# Deploy Log — wiki

| Timestamp | Commit | Branch | Pusher | Outcome | Volumes |
|---|---|---|---|---|---|
| 2026-05-06 00:15 UTC | abc1234 | master | traveler | ✅ deployed | 3 vol(s) |

**2026-05-06 00:15 UTC** — ✅ deployed
​```
wiki deployed successfully
Branch: master
Pushed by: traveler
SHA: abc1234
Gatus monitors: Wiki
​```
```

---

## Maintenance Mode

Set `maintenance: true` in `gremlin/config.yaml` to pause the pipeline. Any push will receive a single ntfy notification and be silently dropped. No deploys, no fixes, no errors.

```yaml
maintenance: true
maintenance_message: "Upgrading ZFS pool — pipeline paused"
```

Push the config change to activate, push again with `maintenance: false` to resume.

---

## Common Workflows

### Deploying a new stack

1. Write the stack file following the standards above
2. Place it at `swarm/stack/<name>/<name>.yaml` or `compose/<node>/<name>.yaml`
3. `git add`, `git commit`, `git push`
4. Watch ntfy — Gremlin will auto-fix any violations and deploy

### Redeploying an existing stack

Just push any change to the file. Even a comment or whitespace change triggers the pipeline.

Or use a commit directive to force a redeploy without changing the file:

```bash
git commit --allow-empty -m "redeploy wiki [gremlin: deploy=true]"
git push
```

### Deploying a vendor file without modifying it

1. Add `gremlin.autofix.mode: "override"` to one service label
2. Push — Gremlin creates the override file with all required labels
3. Subsequent pushes use the override file automatically

Or use commit directives to skip checks entirely:

```bash
git commit -m "deploy vendor stack [gremlin: autofix=false checks.skip=all deploy=true]"
```

### Checking what Gremlin did

- ntfy notifications for immediate status
- `traveler/Netgrimoire/Logs/<stackname>.md` for full history
- Forgejo commit history on `traveler/services` — look for `gremlin:` prefix commits
