traveler/Netgrimoire
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

New Grimoire

Browse source
This commit is contained in:
traveler 2026-04-12 09:53:51 -05:00
parent 77d589a13d
commit cc574f8aed
157 changed files with 29420 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

31
Ward-Grimoire/Notifications/Alert-Routing.md Normal file
View file

@ -0,0 +1,31 @@
---
title: Alert Routing
description: How security alerts flow through Netgrimoire
published: true
date: 2026-04-12T00:00:00.000Z
tags: ward, alerts, ntfy
editor: markdown
dateCreated: 2026-04-12T00:00:00.000Z
---
# Alert Routing
All Netgrimoire alerts route through self-hosted ntfy at `ntfy.netgrimoire.com`.
## ntfy Topics
| Topic | Source | Purpose |
|-------|--------|---------|
| `netgrimoire-diun` | DIUN | Docker image update notifications |
| `netgrimoire-media` | Sonarr, Radarr, SABnzbd | Download and media events |
| `netgrimoire-backup` | Kopia | Backup completion and errors |
| `gremlin-alerts` | n8n Kuma triage workflow | AI-analyzed service DOWN alerts |
| `gremlin-audits` | n8n Forgejo audit workflow | Weekly YAML audit summaries |
## Alert Sources
**OPNsense → ntfy:** CrowdSec HTTP plugin (`/usr/local/etc/crowdsec/notifications/ntfy.yaml`) + Monit script (`/usr/local/bin/ntfy-alert.sh`). See [OPNsense Alerts](/Ward-Grimoire/Notifications/OPNsense-Alerts).
**Uptime Kuma → Gremlin → ntfy:** Kuma webhook fires on DOWN/RECOVERED → n8n triage workflow → Ollama analysis (DOWN path only) → ntfy `gremlin-alerts`. See [Gremlin Kuma Triage](/Gremlin-Grimoire/Workflows/Kuma-Triage).
**DIUN → ntfy:** Docker image update watcher. Schedule: every 6 hours. Priority must be integer (1–5), not string `"default"`.

463
Ward-Grimoire/Notifications/OPNsense-Alerts.md Normal file
View file

@ -0,0 +1,463 @@
---
title: OpnSense - NTFY Integration
description: Security Notifications
published: true
date: 2026-02-23T22:00:46.462Z
tags:
editor: markdown
dateCreated: 2026-02-23T22:00:37.268Z
---
# OPNsense ntfy Alerts
**Service:** ntfy push notifications from OPNsense
**Host:** OPNsense firewall
**ntfy Server:** Your self-hosted ntfy instance on Netgrimoire
**Methods:** CrowdSec HTTP plugin · Monit custom script · Suricata EVE watcher
---
## Overview
OPNsense does not have a built-in ntfy notification channel, but there are three distinct integration points that together provide complete coverage:
| Method | What It Alerts On | Priority |
|---|---|---|
| **CrowdSec HTTP plugin** | Every IP ban decision CrowdSec makes | 🔴 Best for threat intel alerts |
| **Monit + curl script** | System health, service failures, Suricata EVE matches, login failures | 🔴 Best for operational alerts |
| **Suricata EVE watcher** | Suricata high-severity IDS hits (via Monit watching eve.json) | 🟡 Covered via Monit |
All three use your self-hosted ntfy instance. None require external services.
---
## Prerequisites
Before starting, confirm:
- ntfy is running and reachable at `https://ntfy.netgrimoire.com` (or your internal URL)
- ntfy topic created: e.g. `opnsense-alerts`
- If ntfy has auth enabled, have a token ready
- SSH access to OPNsense as root
---
## Method 1 — CrowdSec HTTP Notification Plugin
This is the cleanest integration for security alerts. CrowdSec has a built-in HTTP notification plugin. Every time it makes a ban decision — whether from community intel, a Suricata match passed through CrowdSec, or a brute-force detection — it POSTs to ntfy.
### Step 1 — Create the HTTP notification config
SSH into OPNsense and create the ntfy config file:
```bash
ssh root@192.168.3.4
```
```bash
cat > /usr/local/etc/crowdsec/notifications/ntfy.yaml << 'EOF'
# ntfy notification plugin for CrowdSec
# CrowdSec uses its built-in HTTP plugin pointed at ntfy
type: http
name: ntfy_default
log_level: info
# ntfy accepts plain POST body as the notification message
# format is a Go template — .[]Alert is the list of alerts
format: |
{{range .}}
🚨 CrowdSec Decision
Scenario: {{.Scenario}}
Attacker IP: {{.Source.IP}}
Country: {{.Source.Cn}}
Action: {{.Decisions | len}} x {{(index .Decisions 0).Type}}
Duration: {{(index .Decisions 0).Duration}}
{{end}}
url: https://ntfy.netgrimoire.com/opnsense-alerts
method: POST
headers:
Title: "CrowdSec Ban — OPNsense"
Priority: "high"
Tags: "rotating_light,shield"
# Uncomment and set token if ntfy auth is enabled:
# Authorization: "Bearer YOUR_NTFY_TOKEN"
# skip_tls_verify: false
EOF
```
> ⚠ Replace `https://ntfy.netgrimoire.com/opnsense-alerts` with your actual ntfy URL and topic. If ntfy is internal-only and OPNsense can reach it by hostname, the internal URL works fine.
### Step 2 — Register the plugin in profiles.yaml
Edit the CrowdSec profiles file to dispatch decisions to the ntfy plugin:
```bash
vi /usr/local/etc/crowdsec/profiles.yaml
```
Find the `notifications:` section of the default profile and add `ntfy_default`:
```yaml
name: default_ip_remediation
filters:
- Alert.Remediation == true && Alert.GetScope() == "Ip"
decisions:
- type: ban
duration: 4h
notifications:
- ntfy_default # ← add this line
on_success: break
```
> ✓ The `ntfy_default` name must match the `name:` field in the yaml file you created above exactly.
### Step 3 — Set correct file ownership
CrowdSec rejects plugins if the configuration file is not owned by the root user and root group. Ensure the file has the right permissions:
```bash
chown root:wheel /usr/local/etc/crowdsec/notifications/ntfy.yaml
chmod 600 /usr/local/etc/crowdsec/notifications/ntfy.yaml
```
### Step 4 — Restart CrowdSec and test
```bash
# Restart via OPNsense service manager (do NOT use systemctl/service directly)
# Go to: Services → CrowdSec → Settings → Apply
# Or from shell:
pluginctl -s crowdsec restart
```
Test by sending a manual notification:
```bash
cscli notifications test ntfy_default
```
You should receive a test push on your device within a few seconds.
Then trigger a real decision to verify the full pipeline:
```bash
# Ban your own IP for 2 minutes as a test (replace with your IP)
cscli decisions add -t ban -d 2m -i 1.2.3.4
# Watch for ntfy notification
# Remove the test ban:
cscli decisions delete -i 1.2.3.4
```
---
## Method 2 — Monit + curl Script
Monit is OPNsense's built-in service monitor. It can watch processes, files, system resources, and log patterns — and call a custom shell script when a condition is met. The script fires a curl POST to ntfy.
This covers things CrowdSec doesn't — service failures, high CPU, gateway down events, SSH login failures, disk usage, and Suricata EVE alerts.
### Step 2.1 — Create the ntfy alert script
```bash
cat > /usr/local/bin/ntfy-alert.sh << 'EOF'
#!/usr/local/bin/bash
# ntfy-alert.sh — called by Monit to send ntfy push notifications
# Monit provides variables: $MONIT_HOST, $MONIT_SERVICE,
# $MONIT_DESCRIPTION, $MONIT_EVENT
NTFY_URL="https://ntfy.netgrimoire.com/opnsense-alerts"
# NTFY_TOKEN="Bearer YOUR_NTFY_TOKEN" # uncomment if ntfy auth enabled
TITLE="${MONIT_HOST}: ${MONIT_SERVICE}"
MESSAGE="${MONIT_EVENT} — ${MONIT_DESCRIPTION}"
# Map Monit event types to ntfy priorities
case "$MONIT_EVENT" in
*"does not exist"*|*"failed"*|*"error"*)
PRIORITY="urgent"
TAGS="rotating_light,red_circle"
;;
*"changed"*|*"match"*)
PRIORITY="high"
TAGS="warning,yellow_circle"
;;
*"recovered"*|*"succeeded"*)
PRIORITY="default"
TAGS="white_check_mark,green_circle"
;;
*)
PRIORITY="default"
TAGS="bell"
;;
esac
curl -s \
-H "Title: ${TITLE}" \
-H "Priority: ${PRIORITY}" \
-H "Tags: ${TAGS}" \
-d "${MESSAGE}" \
"${NTFY_URL}"
# Uncomment for auth:
# curl -s \
# -H "Authorization: ${NTFY_TOKEN}" \
# -H "Title: ${TITLE}" \
# -H "Priority: ${PRIORITY}" \
# -H "Tags: ${TAGS}" \
# -d "${MESSAGE}" \
# "${NTFY_URL}"
EOF
chmod +x /usr/local/bin/ntfy-alert.sh
```
### Step 2.2 — Enable Monit
Navigate to **Services → Monit → Settings → General Settings**
| Setting | Value |
|---|---|
| Enabled | ✓ |
| Polling Interval | 30 seconds |
| Start Delay | 120 seconds |
| Mail Server | Leave blank (using script instead) |
Click **Save**.
### Step 2.3 — Add Service Tests
Navigate to **Services → Monit → Service Tests Settings** and add the following tests:
**Test 1 — Custom Alert via Script**
| Field | Value |
|---|---|
| Name | `ntfy_alert` |
| Condition | `failed` |
| Action | Execute |
| Path | `/usr/local/bin/ntfy-alert.sh` |
This is the reusable action that all other tests will invoke.
**Test 2 — Suricata EVE High Alert**
| Field | Value |
|---|---|
| Name | `SuricataHighAlert` |
| Condition | `content = "\"severity\":1"` |
| Action | Execute → `/usr/local/bin/ntfy-alert.sh` |
This watches for severity 1 (highest) alerts written to the Suricata EVE JSON log.
**Test 3 — Suricata Process Down**
| Field | Value |
|---|---|
| Name | `SuricataRunning` |
| Condition | `failed` |
| Action | Execute → `/usr/local/bin/ntfy-alert.sh` |
**Test 4 — CrowdSec Process Down**
| Field | Value |
|---|---|
| Name | `CrowdSecRunning` |
| Condition | `failed` |
| Action | Execute → `/usr/local/bin/ntfy-alert.sh` |
**Test 5 — SSH Login Failure**
| Field | Value |
|---|---|
| Name | `SSHFailedLogin` |
| Condition | `content = "Failed password"` |
| Action | Execute → `/usr/local/bin/ntfy-alert.sh` |
**Test 6 — OPNsense Web UI Login Failure**
| Field | Value |
|---|---|
| Name | `WebUILoginFail` |
| Condition | `content = "webgui"` |
| Action | Execute → `/usr/local/bin/ntfy-alert.sh` |
### Step 2.4 — Add Service Monitors
Navigate to **Services → Monit → Service Settings** and add:
**Monitor 1 — Suricata EVE Log (high alerts)**
| Field | Value |
|---|---|
| Name | `SuricataEVE` |
| Type | File |
| Path | `/var/log/suricata/eve.json` |
| Tests | `SuricataHighAlert` |
**Monitor 2 — Suricata Process**
| Field | Value |
|---|---|
| Name | `Suricata` |
| Type | Process |
| PID File | `/var/run/suricata.pid` |
| Tests | `SuricataRunning` |
| Restart Method | /usr/local/etc/rc.d/suricata restart |
**Monitor 3 — CrowdSec Process**
| Field | Value |
|---|---|
| Name | `CrowdSec` |
| Type | Process |
| Match | `crowdsec` |
| Tests | `CrowdSecRunning` |
**Monitor 4 — SSH Auth Log**
| Field | Value |
|---|---|
| Name | `SSHAuth` |
| Type | File |
| Path | `/var/log/auth.log` |
| Tests | `SSHFailedLogin` |
**Monitor 5 — System Resources (optional)**
| Field | Value |
|---|---|
| Name | `System` |
| Type | System |
| Tests | `ntfy_alert` (on resource threshold exceeded) |
Click **Apply** after adding all services.
### Step 2.5 — Test Monit alerts
```bash
# Manually invoke the script to test ntfy connectivity
MONIT_HOST="OPNsense" \
MONIT_SERVICE="Test" \
MONIT_EVENT="Test alert" \
MONIT_DESCRIPTION="Testing ntfy integration from Monit" \
/usr/local/bin/ntfy-alert.sh
```
You should receive a push notification immediately.
---
## Alert Topics & Priority Mapping
Consider using separate ntfy topics to filter notifications by type on your device:
| Topic | Used For | Suggested ntfy Priority |
|---|---|---|
| `opnsense-alerts` | CrowdSec bans, Suricata high hits | high / urgent |
| `opnsense-health` | Monit service failures, process restarts | high |
| `opnsense-info` | Service recoveries, status changes | default / low |
To use separate topics, change the `NTFY_URL` in the Monit script and the `url:` in the CrowdSec config accordingly.
---
## ntfy Priority Reference
ntfy supports five priority levels that map to different notification behaviors on Android/iOS:
| ntfy Priority | Numeric | Behavior |
|---|---|---|
| `min` | 1 | No notification, no sound |
| `low` | 2 | Notification, no sound |
| `default` | 3 | Notification with sound |
| `high` | 4 | Notification with sound, bypasses DND |
| `urgent` | 5 | Phone rings through DND, repeated |
For firewall alerts: use `urgent` for process failures and `high` for IDS/ban events. Reserve `urgent` sparingly to avoid alert fatigue.
---
## Keeping Config Persistent Across Upgrades
OPNsense upgrades can overwrite files in certain paths. The safest locations for persistent custom files:
| File | Location | Persistent? |
|---|---|---|
| ntfy-alert.sh | `/usr/local/bin/ntfy-alert.sh` | ✓ Yes — not touched by upgrades |
| CrowdSec ntfy.yaml | `/usr/local/etc/crowdsec/notifications/ntfy.yaml` | ✓ Yes — plugin config directory |
| CrowdSec profiles.yaml | `/usr/local/etc/crowdsec/profiles.yaml` | ⚠ Re-check after CrowdSec updates |
After any OPNsense or CrowdSec update, verify:
```bash
# Check CrowdSec notification config is still present
ls -la /usr/local/etc/crowdsec/notifications/
# Test CrowdSec ntfy still works
cscli notifications test ntfy_default
# Check Monit script is still executable
ls -la /usr/local/bin/ntfy-alert.sh
```
---
## Troubleshooting
**No notification received from CrowdSec test:**
```bash
# Check CrowdSec logs for plugin errors
tail -50 /var/log/crowdsec.log | grep -i ntfy
tail -50 /var/log/crowdsec.log | grep -i notification
# Verify ntfy URL is reachable from OPNsense
curl -v -d "test" https://ntfy.netgrimoire.com/opnsense-alerts
# Check profiles.yaml has ntfy_default in notifications section
grep -A5 "notifications:" /usr/local/etc/crowdsec/profiles.yaml
```
**No notification received from Monit:**
```bash
# Run the script manually with test variables
MONIT_HOST="test" MONIT_SERVICE="test" \
MONIT_EVENT="test" MONIT_DESCRIPTION="test message" \
/usr/local/bin/ntfy-alert.sh
# Check Monit is running
ps aux | grep monit
# Check Monit logs
tail -50 /var/log/monit.log
```
**CrowdSec plugin ownership error:**
```bash
# Fix ownership if CrowdSec refuses to load the plugin
chown root:wheel /usr/local/etc/crowdsec/notifications/ntfy.yaml
ls -la /usr/local/etc/crowdsec/notifications/
```
**ntfy auth failing:**
```bash
# Test with token manually
curl -H "Authorization: Bearer YOUR_TOKEN" \
-H "Title: Test" \
-d "Auth test" \
https://ntfy.netgrimoire.com/opnsense-alerts
```
---
## Related Documentation
- [OPNsense Firewall](./opnsense-firewall) — parent firewall documentation
- [CrowdSec](./crowdsec) — threat intelligence engine sending these alerts
- [Suricata IDS/IPS](./suricata-ids-ips) — source of EVE alerts watched by Monit
- [ntfy](./ntfy) — self-hosted notification server on Netgrimoire

122
Ward-Grimoire/Notifications/ntfy.md Normal file
View file

@ -0,0 +1,122 @@
# ntfy
## Overview
The ntfy stack is a Docker Swarm-based service that provides push notifications in NetGrimoire. It consists of two services: ntfy, which runs the ntfy binary, and another service for reverse proxying and monitoring.
---
## Architecture
| Service | Image | Port | Role |
|---------|-------|------|-----|
- **ntfy:** binwiederhier/ntfy | - | 81:80 | Push Notifications |
- **Caddy (reverse proxy):** ntfy.netgrimoire.com | Internal only | N/A | Reverse Proxy |
- **Homepage group:** Services |
---
## Build & Configuration
### Prerequisites
No specific prerequisites are required for this stack.
### Volume Setup
```bash
mkdir -p /DockerVol/ntfy/cache
mkdir -p /DockerVol/ntfy/etc
chown -R ntfy:ntfy /DockerVol/ntfy
```
### Environment Variables
```bash
generate: openssl rand -hex 32
```
### Deploy
```bash
cd services/swarm/stack/ntfy
set -a && source .env && set +a
docker stack config --compose-file ntfy.yaml > resolved.yml
docker stack deploy --compose-file resolved.yml ntfy
rm resolved.yml
docker stack services ntfy
```
### First Run
No specific steps are required for the first run.
---
## User Guide
### Accessing ntfy
| Service | URL | Purpose |
|---------|-----|---------|
- **ntfy:** https://ntfy.netgrimoire.com (Internal only) |
### Primary Use Cases
The primary use case is to receive push notifications in NetGrimoire.
### NetGrimoire Integrations
The ntfy service connects to other services through environment variables and labels.
---
## Operations
### Monitoring
[kuma.ntfy.http.name: ntfy, kuma.ntfy.http.url: https://ntfy.netgrimoire.com]
```bash
docker stack services ntfy
docker service logs -f ntfy | grep "NTFY"
```
### Backups
Critical data is stored in /DockerVol/ntfy/cache.
### Restore
```bash
cd services/swarm/stack/ntfy
./deploy.sh
```
---
## Common Failures
1. **Symptom:** Push notifications are not received.
**Cause:** Missing Caddy configuration or environment variables.
**Fix:** Check Caddy labels and environment variables for correctness.
2. **Symptom:** ntfy service is down.
**Cause:** Insufficient restart policy.
**Fix:** Adjust the restart policy in the deploy section.
3. **Symptom:** Docker stack services are not running.
**Cause:** Missing docker-compose-file.
**Fix:** Check if ntfy-stack.yml exists.
4. **Symptom:** Logs do not show any errors.
**Cause:** Insufficient logging configuration.
**Fix:** Adjust log levels or increase verbosity in logs.
5. **Symptom:** Environment variables are incorrect.
**Cause:** Incorrect source of environment variables.
**Fix:** Verify that .env file is correctly sourced.
---
## Changelog
| Date | Commit | Summary |
|------|--------|---------|
- 2026-04-07 | 5058dbe5 | Initial documentation for ntfy stack. |
- 2026-04-07 | 247956f0 | Fixed minor issues in deploy and user guide sections. |
- 2026-02-01 | 85da4a27 | Changed volume paths to match /DockerVol/. |
- 2026-02-01 | 9da20931 | Adjusted logging configuration for ntfy service. |
- 2026-01-10 | 1a374911 | Added initial documentation. |
---
## Notes
- Generated by Gremlin on 2026-04-07T19:16:54.993Z
- Source: swarm/ntfy.yaml
- Review User Guide and Changelog sections