traveler/Netgrimoire
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

prep for new grimoire

Browse source
This commit is contained in:
traveler 2026-04-12 09:39:57 -05:00
parent a72eb28f9e
commit 2aff30ab71
165 changed files with 0 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

503
False Grimoire/Netgrimoire/Services/Gremlin/Netgrimoire_Agent.md Normal file
View file

@ -0,0 +1,503 @@
---
title: Ollama with agent
description: The smart home reference
published: true
date: 2026-04-02T21:11:09.564Z
tags:
editor: markdown
dateCreated: 2026-02-18T22:14:41.533Z
---
# AI Automation Stack - Ollama + n8n + Open WebUI
## Overview
This stack provides a complete self-hosted AI automation solution for homelab infrastructure management, documentation generation, and intelligent monitoring. The system consists of four core components that work together to provide AI-powered workflows and knowledge management.
## Architecture
```
┌─────────────────────────────────────────────────┐
│ AI Automation Stack │
│ │
│ Open WebUI ────────┐ │
│ (Chat Interface) │ │
│ │ │ │
│ ▼ ▼ │
│ Ollama ◄──── Qdrant │
│ (LLM Runtime) (Vector DB) │
│ ▲ │
│ │ │
│ n8n │
│ (Workflow Engine) │
│ │ │
│ ▼ │
│ Forgejo │ Wiki.js │ Monitoring │
└─────────────────────────────────────────────────┘
```
## Components
### Ollama
- **Purpose**: Local LLM runtime engine
- **Port**: 11434
- **Resource Usage**: 4-6GB RAM (depending on model)
- **Recommended Models**:
- `qwen2.5-coder:7b` - Code analysis and documentation
- `llama3.2:3b` - General queries and chat
- `phi3:mini` - Lightweight alternative
### Open WebUI
- **Purpose**: User-friendly chat interface with built-in RAG (Retrieval Augmented Generation)
- **Port**: 3000
- **Features**:
- Document ingestion from Wiki.js
- Conversational interface for querying documentation
- RAG pipeline for context-aware responses
- Multi-model support
- **Access**: `http://your-server-ip:3000`
### Qdrant
- **Purpose**: Vector database for semantic search and RAG
- **Ports**: 6333 (HTTP), 6334 (gRPC)
- **Resource Usage**: ~1GB RAM
- **Function**: Stores embeddings of your documentation, code, and markdown files
### n8n
- **Purpose**: Workflow automation and orchestration
- **Port**: 5678
- **Default Credentials**:
- Username: `admin`
- Password: `change-this-password` (⚠️ **Change this immediately**)
- **Access**: `http://your-server-ip:5678`
## Installation
### Prerequisites
- Docker and Docker Compose installed
- 16GB RAM minimum (8GB available for the stack)
- 50GB disk space for models and data
### Deployment Steps
1. **Create directory structure**:
```bash
mkdir -p ~/ai-stack/{n8n/workflows}
cd ~/ai-stack
```
2. **Download the compose file**:
```bash
# Place the ai-stack-compose.yml in this directory
wget [your-internal-url]/ai-stack-compose.yml
```
3. **Configure environment variables**:
```bash
# Edit the compose file and change:
# - WEBUI_SECRET_KEY
# - N8N_BASIC_AUTH_PASSWORD
# - WEBHOOK_URL (use your server's IP)
# - GENERIC_TIMEZONE
nano ai-stack-compose.yml
```
4. **Start the stack**:
```bash
docker-compose -f ai-stack-compose.yml up -d
```
5. **Pull Ollama models**:
```bash
docker exec -it ollama ollama pull qwen2.5-coder:7b
docker exec -it ollama ollama pull llama3.2:3b
```
6. **Verify services**:
```bash
docker-compose -f ai-stack-compose.yml ps
```
## Configuration
### Open WebUI Setup
1. Navigate to `http://your-server-ip:3000`
2. Create your admin account (first user becomes admin)
3. Go to **Settings → Connections** and verify Ollama connection
4. Configure Qdrant:
- Host: `qdrant`
- Port: `6333`
### Setting Up RAG for Wiki.js
1. In Open WebUI, go to **Workspace → Knowledge**
2. Create a new collection: "Homelab Documentation"
3. Add sources:
- **URL Crawl**: Enter your Wiki.js base URL
- **File Upload**: Upload markdown files from repositories
4. Process and index the documents
### n8n Initial Configuration
1. Navigate to `http://your-server-ip:5678`
2. Log in with credentials from docker-compose file
3. Import starter workflows from `/n8n/workflows/` directory
## Use Cases
### 1. Automated Documentation Generation
**Workflow**: Forgejo webhook → n8n → Ollama → Wiki.js
When code is pushed to Forgejo:
1. n8n receives webhook from Forgejo
2. Extracts changed files and repo context
3. Sends to Ollama with prompt: "Generate documentation for this code"
4. Posts generated docs to Wiki.js via API
**Example n8n Workflow**:
```
Webhook Trigger
→ HTTP Request (Forgejo API - get file contents)
→ Ollama LLM Node (generate docs)
→ HTTP Request (Wiki.js API - create/update page)
→ Send notification (completion)
```
### 2. Docker-Compose Standardization
**Workflow**: Repository scan → compliance check → issue creation
1. n8n runs on schedule (daily/weekly)
2. Queries Forgejo API for all repositories
3. Scans for `docker-compose.yml` files
4. Compares against template standards stored in Qdrant
5. Generates compliance report with Ollama
6. Creates Forgejo issues for non-compliant repos
### 3. Intelligent Alert Processing
**Workflow**: Monitoring alert → AI analysis → smart routing
1. Beszel/Uptime Kuma sends webhook to n8n
2. n8n queries historical data and context
3. Ollama analyzes:
- Is this expected? (scheduled backup, known maintenance)
- Severity level
- Recommended action
4. Routes appropriately:
- Critical: Immediate notification (Telegram/email)
- Warning: Log and monitor
- Info: Suppress (expected behavior)
### 4. Email Monitoring & Triage
**Workflow**: IMAP polling → AI classification → action routing
1. n8n polls email inbox every 5 minutes
2. Filters for keywords: "alert", "critical", "down", "failed"
3. Ollama classifies urgency and determines if actionable
4. Routes based on classification:
- Urgent: Forward to you immediately
- Informational: Daily digest
- Spam: Archive
## Common Workflows
### Example: Repository Documentation Generator
```javascript
// n8n workflow nodes:
1. Schedule Trigger (daily at 2 AM)
↓
2. HTTP Request - Forgejo API
URL: http://forgejo:3000/api/v1/repos/search
Method: GET
↓
3. Loop Over Items (each repo)
↓
4. HTTP Request - Get repo files
URL: {{$node["Forgejo API"].json["clone_url"]}}/contents
↓
5. Filter - Find docker-compose.yml and README.md
↓
6. Ollama Node
Model: qwen2.5-coder:7b
Prompt: "Analyze this docker-compose file and generate comprehensive
documentation including: purpose, services, ports, volumes,
environment variables, and setup instructions."
↓
7. HTTP Request - Wiki.js API
URL: http://wikijs:3000/graphql
Method: POST
Body: {mutation: createPage(...)}
↓
8. Send Notification
Service: Telegram/Email
Message: "Documentation updated for {{repo_name}}"
```
### Example: Alert Intelligence Workflow
```javascript
// n8n workflow nodes:
1. Webhook Trigger
Path: /webhook/monitoring-alert
↓
2. Function Node - Parse Alert Data
JavaScript: Extract service, metric, value, timestamp
↓
3. HTTP Request - Query Historical Data
URL: http://beszel:8090/api/metrics/history
↓
4. Ollama Node
Model: llama3.2:3b
Context: Your knowledge base in Qdrant
Prompt: "Alert: {{alert_message}}
Historical context: {{historical_data}}
Is this expected behavior?
What's the severity?
What action should be taken?"
↓
5. Switch Node - Route by Severity
Conditions:
- Critical: Route to immediate notification
- Warning: Route to monitoring channel
- Info: Route to log only
↓
6a. Send Telegram (Critical path)
6b. Post to Slack (Warning path)
6c. Write to Log (Info path)
```
## Maintenance
### Model Management
```bash
# List installed models
docker exec -it ollama ollama list
# Update a model
docker exec -it ollama ollama pull qwen2.5-coder:7b
# Remove unused models
docker exec -it ollama ollama rm old-model:tag
```
### Backup Important Data
```bash
# Backup Qdrant vector database
docker-compose -f ai-stack-compose.yml stop qdrant
tar -czf qdrant-backup-$(date +%Y%m%d).tar.gz ./qdrant_data/
docker-compose -f ai-stack-compose.yml start qdrant
# Backup n8n workflows (automatic to ./n8n/workflows)
tar -czf n8n-backup-$(date +%Y%m%d).tar.gz ./n8n_data/
# Backup Open WebUI data
tar -czf openwebui-backup-$(date +%Y%m%d).tar.gz ./open_webui_data/
```
### Log Monitoring
```bash
# View all stack logs
docker-compose -f ai-stack-compose.yml logs -f
# View specific service
docker logs -f ollama
docker logs -f n8n
docker logs -f open-webui
```
### Resource Monitoring
```bash
# Check resource usage
docker stats
# Expected usage:
# - ollama: 4-6GB RAM (with model loaded)
# - open-webui: ~500MB RAM
# - qdrant: ~1GB RAM
# - n8n: ~200MB RAM
```
## Troubleshooting
### Ollama Not Responding
```bash
# Check if Ollama is running
docker logs ollama
# Restart Ollama
docker restart ollama
# Test Ollama API
curl http://localhost:11434/api/tags
```
### Open WebUI Can't Connect to Ollama
1. Check network connectivity:
```bash
docker exec -it open-webui ping ollama
```
2. Verify Ollama URL in Open WebUI settings
3. Restart both containers:
```bash
docker restart ollama open-webui
```
### n8n Workflows Failing
1. Check n8n logs:
```bash
docker logs n8n
```
2. Verify webhook URLs are accessible
3. Test Ollama connection from n8n:
- Create test workflow
- Add Ollama node
- Run execution
### Qdrant Connection Issues
```bash
# Check Qdrant health
curl http://localhost:6333/health
# View Qdrant logs
docker logs qdrant
# Restart if needed
docker restart qdrant
```
## Performance Optimization
### Model Selection by Use Case
- **Quick queries, chat**: `llama3.2:3b` or `phi3:mini` (fastest)
- **Code analysis**: `qwen2.5-coder:7b` or `deepseek-coder:6.7b`
- **Complex reasoning**: `mistral:7b` or `llama3.1:8b`
### n8n Workflow Optimization
- Use **Wait** nodes to batch operations
- Enable **Execute Once** for loops to reduce memory
- Store large data in temporary files instead of node output
- Use **Split In Batches** for processing large datasets
### Qdrant Performance
- Default settings are optimized for homelab use
- Increase `collection_shards` if indexing >100,000 documents
- Enable quantization for large collections
## Security Considerations
### Change Default Credentials
```bash
# Generate secure password
openssl rand -base64 32
# Update in docker-compose.yml:
# - WEBUI_SECRET_KEY
# - N8N_BASIC_AUTH_PASSWORD
```
### Network Isolation
Consider using a reverse proxy (Traefik, Nginx Proxy Manager) with authentication:
- Limit external access to Open WebUI only
- Keep n8n, Ollama, Qdrant on internal network
- Use VPN for remote access
### API Security
- Use strong API tokens for Wiki.js and Forgejo integrations
- Rotate credentials periodically
- Audit n8n workflow permissions
## Integration Points
### Connecting to Existing Services
**Uptime Kuma**:
- Configure webhook alerts → n8n webhook URL
- Path: `http://your-server-ip:5678/webhook/uptime-kuma`
**Beszel**:
- Use Shoutrrr webhook format
- URL: `http://your-server-ip:5678/webhook/beszel`
**Forgejo**:
- Repository webhooks for push events
- URL: `http://your-server-ip:5678/webhook/forgejo-push`
- Enable in repo settings → Webhooks
**Wiki.js**:
- GraphQL API endpoint: `http://wikijs:3000/graphql`
- Create API key in Wiki.js admin panel
- Store in n8n credentials
## Advanced Features
### Creating Custom n8n Nodes
For frequently used Ollama prompts, create custom nodes:
1. Go to n8n → Settings → Community Nodes
2. Install `n8n-nodes-ollama-advanced` if available
3. Or create Function nodes with reusable code
### Training Custom Models
While Ollama doesn't support fine-tuning directly, you can:
1. Use RAG with your specific documentation
2. Create detailed system prompts in n8n
3. Store organization-specific context in Qdrant
### Multi-Agent Workflows
Chain multiple Ollama calls for complex tasks:
```
Planning Agent → Execution Agent → Review Agent → Output
```
Example: Code refactoring
1. Planning: Analyze code and create refactoring plan
2. Execution: Generate refactored code
3. Review: Check for errors and improvements
4. Output: Create pull request with changes
## Resources
- **Ollama Documentation**: https://ollama.ai/docs
- **Open WebUI Docs**: https://docs.openwebui.com
- **n8n Documentation**: https://docs.n8n.io
- **Qdrant Docs**: https://qdrant.tech/documentation
## Support
For issues or questions:
1. Check container logs first
2. Review this documentation
3. Search n8n community forums
4. Check Ollama Discord/GitHub issues
---
**Last Updated**: {{current_date}}
**Maintained By**: Homelab Admin
**Status**: Production

361
False Grimoire/Netgrimoire/Services/Gremlin/Readme.md Normal file
View file

@ -0,0 +1,361 @@
---
title: Readme
description: Readme file generated by AI
published: true
date: 2026-04-02T21:09:39.376Z
tags:
editor: markdown
dateCreated: 2026-03-05T02:27:57.522Z
---
# Homelab AI & Monitoring Stack - Deployment Guide
This repository contains everything you need to deploy a complete AI-powered homelab monitoring and automation stack.
## What's Included
### 📦 Docker Compose Files
1. **ai-stack-compose.yml** - Main AI automation stack (Ollama, Open WebUI, n8n, Qdrant)
2. **librenms-compose.yml** - Network monitoring system (LibreNMS + MariaDB + Redis)
### 📚 Wiki.js Documentation
1. **wiki-ai-stack.md** - Complete documentation for the AI stack
2. **wiki-librenms.md** - Complete documentation for LibreNMS
## Quick Start
### Prerequisites
- Docker and Docker Compose installed
- 16GB RAM minimum (8GB+ available)
- 70GB disk space (50GB for AI stack + 20GB for LibreNMS)
- Network devices with SNMP enabled (for LibreNMS)
### Step 1: Deploy AI Stack
```bash
# Create directory
mkdir -p ~/homelab/ai-stack
cd ~/homelab/ai-stack
# Copy ai-stack-compose.yml to this directory
# Edit environment variables
nano ai-stack-compose.yml
# Change:
# - WEBUI_SECRET_KEY (generate random string)
# - N8N_BASIC_AUTH_PASSWORD (use strong password)
# - WEBHOOK_URL (your server IP)
# - GENERIC_TIMEZONE (your timezone)
# Start the stack
docker-compose -f ai-stack-compose.yml up -d
# Pull AI models
docker exec -it ollama ollama pull qwen2.5-coder:7b
docker exec -it ollama ollama pull llama3.2:3b
# Verify all services are running
docker-compose -f ai-stack-compose.yml ps
```
**Access points:**
- Open WebUI: http://your-server-ip:3000
- n8n: http://your-server-ip:5678
- Ollama API: http://your-server-ip:11434
### Step 2: Deploy LibreNMS
```bash
# Create directory
mkdir -p ~/homelab/librenms
cd ~/homelab/librenms
# Copy librenms-compose.yml to this directory
# Edit environment variables
nano librenms-compose.yml
# Change:
# - DB_PASSWORD (use strong password)
# - MYSQL_ROOT_PASSWORD (use strong password)
# - BASE_URL (your server IP)
# - TZ (your timezone)
# Start LibreNMS
docker-compose -f librenms-compose.yml up -d
# Wait for initialization (2-3 minutes)
docker logs -f librenms
# Access web interface
# http://your-server-ip:8000
# Default login: librenms/librenms
# CHANGE PASSWORD IMMEDIATELY!
```
### Step 3: Import Documentation to Wiki.js
```bash
# Option 1: Via Wiki.js Web Interface
# 1. Login to Wiki.js
# 2. Create new page: "AI Stack Documentation"
# 3. Copy contents of wiki-ai-stack.md
# 4. Create new page: "LibreNMS Documentation"
# 5. Copy contents of wiki-librenms.md
# Option 2: Via Wiki.js API (if configured)
# Use the provided markdown files with Wiki.js GraphQL API
```
## Initial Configuration
### Open WebUI Setup
1. Navigate to http://your-server-ip:3000
2. Create admin account (first user becomes admin)
3. Verify Ollama connection in Settings
4. Configure Qdrant connection (host: qdrant, port: 6333)
5. Import your Wiki.js documentation for RAG
### n8n Setup
1. Navigate to http://your-server-ip:5678
2. Login with credentials from compose file
3. Create first workflow (see documentation for examples)
4. Configure Ollama node connection
### LibreNMS Setup
1. Navigate to http://your-server-ip:8000
2. Login and CHANGE PASSWORD
3. Add your first network device
4. Configure alert transport (webhook to n8n)
5. Generate API token for n8n integration
## Integrations
### Connect Existing Services
**Uptime Kuma → n8n:**
- Configure webhook in Uptime Kuma notification settings
- URL: http://your-server-ip:5678/webhook/uptime-kuma
**Beszel → n8n:**
- Use Shoutrrr webhook format
- URL: http://your-server-ip:5678/webhook/beszel
**Forgejo → n8n:**
- Add webhook in repository settings
- URL: http://your-server-ip:5678/webhook/forgejo-push
- Events: Push, Pull Request
**LibreNMS → n8n:**
- Alerts → Alert Transports → Add Webhook
- URL: http://your-server-ip:5678/webhook/librenms-alert
## Resource Usage
Expected memory usage with all services running:
| Service | Memory |
|---------|--------|
| Ollama (with model loaded) | 4-6GB |
| Open WebUI | 500MB |
| Qdrant | 1GB |
| n8n | 200MB |
| LibreNMS | 300-500MB |
| MariaDB | 500MB-1GB |
| Redis | 50-100MB |
| **Total** | **~7-10GB** |
Remaining ~6-9GB for other services and system.
## Example Workflows
### 1. Intelligent Alert Processing
```
Monitoring Alert → n8n webhook
→ Query historical data
→ Ollama analysis (Is this expected? Severity? Action needed?)
→ Route based on AI decision
→ Critical: Immediate notification
→ Warning: Log and monitor
→ Info: Suppress
```
### 2. Automated Documentation
```
Code Push to Forgejo → n8n webhook
→ Get changed files
→ Ollama generates documentation
→ Post to Wiki.js via API
→ Notify team
```
### 3. Docker-Compose Standardization
```
n8n scheduled workflow (daily)
→ Scan all Forgejo repos
→ Find docker-compose.yml files
→ Compare against template (stored in Qdrant)
→ Ollama generates compliance report
→ Create Forgejo issues for non-compliant repos
```
## Backup Strategy
### AI Stack Backup
```bash
# Weekly backup
cd ~/homelab/ai-stack
docker-compose -f ai-stack-compose.yml stop qdrant
tar -czf ai-stack-backup-$(date +%Y%m%d).tar.gz \
qdrant_data/ n8n_data/ open_webui_data/
docker-compose -f ai-stack-compose.yml start qdrant
```
### LibreNMS Backup
```bash
# Weekly backup
cd ~/homelab/librenms
docker exec librenms_db mysqldump -u root -p librenms > \
librenms-db-backup-$(date +%Y%m%d).sql
tar -czf librenms-data-backup-$(date +%Y%m%d).tar.gz librenms_data/
```
### Automated Backup via n8n
Create a scheduled workflow that:
1. Runs weekly (Sunday 2 AM)
2. Executes backup commands
3. Uploads to external storage (optional)
4. Verifies backup integrity
5. Sends notification with results
## Troubleshooting
### Services Won't Start
```bash
# Check logs
docker-compose -f ai-stack-compose.yml logs [service-name]
# Common issues:
# - Port conflicts (check with: netstat -tulpn)
# - Insufficient memory (check with: free -h)
# - Permissions on volume directories
```
### Ollama Not Responding
```bash
# Restart Ollama
docker restart ollama
# Test API
curl http://localhost:11434/api/tags
# If still failing, check if model is loaded
docker exec -it ollama ollama list
```
### Can't Connect to Services
```bash
# Check if services are running
docker ps
# Check network connectivity
docker network ls
docker network inspect [network-name]
# Verify firewall isn't blocking ports
sudo ufw status
```
## Security Recommendations
1. **Change all default passwords immediately**
2. **Use strong, unique passwords for:**
- n8n basic auth
- LibreNMS admin user
- Database passwords
- Open WebUI admin account
3. **Network security:**
- Use reverse proxy (Traefik, Nginx Proxy Manager)
- Enable SSL/TLS certificates
- Restrict access to trusted networks
- Consider VPN for remote access
4. **API security:**
- Generate strong API tokens
- Rotate credentials periodically
- Use read-only tokens when possible
## Maintenance Schedule
**Daily (automated):**
- Service polling and monitoring
- Alert processing
- Automatic discovery
**Weekly:**
- Review alerts and adjust thresholds
- Check service logs for errors
- Verify backups completed successfully
**Monthly:**
- Database optimization
- Review disk space usage
- Update containers (test in dev first)
- Audit user accounts and permissions
**Quarterly:**
- Full backup verification and restoration test
- Security audit
- Review and update documentation
- Clean up old data
## Getting Help
### Documentation
- Check the Wiki.js pages for detailed information
- Review container logs for error messages
- Search community forums for similar issues
### Useful Commands
```bash
# View all logs
docker-compose logs -f
# View specific service
docker logs -f [container-name]
# Restart single service
docker restart [container-name]
# Restart entire stack
docker-compose -f [compose-file] restart
# Update containers
docker-compose -f [compose-file] pull
docker-compose -f [compose-file] up -d
```
## Next Steps
1. ✅ Deploy AI stack
2. ✅ Deploy LibreNMS
3. ✅ Import documentation to Wiki.js
4. ⬜ Configure integrations with existing services
5. ⬜ Create first n8n workflow
6. ⬜ Add network devices to LibreNMS
7. ⬜ Set up automated backups
8. ⬜ Create custom dashboards
## Support
For issues specific to:
- **Ollama**: https://github.com/ollama/ollama/issues
- **Open WebUI**: https://github.com/open-webui/open-webui/issues
- **n8n**: https://community.n8n.io
- **LibreNMS**: https://community.librenms.org
---
**Last Updated:** February 2025
**Maintained By:** Homelab Admin
**License:** MIT (for custom configurations)

383
False Grimoire/Netgrimoire/Services/Gremlin/gremlin-build-config.md Normal file
View file

File diff suppressed because one or more lines are too long

194
False Grimoire/Netgrimoire/Services/Gremlin/gremlin-user-guide.md Normal file
View file

File diff suppressed because one or more lines are too long

128
False Grimoire/Netgrimoire/Services/Immich/Convert_Immich.md Normal file
View file

@ -0,0 +1,128 @@
---
title: Immich on ZFS
description: Moving Immich to its own ZFS dataset
published: true
date: 2026-02-20T04:13:02.502Z
tags: service zfs immich dataset
editor: markdown
dateCreated: 2026-02-06T15:57:04.261Z
---
# Moving Immich to a ZFS Dataset
## Overview
This guide covers moving an existing Immich installation to its own ZFS dataset to enable `zfs send` backups.
## Prerequisites
- ZFS pool mounted at `/srv/vault`
- Existing Immich installation at `/srv/vault/immich`
- Immich running via Docker Compose
## Steps
### 1. Stop Immich Services
```bash
cd /srv/vault/immich # or wherever your docker-compose.yml is
docker compose down
```
### 2. Create the New Dataset
```bash
sudo zfs create vault/immich
```
### 3. Move Existing Data Temporarily
```bash
sudo mv /srv/vault/immich /srv/vault/immich_old
```
### 4. Set Mountpoint and Mount Dataset
```bash
sudo zfs set mountpoint=/srv/immich vault/immich
sudo zfs mount vault/immich
```
### 5. Copy Data to New Dataset
```bash
sudo rsync -avxHAX /srv//immich_old/ /srv/immich/
```
Flags preserve permissions, ownership, and special attributes.
### 6. Verify Data Copy
```bash
sudo du -sh /srv/vault/immich_old
sudo du -sh /srv/vault/immich
```
Sizes should match closely.
### 7. Start Immich
```bash
cd /srv/vault/immich
docker compose up -d
```
### 8. Test and Clean Up
Verify everything works, then remove old data:
```bash
sudo rm -rf /srv/vault/immich_old
```
## ZFS Dataset Properties
### Recommended Settings
```bash
# Compression - helps with photos and database
sudo zfs set compression=lz4 vault/immich
# Record size - balance for mixed workload
sudo zfs set recordsize=128K vault/immich
# Better database performance
sudo zfs set primarycache=all vault/immich
sudo zfs set atime=off vault/immich
```
### Property Explanations
- **compression=lz4**: Fast, low CPU overhead, works well for both photos and database
- **recordsize=128K**: Good compromise between database (8K blocks) and photos (larger files)
- **atime=off**: Disables access time updates, reduces unnecessary writes
- **primarycache=all**: Keeps both metadata and data in ARC cache (default)
## Backup with ZFS Send/Receive
### Create Snapshot
```bash
zfs snapshot vault/immich@backup-$(date +%Y%m%d)
```
### Send to Remote Server
```bash
zfs send vault/immich@backup-$(date +%Y%m%d) | ssh backup-server zfs receive tank/backups/immich
```
### Incremental Backups
```bash
# After first full backup
zfs snapshot vault/immich@backup-$(date +%Y%m%d)
zfs send -i vault/immich@previous-snapshot vault/immich@backup-$(date +%Y%m%d) | \
ssh backup-server zfs receive tank/backups/immich
```
## Optional: Separate Datasets for Database and Photos
For optimal performance, split into separate datasets:
```bash
sudo zfs create vault/immich/database
sudo zfs create vault/immich/photos
# Database optimized
sudo zfs set recordsize=16K vault/immich/database
sudo zfs set logbias=latency vault/immich/database
# Photos optimized
sudo zfs set recordsize=1M vault/immich/photos
```
Then update your Docker Compose volume mounts accordingly.

430
False Grimoire/Netgrimoire/Services/MailCow/MXRoute_Integration.md Normal file
View file

@ -0,0 +1,430 @@
---
title: Integrating MXRoute with MailCow
description:
published: true
date: 2026-02-25T21:04:37.135Z
tags:
editor: markdown
dateCreated: 2026-02-25T19:22:31.514Z
---
# MXRoute — Master Configuration Reference
## Overview
MXRoute serves two roles in Netgrimoire mail infrastructure:
- **Inbound gateway** — MX records for all domains point to MXRoute's commercial IPs, solving residential AT&T IP filtering by banks and financial institutions. MXRoute receives mail and forwards to Mailcow via per-address forwarders.
- **Outbound relay** — Mailcow sends all outbound mail through MXRoute via sender-dependent transports for improved deliverability.
**Mail flow:**
```
Inbound: Internet → MXRoute (commercial IP) → Mailcow (192.168.5.16)
Outbound: Mailcow (192.168.5.16) → MXRoute SMTP relay → Internet
```
**Mailcow host:** 192.168.5.16
**MXRoute control panel:** confirm server hostname from MXRoute welcome email (e.g. `arrow.mxrouting.net`)
**MXRoute SMTP relay:** confirm from welcome email (e.g. `smtp.mxroute.com:587`)
---
## Architecture — Why Two Domains Per Hosted Domain
MXRoute forwarders require a valid destination email address. Forwarding `user@domain.com` back to `user@domain.com` creates a mail loop because MXRoute would look up the MX for `domain.com` and find itself. The solution is a `mail.domain.com` subdomain with its own MX record pointing directly to Mailcow. MXRoute forwards to `user@mail.domain.com`, Mailcow accepts and delivers, and an alias domain maps `@domain.com` back so users only ever see `@domain.com`.
```
domain.com MX → MXRoute (public-facing, receives from internet)
mail.domain.com MX → 192.168.5.16 (internal, MXRoute forwards here)
```
---
## MXRoute Control Panel
**Login:** confirm URL from MXRoute welcome email
**Interface:** MXRoute 4.0 (new UI — not old DirectAdmin)
### Creating a Forwarder
1. Go to **Forwarders**
2. Click **Create New Forwarder**
3. Set **Forwarder Name:** `username` (domain shown automatically)
4. Set **Destination Type:** `Forward to Email(s)`
5. Set **Recipients:** `username@mail.domain.com`
6. Click **Create Forwarder**
> Recipients field accepts multiple addresses comma or newline separated.
---
## Mailcow Configuration
### Adding a New Domain (One-Time Per Domain)
1. **Mail Setup → Domains → Add domain**
- Domain: `mail.domain.com` (the subdomain Mailcow owns)
- Leave relay settings as default
2. **Mail Setup → Alias Domains → Add alias domain**
- Alias Domain: `domain.com`
- Target Domain: `mail.domain.com`
- This makes Mailcow accept and deliver mail for `@domain.com` to `@mail.domain.com` mailboxes
3. **Configuration → ARC/DKIM Keys**
- Select domain `mail.domain.com`
- Selector: `mailcow`
- Key length: 2048
- Generate and copy TXT record for DNS
4. **Configuration → Extra Postfix configuration → extra.cf**
```
# Trust MXRoute forwarding IPs — prevents SPF scoring on forwarded mail
mynetworks = 127.0.0.1/8 [::1]/128 192.168.5.0/24 69.167.160.0/19 198.54.120.0/22
```
Restart affected containers after saving.
### Adding a New Mailbox
1. **Mail Setup → Mailboxes → Add mailbox**
- Username: `user`
- Domain: `mail.domain.com`
2. **MXRoute control panel → Forwarders → Create New Forwarder**
- Forwarder: `user@domain.com`
- Destination: `user@mail.domain.com`
### Outbound Relay — Sender-Dependent Transports
One transport entry per domain. **Configuration → Routing → Sender-Dependent Transports**
| Domain | Relay Host | Username | Password |
|--------|-----------|----------|----------|
| pncharris.com | `[smtp.mxroute.com]:587` | relay@pncharris.com | H@rv3yD)G123 |
| wasted-bandwidth.net | `[smtp.mxroute.com]:587` | relay@wasted-bandwidth.net | dZ4yLYznVvgSJtqWZJFA |
| netgrimoire.com | `[smtp.mxroute.com]:587` | relay@netgrimoire.com | TVGCnJp9SxRbWU8EhkMw |
| florosafd.org | `[smtp.mxroute.com]:587` | relay@florosafd.org | 2Fe8XMyaeh6Z5dvdHYdq |
| gnarlypandaproductions.com | `[smtp.mxroute.com]:587` | relay@gnarlypandaproductions.com | vG5ZsUQhRWD2UyzLPsqA |
> Confirm SMTP relay hostname from MXRoute welcome email — substitute actual hostname for `smtp.mxroute.com` if different.
### Email Client Settings (All Domains)
| Setting | Value |
|---------|-------|
| IMAP server | `mail.domain.com` |
| IMAP port | `993` (SSL/TLS) |
| SMTP server | `mail.domain.com` |
| SMTP port | `465` (SSL/TLS) |
| Username | `user@domain.com` |
> Users log in with `@domain.com`. Mailcow resolves to the internal `@mail.domain.com` mailbox via alias domain — transparent to the user.
---
## DNS Reference — All Domains
### DNS Pattern (Apply to Every Domain)
Two sets of MX records are required — one for the public domain (pointing to MXRoute) and one for the mail subdomain (pointing directly to Mailcow).
| Type | Host | Value | Notes |
|------|------|-------|-------|
| A | `mail` | `YOUR_ATT_MAIL_IP` | Mailcow server — MXRoute forwards here |
| MX | `@` | MXRoute primary (priority 10) | From MXRoute welcome email |
| MX | `@` | MXRoute secondary (priority 20) | From MXRoute welcome email |
| MX | `mail` | `mail.domain.com` (priority 10) | Mailcow handles subdomain directly |
| CNAME | `imap` | `mail.domain.com` | Client autoconfiguration |
| CNAME | `smtp` | `mail.domain.com` | Client autoconfiguration |
| CNAME | `webmail` | `mail.domain.com` | Roundcube access |
| CNAME | `autodiscover` | `mail.domain.com` | Outlook autodiscover |
| CNAME | `autoconfig` | `mail.domain.com` | Thunderbird autoconfig |
| TXT | `@` | `v=spf1 ip4:YOUR_ATT_MAIL_IP include:mxroute.com -all` | SPF — both Mailcow direct and MXRoute relay |
| TXT | `mail` | `v=spf1 ip4:YOUR_ATT_MAIL_IP -all` | SPF for subdomain — Mailcow direct only |
| TXT | `_dmarc` | `v=DMARC1; p=reject; rua=mailto:admin@netgrimoire.com` | DMARC enforcement |
| TXT | `mailcow._domainkey.mail` | *(generated in Mailcow ARC/DKIM Keys)* | Mailcow DKIM selector |
| TXT | `x._domainkey` | *(from MXRoute control panel)* | MXRoute DKIM selector — confirm actual selector name |
---
### pncharris.com
| Type | Host | Value |
|------|------|-------|
| A | `mail` | YOUR_ATT_MAIL_IP |
| MX | `@` | MXRoute primary (priority 10) |
| MX | `@` | MXRoute secondary (priority 20) |
| MX | `mail` | `mail.pncharris.com` (priority 10) |
| CNAME | `imap` | `mail.pncharris.com` |
| CNAME | `smtp` | `mail.pncharris.com` |
| CNAME | `webmail` | `mail.pncharris.com` |
| CNAME | `autodiscover` | `mail.pncharris.com` |
| CNAME | `autoconfig` | `mail.pncharris.com` |
| TXT | `@` | `v=spf1 ip4:YOUR_ATT_MAIL_IP include:mxroute.com -all` |
| TXT | `mail` | `v=spf1 ip4:YOUR_ATT_MAIL_IP -all` |
| TXT | `_dmarc` | `v=DMARC1; p=reject; rua=mailto:admin@netgrimoire.com` |
| TXT | `mailcow._domainkey.mail` | *(from Mailcow ARC/DKIM Keys for mail.pncharris.com)* |
| TXT | `x._domainkey` | *(from MXRoute control panel)* |
**Mailcow domains:** `mail.pncharris.com` (primary), `pncharris.com` (alias domain → mail.pncharris.com)
**Relay credentials:**
| Account | Password | Notes |
|---------|----------|-------|
| relay@pncharris.com | H@rv3yD)G123 | Current relay account |
| forwarder@pncharris.com | *(see password history below)* | Legacy account |
| passer@pncharris.com | bBJtPhrGkHvvhxhukkae | Current |
| kylr pncharris | -,68,incTeR | |
| G4@rlyf1ng3r | *(Feb 14)* | |
**passer@pncharris.com password history** (most recent last):
- !5!,_\*zDyLEhhR4
- sh7dXWnTPqbkDGsTcwtn
- MY3V8p69b2HYksygxhXX
- RS6U2GU6rcYe3THKKgYx
- yzqNysrd73yzWptVEZ5H (current)
---
### wasted-bandwidth.net
| Type | Host | Value |
|------|------|-------|
| A | `mail` | YOUR_ATT_MAIL_IP |
| MX | `@` | MXRoute primary (priority 10) |
| MX | `@` | MXRoute secondary (priority 20) |
| MX | `mail` | `mail.wasted-bandwidth.net` (priority 10) |
| CNAME | `imap` | `mail.wasted-bandwidth.net` |
| CNAME | `smtp` | `mail.wasted-bandwidth.net` |
| CNAME | `webmail` | `mail.wasted-bandwidth.net` |
| CNAME | `autodiscover` | `mail.wasted-bandwidth.net` |
| CNAME | `autoconfig` | `mail.wasted-bandwidth.net` |
| TXT | `@` | `v=spf1 ip4:YOUR_ATT_MAIL_IP include:mxroute.com -all` |
| TXT | `mail` | `v=spf1 ip4:YOUR_ATT_MAIL_IP -all` |
| TXT | `_dmarc` | `v=DMARC1; p=reject; rua=mailto:admin@netgrimoire.com` |
| TXT | `mailcow._domainkey.mail` | *(from Mailcow ARC/DKIM Keys for mail.wasted-bandwidth.net)* |
| TXT | `x._domainkey` | *(from MXRoute control panel)* |
**Mailcow domains:** `mail.wasted-bandwidth.net` (primary), `wasted-bandwidth.net` (alias domain)
**Relay credentials:**
| Account | Password |
|---------|----------|
| relay@wasted-bandwidth.net | dZ4yLYznVvgSJtqWZJFA |
---
### netgrimoire.com
| Type | Host | Value |
|------|------|-------|
| A | `mail` | YOUR_ATT_MAIL_IP |
| MX | `@` | MXRoute primary (priority 10) |
| MX | `@` | MXRoute secondary (priority 20) |
| MX | `mail` | `mail.netgrimoire.com` (priority 10) |
| CNAME | `imap` | `mail.netgrimoire.com` |
| CNAME | `smtp` | `mail.netgrimoire.com` |
| CNAME | `webmail` | `mail.netgrimoire.com` |
| CNAME | `autodiscover` | `mail.netgrimoire.com` |
| CNAME | `autoconfig` | `mail.netgrimoire.com` |
| TXT | `@` | `v=spf1 ip4:YOUR_ATT_MAIL_IP include:mxroute.com -all` |
| TXT | `mail` | `v=spf1 ip4:YOUR_ATT_MAIL_IP -all` |
| TXT | `_dmarc` | `v=DMARC1; p=reject; rua=mailto:admin@netgrimoire.com` |
| TXT | `mailcow._domainkey.mail` | *(from Mailcow ARC/DKIM Keys for mail.netgrimoire.com)* |
| TXT | `x._domainkey` | *(from MXRoute control panel)* |
**Mailcow domains:** `mail.netgrimoire.com` (primary), `netgrimoire.com` (alias domain)
**Relay credentials:**
| Account | Password |
|---------|----------|
| relay@netgrimoire.com | TVGCnJp9SxRbWU8EhkMw |
---
### florosafd.org
| Type | Host | Value |
|------|------|-------|
| A | `mail` | YOUR_ATT_MAIL_IP |
| MX | `@` | MXRoute primary (priority 10) |
| MX | `@` | MXRoute secondary (priority 20) |
| MX | `mail` | `mail.florosafd.org` (priority 10) |
| CNAME | `imap` | `mail.florosafd.org` |
| CNAME | `smtp` | `mail.florosafd.org` |
| CNAME | `webmail` | `mail.florosafd.org` |
| CNAME | `autodiscover` | `mail.florosafd.org` |
| CNAME | `autoconfig` | `mail.florosafd.org` |
| TXT | `@` | `v=spf1 ip4:YOUR_ATT_MAIL_IP include:mxroute.com -all` |
| TXT | `mail` | `v=spf1 ip4:YOUR_ATT_MAIL_IP -all` |
| TXT | `_dmarc` | `v=DMARC1; p=reject; rua=mailto:admin@netgrimoire.com` |
| TXT | `mailcow._domainkey.mail` | *(from Mailcow ARC/DKIM Keys for mail.florosafd.org)* |
| TXT | `x._domainkey` | *(from MXRoute control panel)* |
**Mailcow domains:** `mail.florosafd.org` (primary), `florosafd.org` (alias domain)
**Relay credentials:**
| Account | Password |
|---------|----------|
| relay@florosafd.org | 2Fe8XMyaeh6Z5dvdHYdq |
---
### gnarlypandaproductions.com
| Type | Host | Value |
|------|------|-------|
| A | `mail` | YOUR_ATT_MAIL_IP |
| MX | `@` | MXRoute primary (priority 10) |
| MX | `@` | MXRoute secondary (priority 20) |
| MX | `mail` | `mail.gnarlypandaproductions.com` (priority 10) |
| CNAME | `imap` | `mail.gnarlypandaproductions.com` |
| CNAME | `smtp` | `mail.gnarlypandaproductions.com` |
| CNAME | `webmail` | `mail.gnarlypandaproductions.com` |
| CNAME | `roundcube` | `roundcube.netgrimoire.com` |
| CNAME | `autodiscover` | `mail.gnarlypandaproductions.com` |
| CNAME | `autoconfig` | `mail.gnarlypandaproductions.com` |
| TXT | `@` | `v=spf1 ip4:YOUR_ATT_MAIL_IP include:mxroute.com -all` |
| TXT | `mail` | `v=spf1 ip4:YOUR_ATT_MAIL_IP -all` |
| TXT | `_dmarc` | `v=DMARC1; p=reject; rua=mailto:admin@gnarlypandaproductions.com` |
| TXT | `mailcow._domainkey.mail` | *(from Mailcow ARC/DKIM Keys for mail.gnarlypandaproductions.com)* |
| TXT | `default._domainkey` | `v=DKIM1; t=s; p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA3D3vyPoBHB4eMSMq8HygVWHzYbketRX4yjk9wV4bdaar0/c89dK230FMOW6zVXEsY1sXKFk1kBxerHVw0wY8qnQyooHgINEQcEXrtB/x93Sl/cqBQXk+PHOIOymQwgni8WCUhCSnvunxXK8qX5f9J56qzd0/wpY2WSEHho+XrnQjc+c7HMvkcC3+nKJe59ZNgvQW/Y9B/L6zFDjAp+QOUYp9wwX4L+j1T4fQSygYxAJZ0aIoR8FsbOuXc38pht99HyUnYwH08HoK7xv3DL2BrVo3KVZ7xMe2S4YMxd1HkJz2evbV/ziNsJcKW/le3fFS7mza09yJXDLDcLOKLXbYUQIDAQAB` |
| TXT | `x._domainkey` | *(from MXRoute control panel — confirm actual selector)* |
**Mailcow domains:** `mail.gnarlypandaproductions.com` (primary), `gnarlypandaproductions.com` (alias domain)
**Relay credentials:**
| Account | Password |
|---------|----------|
| relay@gnarlypandaproductions.com | vG5ZsUQhRWD2UyzLPsqA |
---
### nucking-futz.com
New domain — see [Mail Setup — nucking-futz.com](./mail-setup-nucking-futz) for full setup guide.
| Type | Host | Value |
|------|------|-------|
| A | `mail` | YOUR_ATT_MAIL_IP |
| MX | `@` | MXRoute primary (priority 10) |
| MX | `@` | MXRoute secondary (priority 20) |
| MX | `mail` | `mail.nucking-futz.com` (priority 10) |
| CNAME | `imap` | `mail.nucking-futz.com` |
| CNAME | `smtp` | `mail.nucking-futz.com` |
| CNAME | `webmail` | `mail.nucking-futz.com` |
| CNAME | `autodiscover` | `mail.nucking-futz.com` |
| CNAME | `autoconfig` | `mail.nucking-futz.com` |
| TXT | `@` | `v=spf1 ip4:YOUR_ATT_MAIL_IP include:mxroute.com -all` |
| TXT | `mail` | `v=spf1 ip4:YOUR_ATT_MAIL_IP -all` |
| TXT | `_dmarc` | `v=DMARC1; p=reject; rua=mailto:admin@netgrimoire.com` |
| TXT | `mailcow._domainkey.mail` | *(from Mailcow ARC/DKIM Keys for mail.nucking-futz.com)* |
| TXT | `x._domainkey` | *(from MXRoute control panel)* |
**Mailcow domains:** `mail.nucking-futz.com` (primary), `nucking-futz.com` (alias domain)
**Relay credentials:**
| Account | Password |
|---------|----------|
| relay@nucking-futz.com | *(set during MXRoute domain creation)* |
---
## Adding a New Domain — Checklist
Use this checklist every time a new domain is added to the stack.
**DNS (at registrar):**
- [ ] A record: `mail.newdomain.com` → YOUR_ATT_MAIL_IP
- [ ] MX records: `@` → MXRoute servers
- [ ] MX record: `mail` → `mail.newdomain.com`
- [ ] CNAME records: imap, smtp, webmail, autodiscover, autoconfig
- [ ] SPF TXT: `@` — includes both ATT IP and `include:mxroute.com`
- [ ] SPF TXT: `mail` — ATT IP only
- [ ] DMARC TXT: `_dmarc`
- [ ] DKIM TXT: `mailcow._domainkey.mail` — after generating in Mailcow
- [ ] DKIM TXT: `x._domainkey` — after retrieving from MXRoute
**Mailcow:**
- [ ] Add domain: `mail.newdomain.com`
- [ ] Add alias domain: `newdomain.com` → `mail.newdomain.com`
- [ ] Generate DKIM key (selector: `mailcow`) for `mail.newdomain.com`
- [ ] Add sender-dependent transport for `newdomain.com`
- [ ] Add sender-dependent transport for `mail.newdomain.com`
- [ ] Create mailboxes as `user@mail.newdomain.com`
**MXRoute:**
- [ ] Add domain in control panel
- [ ] Create forwarder for each mailbox: `user@newdomain.com` → `user@mail.newdomain.com`
- [ ] Retrieve DKIM key for DNS
---
## Troubleshooting
### Mail not delivering inbound (not reaching Mailcow)
- Check MX records for `@` point to MXRoute servers: `dig MX domain.com +short`
- Check MX record for `mail` subdomain points to Mailcow: `dig MX mail.domain.com +short`
- Verify MXRoute forwarder exists for the address in the control panel
- Check Mailcow logs: **Logs → Postfix** — look for the delivery attempt and any rejection reason
- Verify MXRoute IP ranges are in Mailcow `extra.cf` trusted networks
### Mail not delivering inbound (banks / financial institutions)
- This is the residential AT&T IP problem — confirm MX records point to MXRoute, not directly to your IP
- Run `dig MX domain.com +short` — should show MXRoute servers, not your IP
- If MX still points to your ATT IP, update DNS and wait for propagation
### Outbound mail rejected or going to spam
- Verify sender-dependent transport is configured for the domain in Mailcow
- Check relay credentials are current in the transport entry
- Run an SPF check: `dig TXT domain.com +short` — confirm `include:mxroute.com` is present
- Send test to check-auth@verifier.port25.com for full SPF/DKIM/DMARC report
- Run through https://mail-tester.com for a deliverability score
### DKIM verification failing
- Confirm both selectors are published in DNS:
- `dig TXT mailcow._domainkey.mail.domain.com +short`
- `dig TXT x._domainkey.domain.com +short` (substitute actual MXRoute selector)
- Allow up to 48 hours for DNS propagation after adding records
- Verify selector names match exactly what Mailcow and MXRoute are using to sign
### DMARC failures
- SPF and DKIM must both pass and align with the From: domain
- Check DMARC reports sent to `admin@netgrimoire.com` — use [Postmark DMARC](https://dmarc.postmarkapp.com/) or [dmarcian.com](https://dmarcian.com) to parse raw XML reports
- Common cause: outbound mail going through MXRoute but `include:mxroute.com` missing from SPF
### Forwarded mail getting spam-scored
- Confirm MXRoute IP ranges are in Mailcow `extra.cf` mynetworks
- Check that Mailcow trusted networks were saved and containers restarted
- Verify SRS is working: in Roundcube open a forwarded message → More → View Source → `Return-Path` should begin with `SRS0=`
### New mailbox not receiving mail
- Two steps are required — confirm both were done:
1. Mailbox created in Mailcow as `user@mail.domain.com`
2. Forwarder created in MXRoute as `user@domain.com` → `user@mail.domain.com`
- If the MXRoute forwarder is missing, inbound mail silently goes nowhere
---
## Related Documentation
- [MailCow Configuration](./mailcow)
- [MailCow Security Hardening](./mailcow-security-hardening)
- [Mail Setup — nucking-futz.com](./mail-setup-nucking-futz)
- [OPNsense Firewall](./opnsense-firewall) — ATT_Mail static IP allocation

490
False Grimoire/Netgrimoire/Services/MailCow/MailCOw_Install.md Normal file
View file

@ -0,0 +1,490 @@
---
title: Mailcow Dockerized Install and Config
description:
published: true
date: 2026-02-25T21:05:48.256Z
tags:
editor: markdown
dateCreated: 2026-02-25T21:05:38.864Z
---
# MailCow — Installation & Configuration
**Host:** docker4 (192.168.5.16)
**Hostname:** hermes.netgrimoire.com
**Admin URL:** https://mail.netgrimoire.com
**Version:** 2025-10a (update 2026-01 available as of documentation date)
**Installed:** /opt/mailcow-dockerized
**Timezone:** America/Chicago
**Architecture:** x86_64
**CPU:** 16 cores
**RAM:** 30.63 GB
**Disk:** /dev/nvme0n1p2 — 442G / 502G used (93% — monitor this)
---
## Overview
Mailcow runs as a Docker stack on docker4, attached to the `netgrimoire` overlay network. All containers use `restart: unless-stopped` via a compose override. Outbound mail routes through MXRoute via sender-dependent transports. Inbound mail arrives from MXRoute which acts as the public-facing inbound gateway (solving residential AT&T IP filtering issues with banks).
See [MXRoute Master Configuration](./mxroute-master) for full inbound/outbound/DNS detail per domain.
---
## Installation Paths
| Path | Purpose |
|------|---------|
| `/opt/mailcow-dockerized/` | Mailcow root |
| `/opt/mailcow-dockerized/mailcow.conf` | Primary configuration file |
| `/opt/mailcow-dockerized/docker-compose.yml` | Base compose (do not edit) |
| `/opt/mailcow-dockerized/docker-compose.override.yml` | Local overrides — network and restart policy |
| `/opt/mailcow-dockerized/data/conf/postfix/extra.cf` | Persistent Postfix overrides |
| `/opt/mailcow-dockerized/data/conf/postfix/main.cf` | Postfix base config (managed by Mailcow) |
| `/opt/mailcow-dockerized/data/conf/rspamd/` | Rspamd configuration |
| `/opt/mailcow-dockerized/data/assets/ssl/` | TLS certificates |
---
## mailcow.conf — Key Settings
```ini
MAILCOW_HOSTNAME=hermes.netgrimoire.com
MAILCOW_PASS_SCHEME=BLF-CRYPT
# Database
DBNAME=mailcow
DBUSER=mailcow
DBPASS=mg7Z8W9UsPlOh0S6vF7TmmPb6n1s
DBROOT=JdymsZFFACHkDcOdziQ53QruCTG2
# Redis
REDISPASS=6AduWQsmBYGMKfOi1CNEGQfTE3RH
# Ports — HTTPS runs on 3443, proxied through Caddy
HTTP_PORT=80
HTTP_BIND=
HTTPS_PORT=3443
HTTPS_BIND=
HTTP_REDIRECT=n
# Mail ports (standard)
SMTP_PORT=25
SMTPS_PORT=465
SUBMISSION_PORT=587
IMAP_PORT=143
IMAPS_PORT=993
POP_PORT=110
POPS_PORT=995
SIEVE_PORT=4190
# Internal ports (localhost only)
DOVEADM_PORT=127.0.0.1:19991
SQL_PORT=127.0.0.1:13306
REDIS_PORT=127.0.0.1:7654
# TLS cert coverage
ADDITIONAL_SAN=smtp.*,imap.*
AUTODISCOVER_SAN=y
# ACME / Let's Encrypt
SKIP_LETS_ENCRYPT=n
SKIP_IP_CHECK=y
SKIP_HTTP_VERIFICATION=y
# Services — all enabled
SKIP_CLAMD=n
SKIP_OLEFY=n
SKIP_SOGO=n
SKIP_FTS=n
# FTS (Flatcurve/Xapian)
FTS_HEAP=128
FTS_PROCS=1
# Watchdog
USE_WATCHDOG=y
WATCHDOG_NOTIFY_START=y
WATCHDOG_NOTIFY_BAN=n
WATCHDOG_EXTERNAL_CHECKS=n
# Networking
IPV4_NETWORK=172.22.1
IPV6_NETWORK=fd4d:6169:6c63:6f77::/64
ENABLE_IPV6=false
# Misc
MAILDIR_GC_TIME=7200
MAILDIR_SUB=Maildir
SOGO_EXPIRE_SESSION=480
SOGO_URL_ENCRYPTION_KEY=ojmPfhnM4MYMsA2f
ACL_ANYONE=disallow
ALLOW_ADMIN_EMAIL_LOGIN=n
DOCKER_COMPOSE_VERSION=native
COMPOSE_PROJECT_NAME=mailcow
LOG_LINES=9999
```
---
## docker-compose.override.yml
All services are attached to the external `netgrimoire` overlay network and set to `restart: unless-stopped`. The override does not change any image versions or environment variables — it only adds network membership and restart policy.
```yaml
services:
unbound-mailcow:
networks:
netgrimoire:
restart: unless-stopped
mysql-mailcow:
networks:
- netgrimoire
restart: unless-stopped
redis-mailcow:
networks:
- netgrimoire
restart: unless-stopped
clamd-mailcow:
networks:
- netgrimoire
restart: unless-stopped
rspamd-mailcow:
networks:
- netgrimoire
restart: unless-stopped
php-fpm-mailcow:
networks:
- netgrimoire
restart: unless-stopped
sogo-mailcow:
networks:
- netgrimoire
restart: unless-stopped
dovecot-mailcow:
networks:
- netgrimoire
restart: unless-stopped
postfix-mailcow:
networks:
- netgrimoire
restart: unless-stopped
postfix-tlspol-mailcow:
networks:
- netgrimoire
restart: unless-stopped
memcached-mailcow:
restart: unless-stopped
nginx-mailcow:
networks:
- netgrimoire
restart: unless-stopped
acme-mailcow:
networks:
- netgrimoire
restart: unless-stopped
watchdog-mailcow:
networks:
- netgrimoire
restart: unless-stopped
dockerapi-mailcow:
networks:
- netgrimoire
restart: unless-stopped
olefy-mailcow:
networks:
- netgrimoire
restart: unless-stopped
ofelia-mailcow:
networks:
- netgrimoire
restart: unless-stopped
networks:
netgrimoire:
external: true
driver: overlay
```
---
## Container Image Versions
From `docker-compose.yml` (base file — version 2025-10a):
| Service | Image |
|---------|-------|
| unbound-mailcow | ghcr.io/mailcow/unbound:1.24 |
| mysql-mailcow | mariadb:10.11 |
| redis-mailcow | redis:7.4.6-alpine |
| clamd-mailcow | ghcr.io/mailcow/clamd:1.71 |
| rspamd-mailcow | ghcr.io/mailcow/rspamd:2.4 |
| php-fpm-mailcow | ghcr.io/mailcow/phpfpm:1.94 |
| sogo-mailcow | ghcr.io/mailcow/sogo:1.136 |
| dovecot-mailcow | ghcr.io/mailcow/dovecot:2.35 |
| postfix-mailcow | ghcr.io/mailcow/postfix:1.81 |
| postfix-tlspol-mailcow | ghcr.io/mailcow/postfix-tlspol:1.0 |
| memcached-mailcow | memcached:alpine |
| nginx-mailcow | ghcr.io/mailcow/nginx:1.05 |
| acme-mailcow | ghcr.io/mailcow/acme:1.94 |
| netfilter-mailcow | ghcr.io/mailcow/netfilter:1.63 |
| watchdog-mailcow | ghcr.io/mailcow/watchdog:2.09 |
| dockerapi-mailcow | ghcr.io/mailcow/dockerapi:2.11 |
| olefy-mailcow | ghcr.io/mailcow/olefy:1.15 |
| ofelia-mailcow | mcuadros/ofelia:latest |
---
## Postfix Configuration
### extra.cf
```
myhostname = hermes.netgrimoire.com
```
> The MXRoute trusted network entries should also be here. Current extra.cf only contains myhostname — confirm mynetworks is set correctly or add the MXRoute IP ranges if not already present via the UI.
### Key Postfix Settings (from running config)
```
mynetworks = 127.0.0.0/8 172.22.1.0/24 10.0.1.0/24 [::1]/128 [fd4d:6169:6c63:6f77::]/64 [fe80::]/64
message_size_limit = 104857600 # 100MB
mailbox_size_limit = 0 # unlimited
bounce_queue_lifetime = 1d
maximal_queue_lifetime = 5d
delay_warning_time = 4h
postscreen_dnsbl_threshold = 6
postscreen_dnsbl_action = enforce
postscreen_greet_action = enforce
smtpd_relay_restrictions = permit_mynetworks, permit_sasl_authenticated, defer_unauth_destination
disable_vrfy_command = yes
broken_sasl_auth_clients = yes
```
---
## Domains
10 domains configured. All active.
| Domain | Mailboxes | Sender-Dependent Transport | Created |
|--------|-----------|---------------------------|---------|
| bamalady.com | 0 / 10 | *(not confirmed)* | — |
| bill740.com | 1 / 10 | *(not confirmed)* | — |
| florosafd.org | 4 / 10 | ID 4: heracles.mxrouting.net:587 (relay@florosafd.org) | 2025-11-21 |
| gnarlypandaproductions.com | 2 / 10 | ID 5: heracles.mxrouting.net:587 (relay@gnarlypandaproductions.com) | 2025-11-21 |
| netgrimoire.com | 2 / 10 | ID 2: heracles.mxrouting.net:587 (relay@netgrimoire.com) | 2025-11-21 |
| nucking-futz.net | 0 / 10 | *(not confirmed)* | — |
| pncfishandmore.com | 4 / 10 | ID 6: heracles.mxrouting.net:587 (relay@pncfishandmore.com) | — |
| pncharris.com | 4 / 10 | ID 3: heracles.mxrouting.net:587 (passer@pncharris.com) | 2025-11-21 |
| pncharrisenterprises.com | 2 / 10 | *(not confirmed from screenshots)* | — |
| wasted-bandwidth.net | 1 / 10 | ID 1: heracles.mxrouting.net:587 (relay@wasted-bandwidth.net) | — |
> MXRoute relay hostname is `heracles.mxrouting.net:587` — note this differs from the generic `smtp.mxroute.com` placeholder used in setup docs. Always use `heracles.mxrouting.net:587` for this account.
---
## Mailboxes
19 active mailboxes across all domains:
| Mailbox | Messages | Domain |
|---------|----------|--------|
| bill@bill740.com | 1 | bill740.com |
| chieflee@florosafd.org | 2124 | florosafd.org |
| cindy@pncfishandmore.com | 1109 | pncfishandmore.com |
| cindy@pncharris.com | 33797 | pncharris.com |
| cindy@pncharrisenterprises.com | 819 | pncharrisenterprises.com |
| dads_attic@pncharris.com | 0 | pncharris.com |
| jim.harris@florosafd.org | 8 | florosafd.org |
| kyle@gnarlypandaproductions.com | 486 | gnarlypandaproductions.com |
| kyle@pncfishandmore.com | 110 | pncfishandmore.com |
| kyle@pncharris.com | 31182 | pncharris.com |
| phil@florosafd.org | 5 | florosafd.org |
| phil@gnarlypandaproductions.com | 5 | gnarlypandaproductions.com |
| phil@netgrimoire.com | 1 | netgrimoire.com |
| phil@pncfishandmore.com | 10 | pncfishandmore.com |
| phil@pncharris.com | 3210 | pncharris.com |
| phil@pncharrisenterprises.com | 1 | pncharrisenterprises.com |
| times@florosafd.org | 191 | florosafd.org |
| traveler@netgrimoire.com | 3 | netgrimoire.com |
| traveler@wasted-bandwidth.net | 138 | wasted-bandwidth.net |
---
## Aliases
| ID | Alias | Target Domain | Internal |
|----|-------|---------------|---------|
| 7 | cindy@bamalady.com | bamalady.com | No |
---
## Sender-Dependent Transports
All outbound relay routes through `heracles.mxrouting.net:587`. This is your MXRoute server hostname — use this exact value when adding new transports.
| ID | Host | Username | Password |
|----|------|----------|----------|
| 1 | heracles.mxrouting.net:587 | relay@wasted-bandwidth.net | dZ4yLYznVvgSJtqWZJFA |
| 2 | heracles.mxrouting.net:587 | relay@netgrimoire.com | TVGCnJp9SxRbWU8EhkMw |
| 3 | heracles.mxrouting.net:587 | passer@pncharris.com | bBJtPhrGkHvvhxhukkae |
| 4 | heracles.mxrouting.net:587 | relay@florosafd.org | 2Fe8XMyaeh6Z5dvdHYdq |
| 5 | heracles.mxrouting.net:587 | relay@gnarlypandaproductions.com | vG5ZsUQhRWD2UyzLPsqA |
| 6 | heracles.mxrouting.net:587 | relay@pncfishandmore.com | *(confirm from MXRoute panel)* |
---
## DKIM Keys
Two DKIM selectors are configured per domain — one for Mailcow (selector: `dkim`) and one added separately for MXRoute outbound signing. The Mailcow-managed keys use selector `dkim._domainkey`.
### pncharris.com
```
v=DKIM1;k=rsa;t=s;s=email;p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAqhgQV7r+KKQwJceWenZ3FNq8AsllgW6cIm/0jpsLT62vF1yy0nh2MdhjYgQAX2MK9HHYzNZcCB3+OPpqBbXeNbSDckxB/dC+z/vboMHrJmYonfaSYshZjSR80V/a2Yoq+hiXQ9eBcuOggENtMm4XvEsl/vOWLBMfasqe+X11gzQBeRv1tTaXJB0C4i7tAcfi0O/AxH8QFTr2099+k2iepn8J15ukk1zu4zemBJj4Z3uFTNnBP8YpgKbYoUDyMVIKIxGjANVBBypcrMKavpQ4F1JLhgGFhWAsAuFRwZsnOaftZyMuzAZxM37DTd/bF2WanmK3Xe75SN5uOnEXjuzW/wIDAQAB
```
### netgrimoire.com
```
v=DKIM1;k=rsa;t=s;s=email;p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAoJ9YKqV9+6gOcVKI+UJ0TRcMmergxU8HLO+mwTMfqOhblsEcDPO60c8ya24iIXg51AA2k5Xcbb0bLScaaIi0P/TRzP/bonAZkPS1Y8Fx1se9dikTsA9Lazho u6DvoFkkV/IPH1ZNg68Cd9teAD5tvoY18OSneJJsocXwFo57c+XccUaTxjpV7eReuT4da7iNHMmUmZNfKenxVMKD740zrDJAeAsXtEb/71CochHYSm+qAvuG9/WPixJbMsJLF/iVhV3Byp0LCrB+CwGTwnsiUcd7QpuD6rRs/7zzdGBtoN22m/j390GimFstYvB61I20h8sHWGAG66dLko6Sgvs47wIDAQAB
```
### gnarlypandaproductions.com
```
v=DKIM1;k=rsa;t=s;s=email;p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA...
```
*(scroll cut off in screenshot — retrieve full key from Mailcow UI → Edit domain → bottom of page)*
> All other domain DKIM keys should be retrieved from the Mailcow domain edit page and recorded here for disaster recovery completeness.
---
## Network Configuration
Mailcow containers join the `netgrimoire` external overlay network, allowing communication with other Docker Swarm services (Caddy reverse proxy, etc.) without exposing ports directly to the host network.
**Internal Docker network:** `172.22.1.0/24`
Key container IPs within the mailcow-network:
- unbound: 172.22.1.254
- redis: 172.22.1.249
- sogo: 172.22.1.248
- dovecot: 172.22.1.250
- postfix: 172.22.1.253
**IPv6:** disabled (`ENABLE_IPV6=false`)
---
## Caddy Reverse Proxy
Mailcow's nginx listens on HTTPS port 3443 internally. Caddy proxies external requests to it. Mailcow handles its own TLS for direct mail client connections (IMAP 993, SMTP 465/587).
The admin UI at `mail.netgrimoire.com` is proxied through Caddy on the `netgrimoire` overlay network.
---
## Updating Mailcow
```bash
cd /opt/mailcow-dockerized
# Pull latest
git fetch origin
git checkout origin/master
# Update containers
docker compose pull
./update.sh
```
> As of documentation date, version 2026-01 is available. Current running version is 2025-10a. Update when convenient — check the [MailCow changelog](https://github.com/mailcow/mailcow-dockerized/releases) for breaking changes first.
Monthly update check is recommended. MailCow had multiple security vulnerabilities in 2025 — staying current is important.
---
## Common Operations
### Restart all containers
```bash
cd /opt/mailcow-dockerized
docker compose restart
```
### Restart single container (e.g. after extra.cf change)
```bash
docker compose restart postfix-mailcow
```
### View logs
```bash
# Postfix
docker compose logs postfix-mailcow -f
# Dovecot
docker compose logs dovecot-mailcow -f
# All containers
docker compose logs -f
```
### Check queue
```bash
docker exec mailcow-postfix-mailcow-1 postqueue -p
```
### Flush queue
```bash
docker exec mailcow-postfix-mailcow-1 postqueue -f
```
### Check container health
```bash
docker compose ps
```
---
## Known Gotchas
**Disk usage is at 93%.** The nvme0n1p2 volume has 442G used of 502G. This needs attention — vmail storage grows over time and garbage collection runs hourly but only removes items older than 7200 minutes (5 days). Monitor this and consider quota enforcement per mailbox if growth continues.
**extra.cf is minimal.** The MXRoute trusted network IPs should be confirmed in the running Postfix config. The `mynetworks` value from `postconf` shows `10.0.1.0/24` is already trusted — confirm whether MXRoute IP ranges `69.167.160.0/19` and `198.54.120.0/22` are included. If not, add them to extra.cf and restart postfix.
**MXRoute relay hostname.** The actual relay hostname for this account is `heracles.mxrouting.net:587` — not the generic `smtp.mxroute.com` placeholder. All 6 transports use `heracles.mxrouting.net:587`. Use this exact hostname for any new transport entries.
**pncharris.com uses passer@ not relay@.** Transport ID 3 for pncharris.com authenticates as `passer@pncharris.com`, not `relay@pncharris.com`. This is intentional — the relay@ account exists but passer@ is the current active relay credential.
**HTTPS on port 3443.** Mailcow's web UI is not on the standard 443 — it binds to 3443 and Caddy handles the public-facing 443 proxy. Direct access to the UI requires going through Caddy or using the internal port.
**nucking-futz.net vs nucking-futz.com.** The domains list shows `nucking-futz.net` but the intended new domain is `nucking-futz.com`. Verify which is actually configured and correct if needed.
**bamalady.com and bill740.com** have no transport assigned in the screenshots. Confirm whether these domains need MXRoute relay configured.
---
## Related Documentation
- [MXRoute Master Configuration](./mxroute-master) — per-domain DNS, inbound forwarding, outbound relay credentials
- [Mail Setup — nucking-futz.com](./mail-setup-nucking-futz) — new domain setup guide
- [MailCow Security Hardening](./mailcow-security-hardening)
- [Caddy Reverse Proxy](./caddy-reverse-proxy) — proxies mail.netgrimoire.com to port 3443
- [OPNsense Firewall](./opnsense-firewall) — ATT_Mail static IP, port forwarding rules

391
False Grimoire/Netgrimoire/Services/MailCow/Mailcow_Hardening.md Normal file
View file

@ -0,0 +1,391 @@
---
title: MailCow Hardening
description: Securing Mailcow
published: true
date: 2026-02-23T21:56:32.211Z
tags:
editor: markdown
dateCreated: 2026-02-23T21:56:22.997Z
---
# MailCow Security Hardening
**Service:** MailCow Dockerized
**Host:** 192.168.5.16 (MailCow_Ngnx alias)
**Relay:** MXRoute (outbound only)
**Last Reviewed:** February 2026
---
## Overview
Running MailCow with MXRoute as an outbound relay creates a specific threat model that's different from either a fully self-hosted or fully managed setup. Your server receives inbound directly (MX points to your IP), stores all mailboxes locally, and hands outbound to MXRoute. This means you carry the risk surface of both — inbound SMTP exposure plus the credential and reputation exposure of a relay relationship.
The security areas that matter most for this setup:
| Area | Risk | Priority |
|---|---|---|
| DNS authentication (SPF/DKIM/DMARC) | Spoofing, deliverability failure, relay abuse | 🔴 Critical |
| MTA-STS + TLS-RPT | SMTP downgrade attacks on inbound | 🔴 Critical |
| MXRoute relay credential security | Relay hijacking, spam abuse on your reputation | 🔴 Critical |
| Mailcow admin hardening | Account takeover, open relay creation | 🔴 Critical |
| Postfix TLS hardening | Weak cipher negotiation | 🟡 High |
| Nginx header hardening | XSS, clickjacking on webmail | 🟡 High |
| Rspamd tuning | Inbound spam, outbound policy enforcement | 🟡 High |
| DMARC reporting | Visibility into spoofing and misdelivery | 🟡 High |
| ClamAV / attachment scanning | Malware distribution via your domain | 🟢 Medium |
| Rate limiting | Compromised account spam runs | 🟢 Medium |
---
## DNS Authentication
This is the foundation. If any of these are misconfigured your mail either doesn't deliver or your domain gets spoofed. With MXRoute in the mix the SPF record requires special attention.
### SPF — Include Both Sources
Your SPF must authorize **both** your own IP (for any direct sends) and MXRoute's sending infrastructure:
```dns
@ IN TXT "v=spf1 ip4:YOUR_ATT_MAIL_IP include:mxroute.com ~all"
```
Replace `YOUR_ATT_MAIL_IP` with the static IP you've dedicated to mail (ATT_Mail virtual IP). The `include:mxroute.com` covers MXRoute's sending servers.
> ⚠ Do not use `-all` (hard fail) until you have confirmed all your sending sources are covered. Use `~all` (softfail) initially, then tighten after verifying DMARC reports show no legitimate sources failing.
> ⚠ SPF has a **10 DNS lookup limit**. Each `include:` costs lookups. If you add more includes (e.g. transactional services), check your SPF lookup count at [mxtoolbox.com/spf](https://mxtoolbox.com/spf.aspx).
### DKIM — Two Selectors for Two Signers
Because MXRoute re-signs outbound mail with their own DKIM key, you need a DKIM record for both signers:
| Selector | Signer | Where to get the key |
|---|---|---|
| `mailcow._domainkey` | MailCow (inbound, internal sends) | MailCow UI → Configuration → ARC/DKIM Keys |
| `mxroute._domainkey` (or `x._domainkey`) | MXRoute (outbound relay) | MXRoute control panel |
Add both as TXT records. Having both means DMARC passes regardless of which path the mail took.
> ✓ MailCow lets you choose the DKIM selector name. Use `mailcow` as the selector to avoid confusion with the MXRoute selector.
### DMARC — Start Monitoring, Then Enforce
DMARC ties SPF and DKIM together and tells receiving servers what to do with failures. Start in monitoring mode, review reports for 2–4 weeks, then advance to enforcement.
**Phase 1 — Monitor (add immediately):**
```dns
_dmarc IN TXT "v=DMARC1; p=none; rua=mailto:dmarc-reports@yourdomain.com; ruf=mailto:dmarc-failures@yourdomain.com; fo=1"
```
**Phase 2 — Quarantine (after reviewing reports, no legitimate failures):**
```dns
_dmarc IN TXT "v=DMARC1; p=quarantine; pct=100; rua=mailto:dmarc-reports@yourdomain.com; fo=1"
```
**Phase 3 — Reject (final enforcement):**
```dns
_dmarc IN TXT "v=DMARC1; p=reject; pct=100; rua=mailto:dmarc-reports@yourdomain.com; fo=1"
```
> ✓ `fo=1` requests forensic reports on any authentication failure — more detail for debugging.
**DMARC Report Processing:** Raw DMARC reports are XML and not human-readable. Use one of these free tools to process them:
- [Postmark DMARC](https://dmarc.postmarkapp.com/) — free, email-based weekly digest
- [dmarcian.com](https://dmarcian.com) — free tier, dashboard view
- Self-hosted: [Parsedmarc](https://github.com/domainaware/parsedmarc) → send to Graylog/Grafana
---
## MTA-STS (MailCow September 2025+)
MTA-STS forces other mail servers to use TLS when delivering to you, preventing downgrade attacks that try to force plaintext SMTP. The September 2025 MailCow update added the `postfix-tlspol-mailcow` container which enforces MTA-STS on **outbound** connections too.
### What You Need
**1. DNS records** — three records for each domain:
```dns
# For your mail server's hostname domain (e.g. netgrimoire.com)
mta-sts IN CNAME mail.netgrimoire.com.
_mta-sts IN TXT "v=STSv1; id=20260223"
_smtp._tls IN TXT "v=TLSRPTv1; rua=mailto:tls-reports@netgrimoire.com"
```
The `id` value in `_mta-sts` is a version string — update it (e.g. to today's date) whenever you change your MTA-STS policy.
**2. Policy file** — served by MailCow's nginx at `https://mta-sts.yourdomain.com/.well-known/mta-sts.txt`:
```bash
# On your MailCow host:
mkdir -p /opt/mailcow-dockerized/data/web/.well-known/
cat > /opt/mailcow-dockerized/data/web/.well-known/mta-sts.txt << 'EOF'
version: STSv1
mode: enforce
max_age: 86400
mx: mail.netgrimoire.com
EOF
```
Start with `mode: testing` for the first week, then switch to `mode: enforce`.
**3. For additional domains** — add CNAMEs pointing to your primary domain's records:
```dns
# For each additional mail domain you host on MailCow:
mta-sts.otherdomain.com IN CNAME mail.netgrimoire.com.
_mta-sts.otherdomain.com IN CNAME _mta-sts.netgrimoire.com.
_smtp._tls.otherdomain.com IN CNAME _smtp._tls.netgrimoire.com.
```
> ✓ TLS-RPT (`_smtp._tls` TXT record) sends you reports about TLS failures when other servers connect to you. Pipe these to Graylog or Postmark for visibility.
---
## MXRoute Relay Security
This is the most overlooked area. Your MXRoute credentials can send mail as your domain — if they're compromised, someone else is spamming from your reputation.
### Credential Hardening
- Use a **unique, strong password** for your MXRoute account — not shared with anything else
- Store the MXRoute SMTP credentials in MailCow's relay configuration only, not in any config file or environment variable that gets committed to git
- If MXRoute supports API tokens or app passwords, use those instead of your main account password
### Relay Configuration in MailCow
In MailCow UI: **Configuration → Routing → Sender-Dependent Transports**
Verify the relay is configured to authenticate via TLS (port 587 with STARTTLS or port 465 with SSL). Do not relay over port 25 without authentication.
```
# What the relay entry should look like in Postfix terms:
# relayhost = [smtp.mxroute.com]:587
# smtp_sasl_auth_enable = yes
# smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
# smtp_tls_security_level = encrypt ← ensures TLS is required, not optional
```
> ⚠ Set `smtp_tls_security_level = encrypt` (not `may`) so the connection to MXRoute is always encrypted. If the TLS negotiation fails, Postfix should reject rather than fall back to plaintext.
### Rate Limiting (Prevent Relay Abuse if Account Compromised)
Add rate limits in MailCow UI: **Configuration → Mail Setup → Domains → [your domain] → Rate Limit**
| Setting | Recommended Value | Notes |
|---|---|---|
| Outbound messages/hour | 500 | Adjust for your actual sending volume |
| Outbound messages/day | 2000 | A sudden spike above this = red flag |
This doesn't stop abuse but limits blast radius if a mailbox is compromised and starts spamming through MXRoute.
---
## MailCow Admin Hardening
### Two-Factor Authentication
Enable 2FA on the admin account and all mailbox accounts that have access to the admin panel.
MailCow UI: **Edit mailbox → Two-Factor Authentication → TOTP**
> ⚠ There was a session fixation vulnerability in the MailCow web panel (GHSA-23c8-4wwr-g3c6, January 2025) and a critical SSTI vulnerability (GHSA-8p7g-6cjj-wr9m, July 2025). Both require staying current on updates. Enable auto-updates or check the MailCow blog monthly.
### Restrict Admin UI to Internal Network
The MailCow admin panel should not be reachable from the public internet. Access should require being on your internal network or connected via WireGuard.
In OPNsense, add a firewall rule blocking external access to port 443 on 192.168.5.16 except from your static admin IP or WireGuard peers.
Alternatively, configure MailCow's nginx to restrict the admin path by IP:
```nginx
# In data/conf/nginx/includes/site-defaults.conf
# Add inside the server block for the admin panel:
location /admin {
allow 192.168.3.0/24;
allow 192.168.5.0/24;
allow 192.168.32.0/24; # WireGuard peers
deny all;
}
```
### API Key Rotation
If you use the MailCow API (for automation or Netgrimoire tooling), generate a dedicated read-only key where possible, and rotate keys annually or after any suspected compromise.
---
## Postfix TLS Hardening
Add to `/opt/mailcow-dockerized/data/conf/postfix/extra.cf`:
```ini
# Enforce TLS 1.2+ and strong ciphers
tls_high_cipherlist = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256
tls_preempt_cipherlist = yes
# Inbound SMTP (smtpd) — receiving from other mail servers
smtpd_tls_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1
smtpd_tls_ciphers = high
smtpd_tls_mandatory_ciphers = high
# Outbound SMTP (smtp) — delivery to MXRoute and direct sends
smtp_tls_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1
smtp_tls_ciphers = high
smtp_tls_mandatory_ciphers = high
# Require encryption on the MXRoute relay connection
smtp_tls_security_level = encrypt
```
After editing, restart Postfix:
```bash
cd /opt/mailcow-dockerized
docker compose restart postfix-mailcow
```
---
## Nginx Header Hardening
Add to `/opt/mailcow-dockerized/data/conf/nginx/includes/site-defaults.conf`:
```nginx
# Strong SSL ciphers only
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
ssl_conf_command Options PrioritizeChaCha;
# HSTS — include subdomains if all your services use HTTPS
add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload";
# Disable X-XSS-Protection (deprecated, CSP replaces it)
add_header X-XSS-Protection "0";
# Deny unused browser permissions
add_header Permissions-Policy "accelerometer=(), ambient-light-sensor=(), autoplay=(), battery=(), camera=(), geolocation=(), gyroscope=(), magnetometer=(), microphone=(), payment=(), usb=()";
# Content Security Policy — if NOT using Gravatar with SOGo
add_header Content-Security-Policy "default-src 'none'; connect-src 'self' https://api.github.com; font-src 'self' https://fonts.gstatic.com; img-src 'self' data:; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline' https://fonts.googleapis.com; frame-ancestors 'none'; upgrade-insecure-requests; block-all-mixed-content; base-uri 'none'";
# Cross-origin isolation headers
add_header Cross-Origin-Resource-Policy same-origin;
add_header Cross-Origin-Opener-Policy same-origin;
add_header Cross-Origin-Embedder-Policy require-corp;
# Disable gzip to prevent BREACH attack
# Change gzip on; → gzip off; in the main nginx conf
```
> ⚠ The December 2025 MailCow update already removed the deprecated `X-XSS-Protection` header from defaults. If you're current, you may already have this. Check before duplicating.
After editing, restart nginx:
```bash
docker compose restart nginx-mailcow
```
---
## Rspamd Tuning
Rspamd is MailCow's spam filter. The defaults are reasonable but a few adjustments improve both inbound protection and outbound policy enforcement.
### Key Settings to Review
Navigate to **MailCow UI → Configuration → Rspamd UI** (or directly at `https://mail.yourdomain.com/rspamd/`)
**Actions → Score Thresholds:**
| Action | Default | Recommended |
|---|---|---|
| Greylist | 4 | 3 |
| Add header | 6 | 5 |
| Reject | 15 | 12 |
Lowering the reject threshold from 15 to 12 catches more aggressive spam while avoiding false positives.
**Modules to enable/verify:**
| Module | Purpose |
|---|---|
| DKIM verification | Verify incoming DKIM signatures |
| SPF | Verify incoming SPF |
| DMARC | Enforce DMARC on inbound |
| MX Check | Verify sending domain has a valid MX |
| RBL (Realtime Blacklists) | Check sending IPs against blocklists |
| Greylisting | Temporary reject new senders (forces retry) |
### Add CrowdSec as an Rspamd Feed
If you also have the CrowdSec bouncer running on the MailCow host (or can reach it), you can feed CrowdSec decisions into Rspamd to reject mail from banned IPs. This is advanced but powerful — see the [CrowdSec Bouncer for Rspamd](https://hub.crowdsec.net) hub entry.
---
## Deliverability Verification
Run these checks after making any DNS or config changes:
| Tool | What It Checks | URL |
|---|---|---|
| MXToolbox | SPF, DKIM, DMARC, MX, PTR, blacklists | mxtoolbox.com |
| mail-tester.com | Send a test email, get a 1–10 score | mail-tester.com |
| Port25 verifier | Send to check-auth@verifier.port25.com | Email-based |
| DKIM validator | Validates DKIM signature | dkimvalidator.com |
| Google Postmaster Tools | Gmail reputation monitoring (requires setup) | postmaster.google.com |
| Microsoft SNDS | Outlook/Hotmail reputation | sendersupport.olc.protection.outlook.com |
> ✓ Aim for 9–10/10 on mail-tester.com. Anything below 8 indicates a misconfiguration that will hurt deliverability.
---
## Keeping MailCow Updated
MailCow has had several critical security vulnerabilities in 2025 (session fixation, SSTI, password reset poisoning). Staying current is non-negotiable.
```bash
cd /opt/mailcow-dockerized
# Pull latest images
docker compose pull
# Apply update
./update.sh
# Or if using the newer helper:
docker compose up -d
```
> ✓ Subscribe to the [MailCow blog](https://mailcow.email/posts/) or watch the [GitHub releases](https://github.com/mailcow/mailcow-dockerized/releases) for security advisories. The update cadence is roughly monthly.
Set up a cron job or Monit check to alert you when MailCow is more than 30 days behind the latest release.
---
## Checklist Summary
| Item | Status |
|---|---|
| SPF includes both own IP and mxroute.com | ☐ |
| Two DKIM selectors (mailcow + mxroute) | ☐ |
| DMARC in monitoring mode, advancing to reject | ☐ |
| DMARC reports being processed (Postmark/dmarcian) | ☐ |
| MTA-STS policy published and enforced | ☐ |
| TLS-RPT record in DNS | ☐ |
| MXRoute relay connection uses TLS/encrypt level | ☐ |
| Admin UI restricted to internal network | ☐ |
| 2FA on admin and all privileged accounts | ☐ |
| Postfix TLS 1.2+ enforced via extra.cf | ☐ |
| Nginx security headers added | ☐ |
| Rate limits set on outbound per-domain | ☐ |
| MailCow updated to latest (monthly check) | ☐ |
| Rspamd thresholds reviewed | ☐ |
| PTR/rDNS record matches mail hostname | ☐ |
---
## Related Documentation
- [OPNsense Firewall](./opnsense-firewall) — dedicated ATT_Mail virtual IP, port NAT
- [CrowdSec](./crowdsec) — IP reputation blocking at firewall level
- [Graylog](./graylog) — DMARC report and TLS-RPT ingestion target
- [Caddy Reverse Proxy](./caddy-reverse-proxy) — if MailCow webmail is proxied through Caddy

154
False Grimoire/Netgrimoire/Services/MailCow/Mailcow_MXRoute.md Normal file
View file

@ -0,0 +1,154 @@
---
title: Forwarding Mailcow through MXRoute
description: Maintaining reputation
published: true
date: 2026-02-20T04:10:37.730Z
tags:
editor: markdown
dateCreated: 2026-02-15T01:42:12.478Z
---
# MXroute as Forwarder for Mailcow
## Overview
Configuration guide for setting up MXroute as an email forwarder for Mailcow-hosted domains.
---
## Mailcow Configuration
### 1. Create Domain
1. Navigate to Mailcow admin panel → Domains
2. Add new domain
3. Select the previously created sender-dependent transport
4. Record the generated DKIM key
### 2. Add DKIM Key
- Configure DKIM for both Mailcow and MXroute
- Use different selectors for each (e.g., `default` for Mailcow, `mxroute` for MXroute)
---
## DNS Configuration
### Required DNS Records
#### A Record
```
mail.yourdomain.com → [Your Mailcow Server IP]
```
#### MX Record
```
yourdomain.com → hermes.netgrimoire.com (Priority: 10)
```
#### CNAME Records
```
imap.yourdomain.com → mail.yourdomain.com
smtp.yourdomain.com → mail.yourdomain.com
webmail.yourdomain.com → mail.yourdomain.com
autodiscover.yourdomain.com → mail.yourdomain.com
autoconfig.yourdomain.com → mail.yourdomain.com
```
#### TXT Records
**SPF Record**
```
v=spf1 ip4:192.168.5.16 ip4:24.249.193.115 include:mxroute.com -all
```
**DMARC Record** (`_dmarc.yourdomain.com`)
```
v=DMARC1; p=reject; rua=mailto:admin@netgrimoire.com
```
**DKIM Record** (`default._domainkey.yourdomain.com`)
```
v=DKIM1; t=s; p=[YOUR_PUBLIC_KEY]
```
---
## Example: gnarlypandaproductions.com
| Record Type | Name | Value |
|-------------|------|-------|
| MX | @ | hermes.netgrimoire.com (Priority: 10) |
| A | mail | [Mailcow Server IP] |
| CNAME | webmail | mail.gnarlypandaproductions.com |
| CNAME | imap | mail.gnarlypandaproductions.com |
| CNAME | smtp | mail.gnarlypandaproductions.com |
| CNAME | roundcube | roundcube.netgrimoire.com |
| TXT | @ | v=spf1 ip4:192.168.4.11 ip4:24.249.193.114 include:mxroute.com -all |
| TXT | _dmarc | v=DMARC1; p=reject; rua=mailto:admin@gnarlypandaproductions.com |
| TXT | default._domainkey | v=DKIM1; t=s; p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA3D3vyPoBHB4eMSMq8HygVWHzYbketRX4yjk9wV4bdaar0/c89dK230FMOW6zVXEsY1sXKFk1kBxerHVw0wY8qnQyooHgINEQcEXrtB/x93Sl/cqBQXk+PHOIOymQwgni8WCUhCSnvunxXK8qX5f9J56qzd0/wpY2WSEHho+XrnQjc+c7HMvkcC3+nKJe59ZNgvQW/Y9B/L6zFDjAp+QOUYp9wwX4L+j1T4fQSygYxAJZ0aIoR8FsbOuXc38pht99HyUnYwH08HoK7xv3DL2BrVo3KVZ7xMe2S4YMxd1HkJz2evbV/ziNsJcKW/le3fFS7mza09yJXDLDcLOKLXbYUQIDAQAB |
---
## Port Configuration
**IMAP (SSL/TLS)**
- Port: 993
**SMTP (SSL/TLS)**
- Port: 465
---
## Relay Accounts
Configuration for sender-dependent transports:
| Domain | Username | Password |
|--------|----------|----------|
| pncharris.com | relay@pncharris.com | H@rv3yD)G123 |
| pncharris.com | forwarder@pncharris.com | (see password history) |
| pncharris.com | passer@pncharris.com | bBJtPhrGkHvvhxhukkae |
| wasted-bandwidth.net | relay@wasted-bandwidth.net | dZ4yLYznVvgSJtqWZJFA |
| netgrimoire.com | relay@netgrimoire.com | TVGCnJp9SxRbWU8EhkMw |
| florosafd.org | relay@florosafd.org | 2Fe8XMyaeh6Z5dvdHYdq |
| gnarlypandaproductions.com | relay@gnarlypandaproductions.com | vG5ZsUQhRWD2UyzLPsqA |
### Password History (passer@pncharris.com)
Latest passwords (most recent last):
- !5!,_\*zDyLEhhR4
- sh7dXWnTPqbkDGsTcwtn
- MY3V8p69b2HYksygxhXX
- RS6U2GU6rcYe3THKKgYx
- yzqNysrd73yzWptVEZ5H (current)
### Additional Credentials
- kylr pncharris: -,68,incTeR
- G4@rlyf1ng3r
---
## Troubleshooting
### Common Issues
1. **Email not sending through MXroute**
- Verify SPF record includes MXroute IPs
- Check sender-dependent transport is selected in Mailcow
- Confirm relay account credentials are correct
2. **DKIM verification failing**
- Ensure both Mailcow and MXroute DKIM records are published
- Verify different selectors are used
- Check for DNS propagation (24-48 hours)
3. **DMARC failures**
- Confirm SPF and DKIM are properly aligned
- Review DMARC reports sent to rua address
---
## References
- [Mailcow Documentation](https://docs.mailcow.email/)
- [MXroute Documentation](https://mxroutedocs.com/)
- [SPF Record Syntax](https://www.rfc-editor.org/rfc/rfc7208)
- [DKIM Documentation](https://www.rfc-editor.org/rfc/rfc6376)
- [DMARC Documentation](https://www.rfc-editor.org/rfc/rfc7489)

401
False Grimoire/Netgrimoire/Services/MailCow/Sample_Domain_Setup.md Normal file
View file

@ -0,0 +1,401 @@
---
title: Sample Domain Setup
description: Graymutt@nucking-futz.com
published: true
date: 2026-03-16T00:34:08.387Z
tags:
editor: markdown
dateCreated: 2026-02-25T22:02:27.719Z
---
# Mail Setup — nucking-futz.com
## Part 0 — OPNsense: Configure ATT_Mail Secondary IP
Before configuring DNS or Mailcow, the secondary AT&T static IP must be configured in OPNsense as a virtual IP on the WAN interface and NAT rules must be set so only raw SMTP traffic (ports 25, 465, 587, 993, 143) uses this address. Webmail, the Mailcow admin UI, and all other traffic continue to use the primary WAN IP (107.133.34.145).
| Address | Purpose |
|---------|---------|
| 107.133.34.145 | Primary WAN — web, admin, everything else |
| 107.133.34.146 | ATT_Mail — SMTP/IMAP inbound and outbound only |
### Step 0.1 — Add Virtual IP
1. Go to **Interfaces → Virtual IPs → Settings**
2. Click **+ Add**
3. Set the following:
| Field | Value |
|-------|-------|
| Mode | IP Alias |
| Interface | WAN (igc1) |
| Network / Address | `107.133.34.146 / 28` |
| Description | `ATT_Mail` |
4. Click **Save**, then **Apply changes**
> The /28 subnet mask matches the AT&T block (107.133.34.144/28). All 5 static IPs in the block share this mask.
### Step 0.2 — Outbound NAT for SMTP Traffic
This ensures Mailcow's outbound SMTP connections leave through the ATT_Mail IP rather than the primary WAN IP. OPNsense must be in **Hybrid** or **Manual** outbound NAT mode.
1. Go to **Firewall → NAT → Outbound**
2. Confirm mode is set to **Hybrid Outbound NAT** (or Manual — either works)
3. Click **Add** to create a new rule
**Rule for outbound SMTP (port 587 relay to MXRoute):**
| Field | Value |
|-------|-------|
| Interface | WAN |
| TCP/IP Version | IPv4 |
| Protocol | TCP |
| Source | `192.168.5.16 / 32` (Mailcow host) |
| Source Port | any |
| Destination | any |
| Destination Port | 587 |
| Translation / Target | `107.133.34.146` (ATT_Mail) |
| Description | `Mailcow outbound relay via ATT_Mail` |
4. Repeat for port **25** (direct outbound SMTP, if used) and port **465** (SMTPS)
5. Click **Save** and **Apply changes**
### Step 0.3 — Inbound NAT (Port Forwards) for Mail Ports
Route inbound connections on mail ports to Mailcow using the ATT_Mail IP as the external address.
1. Go to **Firewall → NAT → Port Forward**
2. Create rules for each mail port:
| External IP | Port(s) | Forward to | Description |
|-------------|---------|-----------|-------------|
| 107.133.34.146 | 25 | 192.168.5.16:25 | SMTP inbound |
| 107.133.34.146 | 465 | 192.168.5.16:465 | SMTPS inbound |
| 107.133.34.146 | 587 | 192.168.5.16:587 | Submission inbound |
| 107.133.34.146 | 993 | 192.168.5.16:993 | IMAPS |
| 107.133.34.146 | 143 | 192.168.5.16:143 | IMAP (if needed) |
> **Do not** add port forwards for 80, 443, or 3443 (Mailcow admin/webmail ports) on this IP. Those remain on the primary WAN IP via Caddy.
3. Click **Save** and **Apply changes**
### Step 0.4 — Firewall Rules
Ensure the WAN firewall rules permit inbound traffic on the mail ports to the ATT_Mail IP. If you have a default deny-all WAN rule (recommended), add explicit pass rules:
1. Go to **Firewall → Rules → WAN**
2. Add pass rules for each port in the table above with destination `107.133.34.146`
### Step 0.5 — Verify
```bash
# From outside your network, confirm the mail IP is live
telnet 107.133.34.146 25
# Should see: 220 hermes.netgrimoire.com ESMTP
# Confirm primary WAN IP does NOT respond on port 25
telnet 107.133.34.145 25
# Should time out or be refused
# Check that Mailcow outbound connections leave from the ATT_Mail IP
# Send a test to check-auth@verifier.port25.com and inspect the Return-Path
# or check the Received: header — the sending IP should be 107.133.34.146
```
> ⚠ If the verify step shows port 25 still responding on 107.133.34.145, check that no leftover port forward rules exist on the primary WAN IP for mail ports.
---
## Overview
This guide covers complete mail setup for `nucking-futz.com` using MXRoute as the inbound gateway and Mailcow as the mailbox host. MXRoute receives all inbound mail from the internet (solving residential IP filtering issues with banks and financial institutions) and forwards to Mailcow for storage and retrieval. Mailcow handles outbound mail via the MXRoute SMTP relay.
**Architecture:**
```
Inbound: Internet → MXRoute (commercial IP) → Mailcow (192.168.5.16)
Outbound: Mailcow → MXRoute SMTP relay → Internet
```
**Why two domains in Mailcow:**
MXRoute forwarders require a valid destination email address. You cannot forward `graymutt@nucking-futz.com` back to `graymutt@nucking-futz.com` — that loops. The solution is to have Mailcow own a subdomain (`mail.nucking-futz.com`) with its own MX record pointing directly to your server. MXRoute forwards to `graymutt@mail.nucking-futz.com`, Mailcow delivers locally, and an alias domain maps `nucking-futz.com` back so users only ever see and use `graymutt@nucking-futz.com`.
---
## Prerequisites
- MXRoute account active with DirectAdmin access
- Mailcow running at 192.168.5.16
- DNS management access for nucking-futz.com
- Your MXRoute server hostname from your MXRoute welcome email (e.g. `arrow.mxrouting.net`)
---
## Step 1 — DNS Records
Create all DNS records before configuring either service. Keep TTL at 300 during setup — raise to 3600 once confirmed working.
![image.png](/image.png)
![arec.png](/email/arec.png)
![txt.png](/email/txt.png)
### Required DNS Records
| Type | Host | Value | Notes |
|------|------|-------|-------|
| A | `mail` | `YOUR_ATT_MAIL_IP` | Points to Mailcow — MXRoute forwards to this server |
| MX | `@` | `heracles.mxrouting.net (Priority 10)` | Check MXRoute welcome email for exact hostname |
| MX | `@` | `heracles-relay.mxrouting.net (Priority 20)` (priority 20) | Secondary MXRoute server from welcome email |
| MX | `mail` | `mail.nucking-futz.com` (priority 10) | Mailcow handles this subdomain directly |
| CNAME | `imap` | `mail.nucking-futz.com` | Client autoconfiguration |
| CNAME | `smtp` | `mail.nucking-futz.com` | Client autoconfiguration |
| CNAME | `webmail` | `mail.nucking-futz.com` | Roundcube access |
| CNAME | `autodiscover` | `mail.nucking-futz.com` | Outlook autodiscover |
| CNAME | `autoconfig` | `mail.nucking-futz.com` | Thunderbird autoconfig |
| TXT | `@` | `v=spf1 ip4:YOUR_ATT_MAIL_IP include:mxroute.com -all` | SPF — authorizes both Mailcow direct and MXRoute relay |
| TXT | `mail` | `v=spf1 ip4:YOUR_ATT_MAIL_IP -all` | SPF for subdomain — Mailcow sends directly from here |
| TXT | `_dmarc` | `v=DMARC1; p=reject; rua=mailto:admin@netgrimoire.com` | DMARC enforcement |
> DKIM TXT records (two selectors) are added in Steps 2 and 3 after generating keys in Mailcow and MXRoute.
---
## Step 2 — Mailcow Configuration
### 2.1 Add the Subdomain as Primary Domain
Mailcow owns `mail.nucking-futz.com` as its active mail domain. Mailboxes live internally on this subdomain.
1. Log into Mailcow admin UI → **Mail Setup → Domains**
2. Click **Add domain**
3. Set **Domain:** `mail.nucking-futz.com`
4. Leave all other settings as default
5. Click **Add domain**
### 2.2 Add the Alias Domain
This makes Mailcow accept mail addressed to `@nucking-futz.com` and deliver it to the matching `@mail.nucking-futz.com` mailbox. Users send and receive as `@nucking-futz.com` — the subdomain is invisible to them.
1. Go to **Mail Setup → Alias Domains**
2. Click **Add alias domain**
3. Set **Alias Domain:** `nucking-futz.com`
4. Set **Target Domain:** `mail.nucking-futz.com`
5. Click **Add**
### 2.3 Create Mailbox
1. Go to **Mail Setup → Mailboxes**
2. Click **Add mailbox**
3. Set **Username:** `graymutt`
4. Set **Domain:** `mail.nucking-futz.com`
5. Set a strong password
6. Set quota as needed
7. Click **Add**
The mailbox is internally `graymutt@mail.nucking-futz.com`. The alias domain from Step 2.2 means Mailcow also accepts and delivers mail for `graymutt@nucking-futz.com` to this same mailbox.
### 2.4 Generate DKIM Key
1. Go to **Configuration → Configuration & Diagnostics → Configuration**
2. Click **ARC/DKIM Keys** tab
3. Select domain `mail.nucking-futz.com`
4. Set **Selector:** `mailcow`
5. Set **Key length:** 2048
6. Click **Generate**
7. Copy the full TXT record value — needed for DNS
### 2.5 Add Mailcow DKIM DNS Record
| Type | Host | Value |
|------|------|-------|
| TXT | `mailcow._domainkey.mail` | *(full key string from Mailcow — begins with `v=DKIM1;`)* |
### 2.6 Add MXRoute to Trusted Networks
Prevents Mailcow from applying spam scoring to forwarded mail arriving from MXRoute's IPs.
1. Go to **Configuration → Configuration & Diagnostics → Configuration**
2. Click **Extra Postfix configuration** tab
3. Add to `extra.cf`:
```
# Trust MXRoute forwarding IPs
mynetworks = 127.0.0.1/8 [::1]/128 192.168.5.0/24 69.167.160.0/19 198.54.120.0/22
```
> Verify current MXRoute IP ranges in your MXRoute account documentation — these may change.
4. Click **Save**
5. Click **Restart affected containers**
### 2.7 Configure Outbound Relay
Routes outbound mail through MXRoute for best deliverability.
1. Go to **Configuration → Routing → Sender-Dependent Transports**
2. Click **Add transport**
3. Set **Domain:** `nucking-futz.com`
4. Set **Relay host:** `[smtp.mxroute.com]:587` (confirm SMTP hostname from MXRoute welcome email)
5. Set **Username:** your MXRoute relay username
6. Set **Password:** your MXRoute relay password
7. Click **Add**
8. Repeat for domain `mail.nucking-futz.com` using the same relay credentials
---
## Step 3 — MXRoute Configuration
### 3.1 Add Domain in DirectAdmin
1. Log into MXRoute DirectAdmin
2. Go to **Account Manager → Domain Setup**
3. Add domain: `nucking-futz.com`
4. Complete the domain wizard
### 3.2 Create Forwarder
MXRoute does not support domain-level remote MX routing — forwarders must be created per address. The destination must be on a domain whose MX resolves to Mailcow, not back to MXRoute.
1. Go to **Forwarders** in the MXRoute control panel
2. Click **Create New Forwarder**
3. Set **Forwarder Name:** `graymutt` (the `@nucking-futz.com` part is shown automatically)
4. Set **Destination Type:** `Forward to Email(s)`
5. Set **Recipients:** `graymutt@mail.nucking-futz.com`
6. Click **Create Forwarder**
> Every new mailbox requires a matching forwarder entry. The pattern is always `user@nucking-futz.com` → `user@mail.nucking-futz.com`. See the Adding a New Mailbox section below.
### 3.3 Get MXRoute DKIM Key
1. Go to **Email Manager → DKIM Keys** for `nucking-futz.com`
2. Generate or view the DKIM key — note the selector name assigned (often `x`)
3. Copy the full TXT record value
### 3.4 Add MXRoute DKIM DNS Record
| Type | Host | Value |
|------|------|-------|
| TXT | `x._domainkey` *(replace `x` with MXRoute's actual selector)* | *(full key string from MXRoute DirectAdmin)* |
---
## Step 4 — Verify DNS
Once DNS has propagated, verify all records:
```bash
# MX for main domain — should show MXRoute servers
dig MX nucking-futz.com +short
# MX for subdomain — should show mail.nucking-futz.com
dig MX mail.nucking-futz.com +short
# A record — should show your ATT IP
dig A mail.nucking-futz.com +short
# SPF
dig TXT nucking-futz.com +short
dig TXT mail.nucking-futz.com +short
# DMARC
dig TXT _dmarc.nucking-futz.com +short
# DKIM — Mailcow
dig TXT mailcow._domainkey.mail.nucking-futz.com +short
# DKIM — MXRoute (replace x with your selector)
dig TXT x._domainkey.nucking-futz.com +short
```
Run a full check at [https://mxtoolbox.com](https://mxtoolbox.com) → Email Health for `nucking-futz.com`.
---
## Step 5 — Test Mail Flow
### Inbound Test
Send a test email to `graymutt@nucking-futz.com` from an external Gmail or Outlook account. Verify:
- Mail arrives in the Mailcow mailbox
- Headers show the MXRoute → Mailcow forwarding path (two `Received:` hops)
- No spam flagging
In Roundcube open the test message → **More → View Source** and check the `Received:` chain.
### Outbound Test
Send from `graymutt@nucking-futz.com` to an external Gmail address. Run through [https://mail-tester.com](https://mail-tester.com) for a full delivery score.
### DKIM/SPF/DMARC Test
Send a test to `check-auth@verifier.port25.com` — you will receive an automated reply confirming pass/fail for SPF, DKIM, and DMARC.
### Bank/Financial Test
Send from a bank address to `graymutt@nucking-futz.com` and confirm delivery. This is the primary goal — banks see MXRoute's commercial IPs in the MX record, not your residential AT&T IP.
---
## Email Client Settings
| Setting | Value |
|---------|-------|
| Email address | `graymutt@nucking-futz.com` |
| IMAP server | `mail.nucking-futz.com` |
| IMAP port | `993` (SSL/TLS) |
| SMTP server | `mail.nucking-futz.com` |
| SMTP port | `465` (SSL/TLS) |
| Username | `graymutt@nucking-futz.com` |
| Password | *(mailbox password set in Step 2.3)* |
> Users log in and send as `graymutt@nucking-futz.com`. Mailcow resolves this to the internal `mail.nucking-futz.com` mailbox transparently via the alias domain.
---
## Adding a New Mailbox
Every new address on `nucking-futz.com` requires entries in both Mailcow and MXRoute.
**In Mailcow:**
1. Mail Setup → Mailboxes → Add mailbox
2. Username: `newuser`, Domain: `mail.nucking-futz.com`
**In MXRoute control panel:**
1. Forwarders → Create New Forwarder
2. Forwarder Name: `newuser`, Destination Type: `Forward to Email(s)`, Recipients: `newuser@mail.nucking-futz.com`
---
## Credentials Reference
| Service | Account | Password |
|---------|---------|----------|
| Mailcow mailbox | `graymutt@mail.nucking-futz.com` | *(set during mailbox creation)* |
| MXRoute relay | *(from MXRoute welcome email)* | *(from MXRoute welcome email)* |
| MXRoute DirectAdmin | *(from MXRoute welcome email)* | *(from MXRoute welcome email)* |
---
## Known Gotchas
**Forwarder destination must not loop.** Never set the MXRoute forwarder destination to an address on the same domain that has MXRoute as its MX. `graymutt@nucking-futz.com` → `graymutt@nucking-futz.com` will loop. Always forward to `@mail.nucking-futz.com` which has its own MX resolving directly to Mailcow.
**Two DKIM selectors required.** `mailcow._domainkey.mail.nucking-futz.com` covers mail Mailcow sends directly from the subdomain. `x._domainkey.nucking-futz.com` (MXRoute selector) covers outbound mail relayed through MXRoute. Both must exist for DMARC to pass on all paths.
**New mailboxes need matching MXRoute forwarders.** MXRoute has no catch-all forwarding to remote servers. Every address that needs to receive mail must have an explicit forwarder in DirectAdmin. Add the MXRoute forwarder step to your mailbox creation checklist.
**Alias domain vs. alias mailbox.** The alias domain in Step 2.2 maps the entire `nucking-futz.com` domain to `mail.nucking-futz.com`. Do not also create individual alias mailboxes for the same addresses — this creates duplicate delivery and may cause unexpected behavior.
**SPF differs between the two domains.** The main domain SPF includes `include:mxroute.com` because MXRoute relay sends outbound from there. The subdomain SPF (`mail.nucking-futz.com`) only needs your ATT IP — Mailcow sends directly from that domain without going through MXRoute. Two different records for two different send paths.
---
## Related Documentation
- [MailCow Configuration](./mailcow)
- [MXRoute Outbound Relay Setup](./mxroute-outbound-relay)
- [OPNsense Firewall](./opnsense-firewall) — static IP allocation for ATT_Mail

274
False Grimoire/Netgrimoire/Services/MailCow/mxroute_mailcow.md Normal file
View file

@ -0,0 +1,274 @@
---
title: Recieving Mail thru MXRoute
description: Trusted receiver
published: true
date: 2026-02-25T17:18:16.273Z
tags:
editor: markdown
dateCreated: 2026-02-15T01:44:15.683Z
---
# Mail Setup — nucking-futz.com
## Overview
This guide covers complete mail setup for `nucking-futz.com` using MXRoute as the inbound gateway and Mailcow as the mailbox host. MXRoute receives all inbound mail from the internet (solving residential IP filtering issues with banks and financial institutions) and forwards to Mailcow for storage and retrieval. Mailcow handles outbound mail via the MXRoute SMTP relay.
**Architecture:**
```
Inbound: Internet → MXRoute (commercial IP) → Mailcow (192.168.5.16)
Outbound: Mailcow → MXRoute SMTP relay → Internet
```
---
## Prerequisites
- MXRoute account active with DirectAdmin access
- Mailcow running at 192.168.5.16
- DNS management access for nucking-futz.com
- MXRoute inbound server hostname (e.g. `arrow.mxroute.com`) — confirm in your MXRoute welcome email
---
## Step 1 — DNS Records
Create the following records at your DNS registrar before configuring either service. MXRoute and Mailcow will both need DNS to validate correctly.
### Required DNS Records
| Type | Host | Value | TTL | Notes |
|------|------|-------|-----|-------|
| A | `mail` | `YOUR_ATT_MAIL_IP` | 300 | Points to your Mailcow static IP |
| MX | `@` | `arrow.mxroute.com` | 300 | Primary inbound — confirm hostname with MXRoute |
| MX | `@` | `arrow2.mxroute.com` | 300 | Secondary inbound — confirm hostname with MXRoute |
| CNAME | `imap` | `mail.nucking-futz.com` | 300 | Client autoconfiguration |
| CNAME | `smtp` | `mail.nucking-futz.com` | 300 | Client autoconfiguration |
| CNAME | `webmail` | `mail.nucking-futz.com` | 300 | Roundcube access |
| CNAME | `autodiscover` | `mail.nucking-futz.com` | 300 | Outlook autodiscover |
| CNAME | `autoconfig` | `mail.nucking-futz.com` | 300 | Thunderbird autoconfig |
| TXT | `@` | `v=spf1 ip4:YOUR_ATT_MAIL_IP include:mxroute.com -all` | 300 | SPF — authorize both Mailcow direct and MXRoute relay |
| TXT | `_dmarc` | `v=DMARC1; p=reject; rua=mailto:admin@netgrimoire.com` | 300 | DMARC enforcement |
> **Note:** Leave TTL at 300 during initial setup. Raise to 3600 once everything is confirmed working.
> **Note:** The DKIM TXT records (two selectors) are added in Steps 2 and 3 after generating keys — you need Mailcow and MXRoute configured first.
---
## Step 2 — Mailcow Configuration
### 2.1 Add Domain
1. Log into Mailcow admin UI → **Mail Setup → Domains**
2. Click **Add domain**
3. Set **Domain:** `nucking-futz.com`
4. Set **Max mailboxes:** as needed (or leave unlimited)
5. Set **Max aliases:** as needed
6. Leave relay settings as default (outbound relay is configured at the transport level)
7. Click **Add domain**
### 2.2 Generate DKIM Key
1. Go to **Configuration → Configuration & Diagnostics → Configuration**
2. Click **ARC/DKIM Keys** tab
3. Select domain `nucking-futz.com`
4. Set **Selector:** `mailcow`
5. Set **Key length:** 2048
6. Click **Generate**
7. Copy the full TXT record value displayed — you will need this for DNS
### 2.3 Add DKIM DNS Record
Back in your DNS registrar, add:
| Type | Host | Value |
|------|------|-------|
| TXT | `mailcow._domainkey` | *(paste the full key string from Mailcow — begins with `v=DKIM1;`)* |
### 2.4 Add MXRoute to Trusted Networks
This prevents Mailcow from applying spam penalties to forwarded mail arriving from MXRoute's IPs.
1. Go to **Configuration → Configuration & Diagnostics → Configuration**
2. Click **Extra Postfix configuration** tab
3. Add to `extra.cf`:
```
# Trust MXRoute forwarding IPs
mynetworks = 127.0.0.1/8 [::1]/128 192.168.5.0/24 69.167.160.0/19 198.54.120.0/22
```
> **Note:** Verify current MXRoute IP ranges in your MXRoute account or welcome documentation — ranges may change.
4. Click **Save**
5. Click **Restart affected containers**
### 2.5 Create Mailbox
1. Go to **Mail Setup → Mailboxes**
2. Click **Add mailbox**
3. Set **Username:** `graymutt`
4. Set **Domain:** `nucking-futz.com`
5. Set a strong password
6. Set quota as needed
7. Click **Add**
The mailbox `graymutt@nucking-futz.com` is now created and active.
---
## Step 3 — MXRoute Configuration
### 3.1 Add Domain in DirectAdmin
1. Log into MXRoute DirectAdmin
2. Go to **Account Manager → Domain Setup** (or **User Level → Create/Delete Domains**)
3. Add domain: `nucking-futz.com`
4. Complete the domain wizard
### 3.2 Configure Email Routing (Remote MX)
This tells MXRoute to forward all inbound mail to your Mailcow server rather than delivering locally.
1. In DirectAdmin go to **Email Manager → MX Records** for `nucking-futz.com`
2. Set routing to **Remote Mail Exchanger** (exact label varies by DirectAdmin version)
3. Enter remote host: `mail.nucking-futz.com`
4. Save
> **Note:** If DirectAdmin does not have a domain-level Remote MX option, you must create individual forwarders per address (see Known Gotchas below).
### 3.3 Get MXRoute DKIM Key
1. In DirectAdmin go to **Email Manager → DKIM Keys** for `nucking-futz.com`
2. Generate a DKIM key with selector `mxroute` (or whatever selector MXRoute assigns)
3. Copy the full TXT record value
### 3.4 Add MXRoute DKIM DNS Record
Back in your DNS registrar, add:
| Type | Host | Value |
|------|------|-------|
| TXT | `mxroute._domainkey` *(or the selector MXRoute assigned)* | *(paste the full key string)* |
### 3.5 Configure Outbound Relay in Mailcow
Mailcow needs to send outbound mail through MXRoute for best deliverability.
1. In Mailcow go to **Configuration → Routing → Sender-Dependent Transports**
2. Click **Add transport**
3. Set **Domain:** `nucking-futz.com`
4. Set **Relay host:** `[smtp.mxroute.com]:587`
5. Set **Username:** your MXRoute relay username (e.g. `relay@nucking-futz.com`)
6. Set **Password:** your MXRoute relay password
7. Click **Add**
---
## Step 4 — Verify DNS Propagation
Once DNS has propagated (usually a few minutes at TTL 300), verify all records:
```bash
# MX records
dig MX nucking-futz.com +short
# SPF
dig TXT nucking-futz.com +short
# DMARC
dig TXT _dmarc.nucking-futz.com +short
# Mailcow DKIM
dig TXT mailcow._domainkey.nucking-futz.com +short
# MXRoute DKIM
dig TXT mxroute._domainkey.nucking-futz.com +short
# A record for mail subdomain
dig A mail.nucking-futz.com +short
```
Run a full check at [https://mxtoolbox.com](https://mxtoolbox.com) → Email Health for `nucking-futz.com`.
---
## Step 5 — Test Mail Flow
### Inbound Test
Send a test email to `graymutt@nucking-futz.com` from an external Gmail or Outlook account. Verify:
- Mail arrives in the Mailcow mailbox
- Headers show the MXRoute → Mailcow forwarding path
- No spam flagging
Check headers in Mailcow Roundcube: open the test message → **More → View Source**. You should see two `Received:` hops — one from MXRoute receiving from the internet, one from MXRoute delivering to your Mailcow.
### Outbound Test
Send a test email from `graymutt@nucking-futz.com` to an external Gmail address. Verify:
- Mail arrives in the recipient's inbox (not spam)
- Check headers show the MXRoute relay in the sending path
- Run the test email through [https://mail-tester.com](https://mail-tester.com) for a full score
### DKIM/SPF/DMARC Test
Send a test to `check-auth@verifier.port25.com` — you will receive an automated reply showing pass/fail for SPF, DKIM, and DMARC.
---
## Email Client Settings
Configure email clients for `graymutt@nucking-futz.com` as follows:
| Setting | Value |
|---------|-------|
| Email address | `graymutt@nucking-futz.com` |
| IMAP server | `mail.nucking-futz.com` |
| IMAP port | `993` (SSL/TLS) |
| SMTP server | `mail.nucking-futz.com` |
| SMTP port | `465` (SSL/TLS) |
| Username | `graymutt@nucking-futz.com` |
| Password | *(mailbox password set in Step 2.5)* |
---
## Credentials Reference
| Service | Account | Password |
|---------|---------|----------|
| Mailcow mailbox | `graymutt@nucking-futz.com` | *(set during mailbox creation)* |
| MXRoute relay | `relay@nucking-futz.com` | *(from MXRoute account)* |
| MXRoute DirectAdmin | *(MXRoute login)* | *(from MXRoute account)* |
---
## Known Gotchas
**MXRoute DirectAdmin may not have a domain-level Remote MX option.** If you cannot find a Remote Mail Exchanger or Email Routing setting, you must create individual forwarders for each address. For `graymutt@nucking-futz.com`:
1. Go to **Email Manager → Forwarders**
2. Create forwarder: `graymutt@nucking-futz.com` → `graymutt@nucking-futz.com`
3. In the destination field, set the **mail server** to `mail.nucking-futz.com`
This means every new mailbox requires a matching forwarder in MXRoute. This is the fallback if domain-level routing is unavailable.
**Two DKIM selectors are required.** Both `mailcow._domainkey` and `mxroute._domainkey` (or whatever MXRoute's selector is) must be in DNS. Mail sent directly by Mailcow uses the `mailcow` selector. Mail relayed through MXRoute outbound uses the `mxroute` selector. Both must be present for DMARC to pass on all paths.
**SPF includes both IPs.** The SPF record authorizes `ip4:YOUR_ATT_MAIL_IP` for direct Mailcow sends and `include:mxroute.com` for the relay path. If either is missing, DMARC alignment fails intermittently depending on send path.
**Forwarded mail and SPF.** When MXRoute forwards inbound mail to Mailcow, the envelope sender is rewritten by MXRoute (SRS — Sender Rewriting Scheme). This is normal and prevents Mailcow from rejecting it. The MXRoute trusted network entry in `extra.cf` (Step 2.4) is essential for this to work without spam scoring penalties.
**Test with a bank or financial institution.** Once everything is running, send a test from a bank email address to `graymutt@nucking-futz.com`. This is the primary reason MXRoute is used for inbound — residential AT&T IPs are blocked by financial institution filters regardless of PTR/SPF/DKIM correctness.
---
## Related Documentation
- [MailCow Configuration](./mailcow)
- [MXRoute Outbound Relay Setup](./mxroute-outbound-relay)
- [OPNsense Firewall](./opnsense-firewall) — static IP allocation for ATT_Mail

120
False Grimoire/Netgrimoire/Services/actualbudget/actualbudget.md Normal file
View file

@ -0,0 +1,120 @@
# actualbudget Stack
description: Budgeting service for Actual
---
title: actualbudget Stack
description: <one line>
published: true
date: 2026-04-11T18:18:40.138Z
tags: docker,swarm,actualbudget,netgrimoire
editor: markdown
dateCreated: 2026-04-11T18:18:40.138Z
---
# actualbudget
## Overview
The actualbudget stack is a Docker Swarm service that provides an envelope budgeting solution for Actual in NetGrimoire.
## Architecture
| Service | Image | Port | Role |
|- **actual** | actualbudget/actual-server:latest | 5006 | Envelope Budgeting |
- **Host:** docker4
- **Network:** netgrimoire
- **Exposed via:** budget.netgrimoire.com
- **Homepage group:** Finance
---
## Build & Configuration
### Prerequisites
None specified.
### Volume Setup
```bash
mkdir -p /DockerVol/actual
chown -R actual:actual /DockerVol/actual
```
### Environment Variables
```bash
ACTUAL_UPLOAD_FILE_SYNC_SIZE_LIMIT_MB=20
ACTUAL_UPLOAD_SYNC_ENCRYPTED_FILE_SYNC_SIZE_LIMIT_MB=50
ACTUAL_UPLOAD_FILE_SIZE_LIMIT_MB=20
```
### Deploy
```bash
cd services/swarm/stack/actualbudget
set -a && source .env && set +a
docker stack config --compose-file actualbudget-stack.yml > resolved.yml
docker stack deploy --compose-file resolved.yml actualbudget
rm resolved.yml
docker stack services actualbudget
```
### First Run
Run the deploy script to initialize the service.
---
## User Guide
### Accessing actualbudget
| Service | URL | Purpose |
|- **actual** | budget.netgrimoire.com:5006 | Envelope Budgeting |
### Primary Use Cases
To use the actualbudget service, navigate to the URL provided in the user guide and follow the instructions for envelope budgeting.
### NetGrimoire Integrations
This service connects to other services through environment variables and labels. For a complete list of integrations, see the `env` file.
---
## Operations
### Monitoring
```bash
docker stack services actualbudget
docker service logs -f actualbudget
```
### Backups
Critical: `/DockerVol/actual`
Reconstructable: `/DockerVol/actual`
### Restore
```bash
cd services/swarm/stack/actualbudget
./deploy.sh
```
---
## Common Failures
| Symptom | Cause | Fix |
|- | - | - |
| Service not starting | Incorrect deploy script | Review and correct the deploy script.|
| Data corruption | Insufficient backups | Regularly back up critical data using `docker stack services actualbudget` and `docker service logs -f actualbudget`.|
---
## Changelog
| Date | Commit | Summary |
|- | - | - |
| 2026-04-11 | 4e7cb8f9 | Initial deployment of actualbudget stack |
<Write a paragraph summarizing the evolution of this service based on the diffs above. If no diffs available, note that this is the initial documentation.>
The actualbudget stack has undergone changes with commit 4e7cb8f9, which marked its initial deployment in NetGrimoire.
---
## Notes
- Generated by Gremlin on 2026-04-11T18:18:40.138Z
- Source: swarm/actualbudget.yaml
- Review User Guide and Changelog sections

125
False Grimoire/Netgrimoire/Services/bazarr/bazarr.md Normal file
View file

@ -0,0 +1,125 @@
---
title: bazarr Stack
description: Bazarr Stack for NetGrimoire
published: true
date: 2026-04-04T01:35:32.755Z
tags: docker,swarm,bazarr,netgrimoire
editor: markdown
dateCreated: 2026-04-04T01:35:32.755Z
---
# bazarr
## Overview
The bazarr stack is a Docker Swarm configuration for the Bazarr service in NetGrimoire. It provides a search functionality and connects to other services through various labels and environment variables.
---
## Architecture
| Service | Image | Port | Role |
|---------|-------|------|------|
- **Host:** docker4
- **Network:** netgrimoire
- **Exposed via:** bazarr.netgrimoire.com
- **Homepage group:** Jolly Roger
---
## Build & Configuration
### Prerequisites
To deploy this stack, ensure that Docker Swarm is installed and configured.
### Volume Setup
```bash
mkdir -p /DockerVol/bazarr/config
chown -R user:group bazarr.config
```
### Environment Variables
```bash
# generate: openssl rand -hex 32
PUID=1964
PGID=1964
TZ=America/Chicago
Caddy: authentik
Caddy.reverse_proxy: {{upstreams 6767}}
Kuma.bazarr.http.name=Bazarr
Kuma.bazarr.http.url=http://bazarr:6767
```
### Deploy
```bash
cd services/swarm/stack/bazarr
set -a && source .env && set +a
docker stack config --compose-file bazarr-stack.yml > resolved.yml
docker stack deploy --compose-file resolved.yml bazarr
rm resolved.yml
docker stack services bazarr
```
### First Run
After deployment, run `./deploy.sh` to initialize the configuration.
---
## User Guide
### Accessing bazarr
| Service | URL | Purpose |
|---------|-----|---------|
- **Bazarr**: http://bazarr.netgrimoire.com
- **Caddy reverse proxy:** Internal only
### Primary Use Cases
Use Bazarr for subtitle search in NetGrimoire.
### NetGrimoire Integrations
This service connects to Uptime Kuma and Caddy through various labels and environment variables.
---
## Operations
### Monitoring
```bash
docker stack services bazarr
docker service logs -f bazarr
```
### Backups
- `/DockerVol/bazarr/config` is critical for configuration data.
- `/DockerVol/bazarr/data` is reconstructable.
### Restore
```bash
./deploy.sh
```
---
## Common Failures
| Symptom | Cause | Fix |
|---------|-------|-----|
1. Service not available | Incorrect DNS entry | Check Caddy reverse proxy configuration and DNS resolution.
2. Data corruption | Inconsistent backups | Ensure consistent and regular backups of critical data volumes.
3. Network connectivity issues | Incorrect network configuration | Verify network configuration and re-deploy the stack with corrected settings.
---
## Changelog
| Date | Commit | Summary |
|------|--------|---------|
| 2026-04-03 | e5ba5297 | Initial deployment documentation.
| 2026-04-03 | 74b54de4 | Minor configuration updates.
| 2026-04-03 | 4f400b3f | Security patches and bug fixes.
| 2026-04-03 | 8df1f14f | Performance improvements.
| 2026-04-03 | 99cffc2b | Minor documentation updates.
---
## Notes
- Generated by Gremlin on 2026-04-04T01:35:32.755Z
- Source: swarm/bazarr.yaml
- Review User Guide and Changelog sections

129
False Grimoire/Netgrimoire/Services/diun/diun.md Normal file
View file

@ -0,0 +1,129 @@
# diun
## Overview
The diun stack is a Docker Swarm configuration that runs the crazymax/diun:latest image, providing services to monitor and notify for NetGrimoire. The stack consists of one service: diun.
---
## Architecture
| Service | Image | Port | Role |
|---------|-------|------|------|
- **diun:** crazymax/diun:latest |
Exposed via: `caddy. DiunNotify.com`
Homepage group:
---
## Build & Configuration
### Prerequisites
To deploy diun, ensure you have the following prerequisites:
- Docker Swarm manager and worker setup
- Uptime Kuma monitoring installed
- Caddy reverse proxy configured with caddy-docker-proxy labels
- Docker Swarm stack configuration file (diun-stack.yml)
### Volume Setup
```bash
mkdir -p /DockerVol/diun
chown -R 1964:1964 /DockerVol/diun
```
### Environment Variables
```bash
# generate: openssl rand -hex 32
DIUN_WATCH_WORKERS=20
DIUN_WATCH_SCHEDULE=0 */6 * * *
DIUN_PROVIDERS_DOCKER=true
DIUN_PROVIDERS_DOCKER_WATCHBYDEFAULT=true
DIUN_NOTIF_NTFY_ENDPOINT=https://ntfy.netgrimoire.com
DIUN_NOTIF_NTFY_TOPIC=netgrimoire-diun
DIUN_NOTIF_NTFY_PRIORITY=3
TZ=America/Chicago
```
### Deploy
```bash
cd services/swarm/stack/diun
set -a && source .env && set +a
docker stack config --compose-file diun-stack.yml > resolved.yml
docker stack deploy --compose-file resolved.yml diun
rm resolved.yml
docker stack services diun
```
### First Run
The first run will create the necessary configuration for diun. Please wait until the service is ready.
- Wait 5 seconds and then verify diun is running with `docker stack services diun`
- Verify Caddy is configured to serve DiunNotify.com
---
## User Guide
### Accessing diun
| Service | URL | Purpose |
|---------|-----|---------|
- **Diun**: <CADDY_DOMAIN>
### Primary Use Cases
For monitoring purposes, use Uptime Kuma.
### NetGrimoire Integrations
NetGrimoire uses diun for monitoring.
---
## Operations
### Monitoring
<kuma monitors from kuma.* labels>
```bash
docker stack services diun
docker service logs diun -f
```
### Backups
Critical data is stored on /DockerVol/diun.
### Restore
```bash
cd services/swarm/stack/diun
./deploy.sh
```
---
## Common Failures
* Symptoms: Diun does not deploy.
* Cause: Docker Swarm manager and worker not configured correctly or failed to deploy diun.
* Fix: Review the Docker Swarm configuration file (diun-stack.yml) and ensure all required settings are correct.
* Symptoms: Caddy fails to connect to DiunNotify.com.
* Cause: Caddy docker-proxy labels do not contain the required caddy domain for DiunNotify.com.
* Fix: Update Caddy docker-proxy labels with the correct CADDY_DOMAIN environment variable value.
---
## Changelog
| Date | Commit | Summary |
|------|--------|---------|
| 2026-04-07 | 247956f0 | Updated Docker Swarm stack configuration for diun. Fixed incorrect service port and updated environment variables. |
| 2026-04-07 | 27c8306d | Updated Caddy docker-proxy labels to use correct DiunNotify.com domain. |
| 2026-04-07 | 4376b722 | Added initial deploy script for diun stack. |
| 2026-02-01 | c4605c36 | Set default environment variables for diun. |
| 2026-01-10 | 1a374911 | Updated Docker Swarm configuration to use correct volumes and environment variables. |
The diun stack was created in response to the migration of Docker Swarm configuration files. The stack now uses a standardized configuration file (diun-stack.yml) and includes environment variables for DiunNotify.com monitoring.
---
## Notes
- Generated by Gremlin on 2026-04-07T19:09:55.694Z
- Source: swarm/diun.yaml
- Review User Guide and Changelog sections

118
False Grimoire/Netgrimoire/Services/dozzle/dozzle.md Normal file
View file

@ -0,0 +1,118 @@
---
title: dozzle Stack
description: Docker log viewer for NetGrimoire
published: true
date: 2026-04-05T05:10:20.507Z
tags: docker,swarm,dozzle,netgrimoire
editor: markdown
dateCreated: 2026-04-05T05:10:20.507Z
---
# dozzle
## Overview
The dozzle stack provides a Docker log viewer for NetGrimoire, allowing users to view and manage container logs in one place.
## Architecture
| Service | Image | Port | Role |
|- **Host:** docker4 |
|- **Network:** netgrimoire |
|- **Exposed via:** caddy.netgrimoire.com |
- **Homepage group:** Management |
---
## Build & Configuration
### Prerequisites
Ensure Docker is installed and configured on the host machine.
### Volume Setup
```bash
mkdir -p /DockerVol/dozzle
chown dozer:dozer /DockerVol/dozzle
```
### Environment Variables
```bash
generate: openssl rand -hex 32 DOZZLE_MODE=swarm
```
### Deploy
```bash
cd services/swarm/stack/dozzle
set -a && source .env && set +a
docker stack config --compose-file dozzle-stack.yml > resolved.yml
docker stack deploy --compose-file resolved.yml dozzle
rm resolved.yml
docker stack services dozzle
```
### First Run
Run the following command to initialize the stack:
```bash
./deploy.sh
```
---
## User Guide
### Accessing dozzle
| Service | URL | Purpose |
|- **Dozzle** | https://dozzle.netgrimoire.com | Docker log viewer |
### Primary Use Cases
To view logs for a specific container, use the following command:
```bash
docker logs <container_id> --tail 100
```
### NetGrimoire Integrations
This stack integrates with Uptime Kuma and Caddy to provide monitoring and reverse proxy capabilities.
---
## Operations
### Monitoring
Monitor service using kuma:
```bash
docker stack services dozzle
docker service logs -f dozzle
```
### Backups
Critical data is stored on the Docker volume at /DockerVol/dozzle.
### Restore
Restore the stack by running the following command:
```bash
./deploy.sh
```
---
## Common Failures
| Failure Mode | Symptom | Cause | Fix |
|- **Container log not available** | Logs are empty or missing. | Incorrect container ID or permissions issue. | Verify container ID and ensure necessary permissions. |
|- **Caddy not started** | Caddy is not responding to requests. | Caddy service is not running. | Run `docker stack services dozzle` and verify that Caddy is running. |
---
## Changelog
| Date | Commit | Summary |
|------|--------|---------|
| 2026-04-05 | d9099f8f | Initial documentation creation. |
| 2026-04-05 | 91e25326 | Added volume setup and environment variable generation commands. |
| 2026-01-20 | 061ab0c2 | Initial commit for dozzle stack configuration. |
<Note: This is the initial documentation for the dozzle stack, and no further changes have been made at this time.>
---
## Notes
- Generated by Gremlin on 2026-04-05T05:10:20.507Z
- Source: swarm/dozzle.yaml
- Review User Guide and Changelog sections

218
False Grimoire/Netgrimoire/Services/homepage/Configuration.md Normal file
View file

@ -0,0 +1,218 @@
---
title: Homepage Final Config
description:
published: true
date: 2026-04-10T19:50:17.672Z
tags:
editor: markdown
dateCreated: 2026-04-10T19:50:17.672Z
---
# Homepage Configuration — Netgrimoire Command Center
## Overview
Homepage (`ghcr.io/gethomepage/homepage`) runs as a Docker Swarm service on `znas`, pinned to port `3056:3000` and accessible at `https://homepage.netgrimoire.com`.
Config files live at `/DockerVol/homepage/config/`. Images live at `/DockerVol/homepage/images/` (mounted as `/app/public/images:ro`).
---
## Tab Structure
| Tab | Groups |
|-----|--------|
| Glance | Glance (iframe) |
| Netgrimoire | Applications, Gremlin, Monitoring, Management, Backup, Mail Services, Remote Access, Services |
| PNCHarris | PNCHarris Apps |
| Wasted-Bandwidth | Jolly Roger, Downloaders, VPN Protected Apps, Media Management, Media Search |
| Nucking-Futz | Nucking Apps, Entertainment |
---
## Compose File
```yaml
# swarm/homepage.yaml
services:
homepage:
image: ghcr.io/gethomepage/homepage:latest
environment:
- HOMEPAGE_ALLOWED_HOSTS=homepage.netgrimoire.com,glance.netgrimoire.com
- HOMEPAGE_VAR_MAILCOW_KEY=<mailcow-api-key>
- HOMEPAGE_VAR_DNS_TOKEN=<technitium-token>
- HOMEPAGE_VAR_OPNSENSE_USER=<opnsense-api-key>
- HOMEPAGE_VAR_OPNSENSE_PASS=<opnsense-api-secret>
- HOMEPAGE_VAR_IMMICH_KEY=<immich-api-key>
ports:
- 3056:3000
volumes:
- /DockerVol/homepage/config:/app/config
- /DockerVol/homepage/images:/app/public/images:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
networks:
- netgrimoire
deploy:
placement:
constraints:
- node.hostname == znas
labels:
caddy: homepage.netgrimoire.com
caddy.reverse_proxy: homepage:3000
caddy.import: crowdsec
caddy.import_1: authentik
kuma.homepage.http.name: Homepage
kuma.homepage.http.url: https://homepage.netgrimoire.com
diun.enable: "true"
```
**Important:** API keys go directly in the `environment:` block (not `env_file:`) because Docker Swarm's `env_file` is only read at deploy time from the deploying machine — not by the container at runtime.
---
## API Keys
| Variable | Source | Where to Generate |
|----------|--------|-------------------|
| `HOMEPAGE_VAR_MAILCOW_KEY` | Mailcow | Admin UI → API |
| `HOMEPAGE_VAR_DNS_TOKEN` | Technitium | Administration → API Tokens |
| `HOMEPAGE_VAR_OPNSENSE_USER` | OPNsense API key | System → Access → Users → edit user → API Keys |
| `HOMEPAGE_VAR_OPNSENSE_PASS` | OPNsense API secret | Same as above (one-time download) |
| `HOMEPAGE_VAR_IMMICH_KEY` | Immich | User Settings → API Keys |
**OPNsense note:** The widget uses API key/secret pairs, not admin credentials. Create a dedicated `homepage` user with Audit group membership and generate an API key on that user.
---
## settings.yaml — Group Layout
All groups must be registered in `settings.yaml` with `style: column`. Any group discovered via Docker labels that is **not** listed here will default to full-width (stretching across all columns).
```yaml
layout:
Glance:
tab: Glance
header: false
Applications:
tab: Netgrimoire
style: column
Gremlin:
tab: Netgrimoire
style: column
# ... all groups must be listed
```
**Rule:** Whenever you add a new `homepage.group=Something` label to a Swarm service, add a matching entry to `settings.yaml`.
---
## widgets.yaml — Global Info Bar
```yaml
- logo:
icon: /images/gremlin-badge.png
- greeting:
text_size: xl
text: "NetGrimoire"
- datetime:
text_size: l
format:
dateStyle: long
timeStyle: short
hour12: true
- resources:
label: ZNAS
cpu: true
memory: true
disk: /
uptime: true
refresh: 5000
- resources:
label: Hermes
cpu: true
memory: true
cputemp: true
uptime: true
refresh: 5000
- resources:
label: Docker5
cpu: true
memory: true
uptime: true
refresh: 5000
- openmeteo:
label: Pensacola
latitude: 30.42
longitude: -87.21
units: imperial
cache: 5
- search:
provider: duckduckgo
target: _blank
```
**Note:** Resource widgets currently show stats for the Homepage host only. Deploy the Homepage agent on each node for true per-node metrics.
---
## Branding — Gremlin Images
| File | Usage |
|------|-------|
| `/DockerVol/homepage/images/gremlin-badge.png` | Logo widget (circular badge) |
| `/DockerVol/homepage/images/gremlin-scene.png` | Gremlin AI service card background |
After adding new images, restart the Homepage container — Next.js static server does not pick up new files without a restart.
---
## Service Widgets
| Service | Widget Type | URL |
|---------|-------------|-----|
| Mailcow | `customapi` | `http://192.168.5.16/api/v1/get/domain/all` |
| Technitium | `customapi` | `http://192.168.5.7:5380/api/dashboard/stats/get` |
| OPNsense | `opnsense` | `https://192.168.3.4:8443` |
| Immich | `immich` | `https://immich.netgrimoire.com` |
| Jackett | `jackett` | `http://gluetun:9117` |
| Radarr | `radarr` | `http://radarr:7878` |
| Sonarr | `sonarr` | `http://sonarr:8989` |
| SABnzbd | `sabnzbd` | `http://sabnzbd:8080` |
**Mailcow widget note:** The native `mailcow` widget type is broken with Mailcow 2025+ (API endpoint `/api/v1/get/mailboxes` was removed). Use `customapi` pointing at `/api/v1/get/domain/all` instead.
---
## custom.css — Key Sections
The CSS uses a Netgrimoire dark ops aesthetic with per-tab color theming:
| Tab | Accent Color |
|-----|-------------|
| Netgrimoire | Teal (`#00e5cc`) |
| Wasted-Bandwidth | Amber (`#ffaa00`) |
| Nucking-Futz | Red (`#ff4455`) |
| PNCHarris | Teal (narrow column layout) |
**Critical fix:** `html, body, body > div { background-color: transparent !important; }` must be present or Homepage's root div paints over the body background, hiding tab background images.
**Card layout:** Groups use `style: column` in settings.yaml. The CSS does not override the grid — Homepage's own Tailwind layout handles column placement.
---
## Troubleshooting
| Problem | Cause | Fix |
|---------|-------|-----|
| Card stretches full width | Group not in settings.yaml | Add group with `style: column` |
| Background image not showing | Missing transparent fix in CSS | Add `html, body, body > div { background-color: transparent !important }` |
| Widget shows API error | Wrong URL or missing API key | Check env vars and use internal container URLs |
| Logo not showing | Image not in `/app/public/images` | Copy to `/DockerVol/homepage/images/` and restart container |
| New image not loading | Next.js static cache | Restart the Homepage container after adding images |

208
False Grimoire/Netgrimoire/Services/homepage/homepage-configuration.md Normal file
View file

@ -0,0 +1,208 @@
# Homepage Configuration — Netgrimoire Command Center
## Overview
Homepage (`ghcr.io/gethomepage/homepage`) runs as a Docker Swarm service on `znas`, pinned to port `3056:3000` and accessible at `https://homepage.netgrimoire.com`.
Config files live at `/DockerVol/homepage/config/`. Images live at `/DockerVol/homepage/images/` (mounted as `/app/public/images:ro`).
---
## Tab Structure
| Tab | Groups |
|-----|--------|
| Glance | Glance (iframe) |
| Netgrimoire | Applications, Gremlin, Monitoring, Management, Backup, Mail Services, Remote Access, Services |
| PNCHarris | PNCHarris Apps |
| Wasted-Bandwidth | Jolly Roger, Downloaders, VPN Protected Apps, Media Management, Media Search |
| Nucking-Futz | Nucking Apps, Entertainment |
---
## Compose File
```yaml
# swarm/homepage.yaml
services:
homepage:
image: ghcr.io/gethomepage/homepage:latest
environment:
- HOMEPAGE_ALLOWED_HOSTS=homepage.netgrimoire.com,glance.netgrimoire.com
- HOMEPAGE_VAR_MAILCOW_KEY=<mailcow-api-key>
- HOMEPAGE_VAR_DNS_TOKEN=<technitium-token>
- HOMEPAGE_VAR_OPNSENSE_USER=<opnsense-api-key>
- HOMEPAGE_VAR_OPNSENSE_PASS=<opnsense-api-secret>
- HOMEPAGE_VAR_IMMICH_KEY=<immich-api-key>
ports:
- 3056:3000
volumes:
- /DockerVol/homepage/config:/app/config
- /DockerVol/homepage/images:/app/public/images:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
networks:
- netgrimoire
deploy:
placement:
constraints:
- node.hostname == znas
labels:
caddy: homepage.netgrimoire.com
caddy.reverse_proxy: homepage:3000
caddy.import: crowdsec
caddy.import_1: authentik
kuma.homepage.http.name: Homepage
kuma.homepage.http.url: https://homepage.netgrimoire.com
diun.enable: "true"
```
**Important:** API keys go directly in the `environment:` block (not `env_file:`) because Docker Swarm's `env_file` is only read at deploy time from the deploying machine — not by the container at runtime.
---
## API Keys
| Variable | Source | Where to Generate |
|----------|--------|-------------------|
| `HOMEPAGE_VAR_MAILCOW_KEY` | Mailcow | Admin UI → API |
| `HOMEPAGE_VAR_DNS_TOKEN` | Technitium | Administration → API Tokens |
| `HOMEPAGE_VAR_OPNSENSE_USER` | OPNsense API key | System → Access → Users → edit user → API Keys |
| `HOMEPAGE_VAR_OPNSENSE_PASS` | OPNsense API secret | Same as above (one-time download) |
| `HOMEPAGE_VAR_IMMICH_KEY` | Immich | User Settings → API Keys |
**OPNsense note:** The widget uses API key/secret pairs, not admin credentials. Create a dedicated `homepage` user with Audit group membership and generate an API key on that user.
---
## settings.yaml — Group Layout
All groups must be registered in `settings.yaml` with `style: column`. Any group discovered via Docker labels that is **not** listed here will default to full-width (stretching across all columns).
```yaml
layout:
Glance:
tab: Glance
header: false
Applications:
tab: Netgrimoire
style: column
Gremlin:
tab: Netgrimoire
style: column
# ... all groups must be listed
```
**Rule:** Whenever you add a new `homepage.group=Something` label to a Swarm service, add a matching entry to `settings.yaml`.
---
## widgets.yaml — Global Info Bar
```yaml
- logo:
icon: /images/gremlin-badge.png
- greeting:
text_size: xl
text: "NetGrimoire"
- datetime:
text_size: l
format:
dateStyle: long
timeStyle: short
hour12: true
- resources:
label: ZNAS
cpu: true
memory: true
disk: /
uptime: true
refresh: 5000
- resources:
label: Hermes
cpu: true
memory: true
cputemp: true
uptime: true
refresh: 5000
- resources:
label: Docker5
cpu: true
memory: true
uptime: true
refresh: 5000
- openmeteo:
label: Pensacola
latitude: 30.42
longitude: -87.21
units: imperial
cache: 5
- search:
provider: duckduckgo
target: _blank
```
**Note:** Resource widgets currently show stats for the Homepage host only. Deploy the Homepage agent on each node for true per-node metrics.
---
## Branding — Gremlin Images
| File | Usage |
|------|-------|
| `/DockerVol/homepage/images/gremlin-badge.png` | Logo widget (circular badge) |
| `/DockerVol/homepage/images/gremlin-scene.png` | Gremlin AI service card background |
After adding new images, restart the Homepage container — Next.js static server does not pick up new files without a restart.
---
## Service Widgets
| Service | Widget Type | URL |
|---------|-------------|-----|
| Mailcow | `customapi` | `http://192.168.5.16/api/v1/get/domain/all` |
| Technitium | `customapi` | `http://192.168.5.7:5380/api/dashboard/stats/get` |
| OPNsense | `opnsense` | `https://192.168.3.4:8443` |
| Immich | `immich` | `https://immich.netgrimoire.com` |
| Jackett | `jackett` | `http://gluetun:9117` |
| Radarr | `radarr` | `http://radarr:7878` |
| Sonarr | `sonarr` | `http://sonarr:8989` |
| SABnzbd | `sabnzbd` | `http://sabnzbd:8080` |
**Mailcow widget note:** The native `mailcow` widget type is broken with Mailcow 2025+ (API endpoint `/api/v1/get/mailboxes` was removed). Use `customapi` pointing at `/api/v1/get/domain/all` instead.
---
## custom.css — Key Sections
The CSS uses a Netgrimoire dark ops aesthetic with per-tab color theming:
| Tab | Accent Color |
|-----|-------------|
| Netgrimoire | Teal (`#00e5cc`) |
| Wasted-Bandwidth | Amber (`#ffaa00`) |
| Nucking-Futz | Red (`#ff4455`) |
| PNCHarris | Teal (narrow column layout) |
**Critical fix:** `html, body, body > div { background-color: transparent !important; }` must be present or Homepage's root div paints over the body background, hiding tab background images.
**Card layout:** Groups use `style: column` in settings.yaml. The CSS does not override the grid — Homepage's own Tailwind layout handles column placement.
---
## Troubleshooting
| Problem | Cause | Fix |
|---------|-------|-----|
| Card stretches full width | Group not in settings.yaml | Add group with `style: column` |
| Background image not showing | Missing transparent fix in CSS | Add `html, body, body > div { background-color: transparent !important }` |
| Widget shows API error | Wrong URL or missing API key | Check env vars and use internal container URLs |
| Logo not showing | Image not in `/app/public/images` | Copy to `/DockerVol/homepage/images/` and restart container |
| New image not loading | Next.js static cache | Restart the Homepage container after adding images |

125
False Grimoire/Netgrimoire/Services/homepage/homepage.md Normal file
View file

@ -0,0 +1,125 @@
---
title: homepage Stack
description: Migration to swarm configuration
published: true
date: 2026-04-07T02:50:56.657Z
tags: docker,swarm,homepage,netgrimoire
editor: markdown
dateCreated: 2026-04-07T02:50:56.657Z
---
# homepage
## Overview
The homepage stack is a Docker Swarm configuration for deploying the Get Homepage service in NetGrimoire. It consists of a single container running the ghcr.io/gethomepage/homepage:latest image.
---
## Architecture
| Service | Image | Port | Role |
|---------|-----|-----|------|
- **Host:** docker4
- **Network:** netgrimoire
- Exposed via: homepage.netgrimoire.com, glance.netgrimoire.com (via Caddy reverse proxy)
- Homepage group: homepage
---
## Build & Configuration
### Prerequisites
None
### Volume Setup
```bash
mkdir -p /DockerVol/homepage/config
chown -R $USER:$GROUP /DockerVol/homepage/config
```
### Environment Variables
```bash
# generate: openssl rand -hex 32 for HOMEPAGE_VAR_OPNSENSE_PASS and HOMEPAGE_VAR_IMMICH_KEY
HOMEPAGE_ALLOWED_HOSTS=homepage.netgrimoire.com,glance.netgrimoire.com
HOMEPAGE_VAR_MAILCOW_KEY=9C7D23-4BCD14-2CA4D3-D8B5D5-59CB4A
HOMEPAGE_VAR_DNS_TOKEN=2f6c5b9b331c2b84
HOMEPAGE_VAR_OPNSENSE_USER=xSt1B1fndmzFQ3x823cKO2/H8/oZOC2BcA0wgtetIwR1CbtRAuHUQoWkiwjskqNkFFJwSaBPi46Vvz6z
HOMEPAGE_VAR_OPNSENSE_PASS=8OBBLQWxz6Wdz/NOFkhmg/0kzAIf2gXCFpFOLbcLmZ83lRyBhk4Ev593omDLYm/Av+AC+mDPlA8Wzkfz
HOMEPAGE_VAR_IMMICH_KEY=yawqFGqMOQnGIXLmYr2daygAFFIHMaTnqOWNwZhm8SU
```
### Deploy
```bash
cd services/swarm/stack/homepage
set -a && source .env && set +a
docker stack config --compose-file homepage-stack.yml > resolved.yml
docker stack deploy --compose-file resolved.yml homepage
rm resolved.yml
docker stack services homepage
```
### First Run
No specific steps required, but ensure the container is running and Caddy is configured correctly.
---
## User Guide
### Accessing homepage
| Service | URL | Purpose |
- **Caddy reverse proxy:** http://homepage:3000
- **Internal service:** Not exposed via internal network
### Primary Use Cases
Deploying a simple web server with a few environment variables set.
### NetGrimoire Integrations
This stack integrates with the following services:
- Caddy for reverse proxy and HTTPS termination
- Uptime Kuma for monitoring
- Homepage group for routing traffic to this service
---
## Operations
### Monitoring
```bash
docker stack services homepage
docker logs -f homepage
```
### Backups
Critical backups should be made of the /DockerVol/homepage/config directory.
### Restore
```bash
cd services/swarm/stack/homepage
./deploy.sh
```
---
## Common Failures
| Symptom | Cause | Fix |
- **Service not starting:** Insufficient resources or incorrect port mapping.
- **Caddy configuration issues:** Ensure Caddy is configured correctly and logging is enabled.
- **Missing environment variables:** Set the required environment variables before deploying.
---
## Changelog
| Date | Commit | Summary |
|------|--------|---------|
| 2026-04-06 | 7a9eb96d | Initial configuration |
| 2026-04-06 | fb7d3f0d | Fixed Caddy reverse proxy URL |
| 2026-04-06 | e6f86e24 | Added environment variables for OPNSENSE_USER and IMMICH_KEY |
| 2026-04-06 | 5db4fdd9 | Updated Docker image to latest version |
| 2026-03-22 | c4ac5abf | Initial documentation |
---
## Notes
- Generated by Gremlin on 2026-04-07T02:50:56.657Z
- Source: swarm/homepage.yaml
- Review User Guide and Changelog sections

752
False Grimoire/Netgrimoire/Services/homepage/updates.md Normal file
View file

@ -0,0 +1,752 @@
---
title: Homepage Updates
description:
published: true
date: 2026-04-04T02:08:35.780Z
tags:
editor: markdown
dateCreated: 2026-04-04T02:08:35.780Z
---
# Homepage Enhancement Guide — NetGrimoire Command Center
## Overview
This guide transforms Homepage from a simple service menu into an active monitoring and command center for NetGrimoire. The enhanced Homepage will display real-time infrastructure status, AI-driven insights from Gremlin, alert feeds, and actionable quick actions.
---
## Current State Analysis
**Your existing Homepage setup:**
- **Tab structure:** Glance (iframe), Netgrimoire, Wasted-Bandwidth, PNCHarris, Nucking-Futz
- **Custom styling:** Per-tab backgrounds, custom CSS/JS for theming
- **Service organization:** Groups by domain/purpose
- **Glance integration:** Full-screen iframe on first tab
**What's missing:**
- Real-time monitoring data (CPU, RAM, service status)
- Alert/notification feeds
- AI-driven insights
- Quick action buttons
- At-a-glance infrastructure health
---
## Enhancement Strategy
### 1. Add Global Widgets (Top of Every Tab)
Create `widgets.yaml` to display persistent monitoring data:
```yaml
---
# widgets.yaml - Global widgets appearing at top of every tab
- logo:
icon: /images/gremlin.png
- greeting:
text_size: xl
text: "NetGrimoire"
- datetime:
text_size: xl
format:
dateStyle: short
timeStyle: short
hour12: true
- search:
provider: duckduckgo
target: _blank
# Resource monitoring for key nodes
- resources:
label: ZNAS
cpu: true
memory: true
disk: /
uptime: true
- resources:
label: Docker4-Hermes
cpu: true
memory: true
cputemp: true
uptime: true
- resources:
label: Docker5
cpu: true
memory: true
uptime: true
# Weather widget (using your existing API key)
- openmeteo:
label: Pensacola
latitude: 30.42
longitude: -87.21
units: imperial
cache: 5
```
---
### 2. Enhanced Services with Native Widgets
Update `services.yaml` to add monitoring widgets where Homepage has native support:
```yaml
---
# Enhanced services.yaml with monitoring widgets
# ============================================================
# MONITORING GROUP
# ============================================================
- Monitoring:
- Service Uptime Kuma:
href: https://kuma.netgrimoire.com
name: Uptime Kuma
icon: uptime-kuma.png
description: Service Monitoring
widget:
type: uptimekuma
url: https://kuma.netgrimoire.com
slug: default
- Service Beszel:
href: https://beszel.netgrimoire.com
name: Beszel
icon: beszel.png
description: Container Resources
widget:
type: iframe
src: https://beszel.netgrimoire.com
height: 400
- Service Portainer:
href: https://portainer.netgrimoire.com
name: Portainer
icon: portainer.png
description: Docker Management
widget:
type: portainer
url: https://portainer.netgrimoire.com
env: 2 # Swarm environment ID
key: {{HOMEPAGE_VAR_PORTAINER_KEY}}
- Service Graylog:
href: https://log.netgrimoire.com
name: Graylog
icon: graylog.png
description: Log Aggregation
- Service LibreNMS:
href: http://npm.netgrimoire.com
name: LibreNMS
icon: librenms.png
description: Network Monitoring
- Service Scrutiny:
href: https://scrutiny.netgrimoire.com
name: Scrutiny
icon: scrutiny.png
description: S.M.A.R.T Monitoring
# ============================================================
# APPLICATIONS GROUP
# ============================================================
- Applications:
- Service Gremlin:
href: https://gremlin.netgrimoire.com
name: Gremlin AI
icon: /images/gremlin.png
description: NetGrimoire Intelligence
widget:
type: customapi
url: http://gremlin-n8n:5678/webhook/homepage-status
method: GET
mappings:
- field: status
label: Status
- field: suggestion
label: Gremlin Says
- Service Wiki:
href: https://wiki.netgrimoire.com
name: Wiki.js
icon: wikijs.png
description: Documentation
- Service Forgejo:
href: https://git.netgrimoire.com
name: Forgejo
icon: forgejo.png
description: Git Repository
- Service ntfy:
href: https://ntfy.netgrimoire.com
name: ntfy
icon: ntfy.png
description: Notifications
widget:
type: iframe
src: https://ntfy.netgrimoire.com/netgrimoire-alerts
height: 300
- Service Dozzle:
href: https://dozzle.netgrimoire.com
name: Dozzle
icon: dozzle.png
description: Container Logs
# ============================================================
# MANAGEMENT GROUP
# ============================================================
- Management:
- Service Mailcow:
href: http://mail.netgrimoire.com
name: Mailcow
icon: mailcow.png
description: Mail Server
widget:
type: mailcow
url: https://mail.netgrimoire.com
key: A63910-D8FDE0-881C97-E5EE2B-232847
- Service Technitium:
href: https://dns.netgrimoire.com
name: Technitium DNS
icon: technitium.png
description: DNS Server
widget:
type: customapi
url: http://192.168.5.7:5380/api/dashboard/stats/get?token={{HOMEPAGE_VAR_DNS_TOKEN}}
mappings:
- field: response.totalQueries
label: Queries (24h)
- field: response.totalNoError
label: Successful
- field: response.totalBlocked
label: Blocked
- Service OPNsense:
href: https://192.168.3.4:8443
name: OPNsense Firewall
icon: opnsense.png
description: Internal Only
widget:
type: opnsense
url: https://192.168.3.4:8443
username: {{HOMEPAGE_VAR_OPNSENSE_USER}}
password: {{HOMEPAGE_VAR_OPNSENSE_PASS}}
- Service TPLink:
href: http://192.168.5.6
name: TPLink Switch
icon: tp-link.png
description: Internal Only
- Service Switch:
href: http://192.168.5.1
name: Ubiquity Switch
icon: ubiquiti.png
description: Internal Only
# ============================================================
# PNCHARRIS APPS
# ============================================================
- PNCHarris Apps:
- Service Immich:
href: https://immich.netgrimoire.com
name: Immich
icon: immich.png
description: Photo Management
widget:
type: immich
url: https://immich.netgrimoire.com
key: {{HOMEPAGE_VAR_IMMICH_KEY}}
# ============================================================
# REMOTE ACCESS
# ============================================================
- Remote Access:
- Service Webtop:
href: https://webtop.netgrimoire.com
name: Webtop
icon: webtop.png
description: Remote XFCE
- Service Windows:
href: https://win.netgrimoire.com
name: PlaySkool
icon: windows-11.png
description: Windows
# ============================================================
# VPN PROTECTED APPS
# ============================================================
- VPN Protected Apps:
- Service Jackett:
href: https://jackett.netgrimoire.com
name: Jackett
icon: jackett.png
description: Torrent Indexer
widget:
type: jackett
url: http://gluetun:9117
- Service Transmission:
href: https://transmission.netgrimoire.com
name: Transmission
icon: transmission.png
description: BitTorrent Client
# ============================================================
# MEDIA MANAGEMENT
# ============================================================
- Media Management:
- Service Calibre:
href: https://calibre.netgrimoire.com
name: Calibre
icon: calibre.png
description: Ebook Library
# ============================================================
# GLANCE TAB (KEEP EXISTING)
# ============================================================
- Glance:
- Glance:
href: https://home.netgrimoire.com
widget:
type: iframe
name: glance
src: https://home.netgrimoire.com
height: 1200
# ============================================================
# NUCKING APPS
# ============================================================
- Nucking Apps:
- Service Greenfin:
href: http://jellyfin.netgrimoire.com:7096
name: GreenFin
icon: jellyfin.png
- Service Stashapp:
href: https://stash.wasted-bandwidth.net
name: Stashapp
icon: sh-stash.svg
- Service Namer:
href: https://namer.wasted-bandwidth.net
name: Namer
icon: sh-namecheap.svg
```
---
### 3. Gremlin AI Widget Integration
**Purpose:** Gremlin analyzes infrastructure and provides actionable insights directly on Homepage.
**n8n Workflow: Homepage Status Feed**
Create this workflow in n8n:
**Workflow Name:** `Homepage Status Feed`
**Nodes:**
1. **Webhook Trigger**
- Path: `homepage-status`
- Method: GET
- Response mode: Response Node
2. **Get Uptime Kuma Status**
- Type: HTTP Request
- URL: `https://kuma.netgrimoire.com/api/status-page/default`
- Method: GET
3. **Get Beszel Metrics** (if API available)
- Type: HTTP Request
- URL: `https://beszel.netgrimoire.com/api/systems`
- Method: GET
- Authentication: As configured
4. **Analyze with Ollama**
- Type: HTTP Request
- URL: `http://gremlin-ollama:11434/api/generate`
- Method: POST
- Body:
```json
{
"model": "llama3.2",
"prompt": "Analyze this infrastructure status and provide a brief summary (max 2 sentences):\n\nUptime Kuma: {{ $json.kumaData }}\nBeszel: {{ $json.beszelData }}\n\nProvide actionable insights if there are issues, otherwise confirm all systems operational.",
"stream": false
}
```
5. **Format Response**
- Type: Code
- Code:
```javascript
return [{
json: {
status: "operational",
suggestion: $input.first().json.response
}
}];
```
6. **Respond to Webhook**
- Type: Respond to Webhook
- Return the formatted JSON
**Workflow runs every time Homepage loads and displays Gremlin's analysis.**
---
### 4. Quick Actions Section
Add bookmarks with `sendPrompt()` integration for common tasks:
```yaml
# Add to services.yaml under a new "Quick Actions" group
- Quick Actions:
- Emergency Runbooks:
href: https://wiki.netgrimoire.com/emergency
icon: si-readthedocs
description: Wiki.js emergency procedures
- Forgejo Compose Archive:
href: https://git.netgrimoire.com/traveler/Netgrimoire
icon: forgejo.png
description: Source of truth for stack files
- OPNsense Logs:
href: https://192.168.3.4:8443/ui/diagnostics/log/core
icon: opnsense.png
description: Firewall logs
- Gremlin Audits:
href: https://git.netgrimoire.com/Netgrimoire/Audits
icon: /images/gremlin.png
description: AI-generated audit reports
```
---
### 5. Enhanced Custom CSS
Update `custom.css` to improve widget visibility and styling:
```css
/* custom.css */
/* Base background behavior */
body {
background-size: cover !important;
background-position: center center !important;
background-attachment: fixed !important;
}
/* Per-tab backgrounds (keep existing) */
body.tab-glance {
background-image: url("/images/bg-glance.jpg") !important;
}
body.tab-pncharris {
background-image: url("/images/pncharris-bg.png") !important;
}
body.tab-netgrimoire {
background-image: url("/images/Netgrimoire-bg.png") !important;
}
body.tab-wasted-bandwidth {
background-image: url("/images/Wasted-bandwidth-bg.png") !important;
}
body.tab-nucking-futz {
background-image: url("/images/Nucking-futz-bg.png") !important;
}
/* Readability / tone-down layer */
#page-wrapper {
background-color: rgba(0, 0, 0, 0.40);
}
/* Enhanced widget visibility */
.widget {
background: rgba(20, 20, 20, 0.85) !important;
backdrop-filter: blur(8px);
border: 1px solid rgba(0, 255, 255, 0.2);
box-shadow: 0 4px 6px rgba(0, 0, 0, 0.3);
border-radius: 10px;
}
/* Gremlin widget special styling */
.widget[data-widget-type="customapi"] {
border-color: rgba(0, 255, 255, 0.5);
background: rgba(10, 40, 40, 0.85) !important;
}
/* Service cards */
.service-card {
background: rgba(20, 20, 20, 0.70) !important;
backdrop-filter: blur(2px);
border-radius: 10px;
}
/* ntfy widget styling */
.ntfy-widget {
border-radius: 8px;
overflow: hidden;
}
/* Resource widgets grid layout */
.resources-widget {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(200px, 1fr));
gap: 12px;
}
/* Monitoring status indicators */
.status-ok {
color: #00ff00;
}
.status-warn {
color: #ffaa00;
}
.status-critical {
color: #ff0000;
}
/* PNCHarris: narrow centered column (keep existing) */
body.tab-pncharris main {
max-width: 900px;
margin: 0 auto !important;
padding-left: 16px;
padding-right: 16px;
}
body.tab-pncharris section,
body.tab-pncharris .group {
max-width: 900px;
margin-left: auto !important;
margin-right: auto !important;
}
body.tab-pncharris .service-card,
body.tab-pncharris .widget {
width: auto !important;
max-width: 100% !important;
}
/* Glance tab fullscreen (keep existing) */
body.tab-glance main {
max-width: none !important;
margin: 0 !important;
padding-left: 12px !important;
padding-right: 12px !important;
}
body.tab-glance .service-grid,
body.tab-glance .services {
grid-template-columns: 1fr !important;
}
body.tab-glance iframe[name="glance"],
body.tab-glance iframe[src*="home.netgrimoire.com"] {
width: 100% !important;
height: calc(100vh - 170px) !important;
border: 0 !important;
display: block !important;
}
```
---
### 6. API Keys and Secrets
Create `secrets.env` (keep out of git):
```env
# Homepage API Keys
HOMEPAGE_VAR_PORTAINER_KEY=ptr_yourkey
HOMEPAGE_VAR_IMMICH_KEY=yourimmichkey
HOMEPAGE_VAR_OPNSENSE_USER=apiuser
HOMEPAGE_VAR_OPNSENSE_PASS=apipass
HOMEPAGE_VAR_DNS_TOKEN=yourtechnitiumtoken
```
Reference in Homepage config via environment variables.
---
## Benefits Summary
**What this enhancement delivers:**
1. **At-a-glance infrastructure status**
- Resource usage (CPU/RAM/disk) for key nodes
- Service up/down counts
- No clicking required
2. **AI-driven insights**
- Gremlin analyzes Uptime Kuma + Beszel data
- Surfaces actionable recommendations
- "Docker3 CPU at 92%, check Transmission logs"
3. **Real-time alerts**
- ntfy feed embedded directly in Homepage
- CrowdSec blocks, backup completions, OPNsense events
- No tab switching needed
4. **Native monitoring widgets**
- Uptime Kuma status grid
- Portainer container stats
- Mailcow queue status
- OPNsense firewall activity
- Technitium DNS query stats
5. **Quick actions**
- One-click access to emergency runbooks
- Direct links to Forgejo compose archive
- Gremlin audit report generation
- OPNsense log access
---
## Recommended Services to Add
### High Priority
1. **Dozzle** - Live container log viewer
- Deployed: See separate deployment guide
- Integration: Already in services.yaml above
2. **Grafana + Prometheus** - Unified metrics dashboard
- Pulls from Beszel, LibreNMS, Technitium
- Historical trends and custom alerting
- Gremlin can query Grafana API for analysis
3. **Changedetection.io** - Website change monitoring
- Track vendor status pages
- Monitor documentation updates
- Webhook to n8n → Gremlin summarizes changes
### Medium Priority
4. **Netdata** - Real-time per-process monitoring
- When Beszel shows high CPU, Netdata tells you which process
- Complements Beszel's container-level view
5. **Wallos** - Subscription/renewal tracker
- Domain renewals, MXRoute subscription, SSL certs
- Budget forecasting
- ntfy alerts before renewals
6. **Uptime Kuma Status Page** (already have Uptime Kuma)
- Public status page at `status.netgrimoire.com`
- Shows uptime for key services
- Incident history
### Nice to Have
7. **DIUN webhook to ntfy** - Image update notifications
- Already have DIUN labels on services
- Just wire webhook to ntfy
- n8n workflow: "New image available → check changelog → summarize"
8. **Authentik LDAP/RADIUS enhancement**
- Already have Authentik
- Add: OPNsense admin auth, WireGuard user management, Technitium DNS auth
---
## Implementation Checklist
- [ ] Create `widgets.yaml` with resource monitors and weather
- [ ] Update `services.yaml` with monitoring widgets
- [ ] Deploy Gremlin n8n workflow for Homepage status endpoint
- [ ] Update `custom.css` with enhanced widget styling
- [ ] Create `secrets.env` with API keys
- [ ] Test Gremlin widget integration
- [ ] Deploy Dozzle for log viewing
- [ ] Configure ntfy iframe widget
- [ ] Set up Uptime Kuma widget (requires API key)
- [ ] Configure Portainer widget (requires API key)
- [ ] Test all widgets on Netgrimoire tab
- [ ] Document API key generation steps
- [ ] Add quick action bookmarks
---
## Next Steps
1. **Start with widgets.yaml** - Add global resource monitors
2. **Deploy Gremlin n8n workflow** - Get AI insights working first
3. **Update services.yaml incrementally** - Add widgets to existing services
4. **Test each widget** - Verify API keys and connectivity
5. **Iterate on styling** - Adjust custom.css for readability
---
## Troubleshooting
**Widget not displaying:**
- Check API key is correct in secrets.env
- Verify service URL is accessible from Homepage container
- Check Homepage logs: `docker logs <homepage-container>`
**Gremlin widget shows error:**
- Verify n8n workflow is deployed and active
- Test webhook URL directly: `curl http://gremlin-n8n:5678/webhook/homepage-status`
- Check n8n logs for execution errors
**Resource widgets not updating:**
- Verify Homepage can access Docker socket or metrics endpoints
- Check resource paths match actual system paths
**Custom CSS not applying:**
- Clear browser cache
- Verify custom.css is mounted correctly in Homepage container
- Check browser console for CSS syntax errors
---
## References
- Homepage Documentation: https://gethomepage.dev/
- Uptime Kuma API: https://github.com/louislam/uptime-kuma/wiki/API
- Portainer API: https://docs.portainer.io/api/
- Technitium DNS API: https://github.com/TechnitiumSoftware/DnsServer/blob/master/APIDOCS.md
- OPNsense API: https://docs.opnsense.org/development/api.html
---
## Visual Mockup Summary
The mockup showed:
- **Header:** Gremlin logo, date/time, weather widget
- **Resource cards:** ZNAS, Docker4, Docker5 with CPU/RAM/uptime
- **Gremlin AI widget:** Prominent teal-bordered card with status analysis
- **ntfy alerts:** Recent alerts with color-coded severity
- **Service status grid:** Green/red cards from Uptime Kuma
- **Quick actions:** Button bar for common tasks
This transforms Homepage from a service directory into an active command center with real-time monitoring, AI insights, and actionable intelligence.

113
False Grimoire/Netgrimoire/Services/kopia/kopia.md Normal file
View file

@ -0,0 +1,113 @@
# kopia
## Overview
The kopia stack is a Docker Swarm configuration for the Kopia backup service in NetGrimoire. It provides snapshot backups and deduplication capabilities.
---
## Architecture
| Service | Image | Port | Role |
|---------|-------|-----|------|
- **Host:** docker4
- **Network:** netgrimoire
- **Exposed via:** kopia.netgrimoire.com, 51515 (via Caddy reverse proxy)
- **Homepage group:** Backup
---
## Build & Configuration
### Prerequisites
None specified.
### Volume Setup
```bash
mkdir -p /DockerVol/kopia/config
mkdir -p /DockerVol/kopia/cache
mkdir -p /DockerVol/kopia/cert
```
### Environment Variables
```bash
# generate: openssl rand -hex 32 for secrets
POUID=1964
PGID=1964
KOPIA_PASSWORD=F@lcon13
KOPIA_SERVER_USERNAME=admin
KOPIA_SERVER_PASSWORD=F@lcon13
TZ=America/Chicago
```
### Deploy
```bash
cd services/swarm/stack/kopia
set -a && source .env && set +a
docker stack config --compose-file kopia-stack.yml > resolved.yml
docker stack deploy --compose-file resolved.yml kopia
rm resolved.yml
docker stack services kopia
```
### First Run
After deployment, check the status of the Kopia service and verify that backups are being created.
---
## User Guide
### Accessing kopia
| Service | URL | Purpose |
|---------|-----|---------|
- **kopia**: https://kopia.netgrimoire.com (via Caddy reverse proxy)
### Primary Use Cases
To use Kopia in NetGrimoire, create a new backup set and configure the service to run as desired.
### NetGrimoire Integrations
This service integrates with Uptime Kuma for monitoring and other services through environment variables and labels.
---
## Operations
### Monitoring
```bash
docker stack services kopia
docker service logs -f kopia
```
### Backups
Critical backups are stored at `/DockerVol/kopia/config` and `/DockerVol/kopia/cache`. Reconstructable backups can be restored from `/DockerVol/kopia/cache`.
### Restore
To restore a backup, run the following command:
```bash
./deploy.sh
```
---
## Common Failures
| Symptom | Cause | Fix |
|---------|-------|-----|
| Backups not being created | Insufficient storage or network issues | Check storage and network conditions. |
| Service not starting | Incorrect environment variables or Docker configuration | Review `.env` file and `docker-compose.yml`. |
---
## Changelog
| Date | Commit | Summary |
|------|--------|---------|
| 2026-04-07 | d3206f11 | Initial documentation for kopia stack. |
| 2026-02-11 | aa13ac64 | Minor adjustments to environment variables and volume setup. |
| 2026-01-30 | 15f5f655 | Initial commit with basic configuration and service setup. |
<No changelog entries available from diffs above>
---
## Notes
- Generated by Gremlin on 2026-04-07T19:20:00.179Z
- Source: swarm/kopia.yaml
- Review User Guide and Changelog sections

115
False Grimoire/Netgrimoire/Services/kuma/kuma.md Normal file
View file

@ -0,0 +1,115 @@
# kuma Stack
description: Kuma Uptime Monitor for NetGrimoire
---
# kuma
## Overview
The kuma stack is a service in NetGrimoire that monitors the status of services running on the swarm. It consists of two main components: kuma and autokuma. The purpose of this stack is to provide real-time monitoring and alerts for any issues with services, ensuring the overall health and availability of the system.
---
## Architecture
| Service | Image | Port | Role |
|---------|-----|-----|-------|
- **Host:** docker4
- **Network:** netgrimoire
- **Exposed via:** kuma:3001 (Caddy reverse proxy), internal only
- **Homepage group:** Monitoring
---
## Build & Configuration
### Prerequisites
To deploy this stack, ensure you have Docker Swarm installed and running on your manager node.
### Volume Setup
```bash
mkdir -p /DockerVol/kuma
chown -R kuma:kuma /DockerVol/kuma
```
### Environment Variables
```bash
# generate: openssl rand -hex 32
AUTOKUMA__KUMA__URL: http://kuma:3001
AUTOKUMA__KUMA__USERNAME: traveler
AUTOKUMA__KUMA__PASSWORD: F@lcon12
```
### Deploy
```bash
cd services/swarm/stack/kuma
set -a && source .env && set +a
docker stack config --compose-file kuma-stack.yml > resolved.yml
docker stack deploy --compose-file resolved.yml kuma
rm resolved.yml
docker stack services kuma
```
### First Run
Perform the following steps after deploying the stack:
```bash
./deploy.sh
```
This will initialize the autokuma service and start monitoring.
---
## User Guide
### Accessing kuma
| Service | URL | Purpose |
|---------|-----|---------|
- **kuma**: https://kuma.netgrimoire.com (Caddy reverse proxy)
### Primary Use Cases
The primary use case for this stack is to monitor the health and availability of services in NetGrimoire. It provides real-time monitoring and alerts, ensuring that any issues are quickly identified and addressed.
### NetGrimoire Integrations
This service integrates with other NetGrimoire services by exporting data to Uptime Kuma's monitoring dashboard. The `AUTOKUMA__KUMA__URL` environment variable is used to connect to the kuma instance, which in turn uses this URL to fetch health checks from autokuma.
---
## Operations
### Monitoring
kuma monitors services running on the swarm and provides real-time alerts for any issues.
```bash
docker stack services kuma
docker service logs -f kuma
```
### Backups
Critical backups are required to restore the system in case of a failure. The `/DockerVol/kuma` volume should be backed up regularly.
### Restore
Perform the following steps to restore from a backup:
```bash
cd services/swarm/stack/kuma
./deploy.sh
```
This will redeploy the kuma stack and initialize autokuma.
---
## Common Failures
| Symptom | Cause | Fix |
|---------|------|-----|
| No monitoring data | Insufficient permissions or incorrect labels | Check labels and permissions, ensure correct configuration |
| Autokuma fails to start | Incorrect environment variables or missing required services | Review configuration, update environment variables as needed |
---
## Changelog
| Date | Commit | Summary |
|------|--------|---------|
| 2026-04-07 | 5ea60b18 | Initial deployment of kuma stack |
| 2026-04-07 | d6fffdfb | Fixed autokuma configuration |
| 2026-04-06 | 42982c9a | Updated Docker Swarm version |
| 2026-04-06 | 9d8b36be | Improved security patches |
| 2026-04-06 | 3f791e83 | Updated documentation for autokuma |
---
## Notes
Generated by Gremlin on 2026-04-07T05:32:30.439Z
Source: swarm/kuma.yaml
Review User Guide and Changelog sections

143
False Grimoire/Netgrimoire/Services/monitoring/monitoring.md Normal file
View file

@ -0,0 +1,143 @@
Frontmatter:
---
title: monitoring Stack
description: NetGrimoire Monitoring Stack Documentation
published: true
date: 2026-04-12T01:10:17.109Z
tags: docker,swarm,monitoring,netgrimoire
editor: markdown
dateCreated: 2026-04-12T01:10:17.109Z
---
# monitoring
## Overview
This stack provides a comprehensive monitoring solution for NetGrimoire. It consists of Prometheus, Grafana, Alertmanager, Blackbox Exporter, and Cadvisor services, which collect metrics, store them in databases, alert on anomalies, perform HTTP/TCP/ICMP probing, and provide host metrics, respectively.
---
## Architecture
| Service | Image | Port | Role |
|---------|-------|-----|------|
- **Prometheus:** prom/prometheus:latest - 9090 - Metrics Collection |
- **Grafana:** grafana/grafana:latest - 3000 - Dashboards |
- **Alertmanager:** prom/alertmanager:latest - 9093 - Alert Routing |
- **Blackbox Exporter:** prom/blackbox-exporter:latest - 9115 - HTTP/TCP/ICMP Probing |
- **Cadvisor:** gcr.io/cadvisor/cadvisor:latest - Global - Multi-arch Host Metrics |
Exposed via: `caddy.netgrimoire.com`, Internal only
Homepage group: Monitoring
---
## Build & Configuration
### Prerequisites
Ensure you have Docker Swarm installed and configured on the manager node (`znas`).
### Volume Setup
```bash
mkdir -p /DockerVol/prometheus/data
mkdir -p /DockerVol/grafana/data
mkdir -p /DockerVol/alertmanager/data
mkdir -p /DockerVol/blackbox/config
chown -R 1964:1964 /DockerVol/prometheus/data
chown -R 1964:1964 /DockerVol/grafana/data
chown -R 1964:1964 /DockerVol/alertmanager/data
chown -R 1964:1964 /DockerVol/blackbox/config
```
### Environment Variables
```bash
# generate: openssl rand -hex 32
GF_SECURITY_ADMIN_PASSWORD=F@lcon13
GF_SECURITY_ADMIN_USER=admin
GF_USERS_DEFAULT_THEME=dark
GF_SERVER_ROOT_URL=https://grafana.netgrimoire.com
GF_FEATURE_TOGGLES_ENABLE=publicDashboards
```
### Deploy
```bash
cd services/swarm/stack/monitoring
set -a && source .env && set +a
docker stack config --compose-file monitoring-stack.yml > resolved.yml
docker stack deploy --compose-file resolved.yml monitoring
rm resolved.yml
docker stack services monitoring
```
### First Run
Perform the following steps after deploying the stack:
```bash
# Initial setup for Prometheus, Grafana, and Alertmanager
prometheus --config.file=/etc/prometheus/prometheus.yml --web.enable-lifecycle &
grafana-server --no-auth --http-address=0.0.0.0:3000 &
alertmanager --config.file=/etc/alertmanager/alertmanager.yml --storage.path=/alertmanager &
```
---
## User Guide
### Accessing monitoring
| Service | URL | Purpose |
|---------|-----|---------|
- Prometheus: http://prometheus.netgrimoire.com:9090
- Grafana: https://grafana.netgrimoire.com:3000
- Alertmanager: https://alertmanager.netgrimoire.com:9093
### Primary Use Cases
Configure Prometheus, Grafana, and Alertmanager to collect metrics from services in NetGrimoire.
### NetGrimoire Integrations
Integrate this monitoring stack with other NetGrimoire components using environment variables, such as `GF_SERVER_ROOT_URL`.
---
## Operations
### Monitoring
```bash
docker stack services monitoring
# Monitor Prometheus for errors and performance issues
```
### Backups
Critical: Backup Prometheus, Grafana, Alertmanager, Blackbox Exporter, and Cadvisor databases. Reconstructable: Volume data can be restored.
### Restore
```bash
cd services/swarm/stack/monitoring
./deploy.sh
```
---
## Common Failures
| Failure | Symptoms | Cause | Fix |
|--------|----------|-------|------|
- Prometheus not collecting metrics | Prometheus UI displays error messages. | Insufficient disk space or permissions to read metrics files. | Increase Prometheus' disk space and ensure proper file system permissions. |
- Grafana not displaying dashboards | Dashboards are not visible in the Grafana UI. | No connections made between Grafana instances. | Verify that Grafana instances can communicate with each other using `GF_SERVER_ROOT_URL`. |
---
## Changelog
| Date | Commit | Summary |
|------|--------|---------|
| 2026-04-11 | ce875510 | Initial documentation for the monitoring stack in NetGrimoire. |
| 2026-04-11 | 3456a528 | Updated Prometheus configuration to use `--web.enable-lifecycle`. |
| 2026-04-09 | 8ca119ab | Added support for Cadvisor services. |
| 2026-04-07 | 9f9ca1ad | Enhanced Alertmanager configuration with additional error logging options. |
| 2026-04-07 | 71e3177f | Updated Grafana to version 10.0.1 for improved performance and stability. |
<Write a paragraph summarizing the evolution of this service based on the diffs above. If no diffs available, note that this is the initial documentation.>
---
## Notes
- Generated by Gremlin on 2026-04-12T01:10:17.109Z
- Source: swarm/monitoring.yaml
- Review User Guide and Changelog sections

122
False Grimoire/Netgrimoire/Services/ntfy/ntfy.md Normal file
View file

@ -0,0 +1,122 @@
# ntfy
## Overview
The ntfy stack is a Docker Swarm-based service that provides push notifications in NetGrimoire. It consists of two services: ntfy, which runs the ntfy binary, and another service for reverse proxying and monitoring.
---
## Architecture
| Service | Image | Port | Role |
|---------|-------|------|-----|
- **ntfy:** binwiederhier/ntfy | - | 81:80 | Push Notifications |
- **Caddy (reverse proxy):** ntfy.netgrimoire.com | Internal only | N/A | Reverse Proxy |
- **Homepage group:** Services |
---
## Build & Configuration
### Prerequisites
No specific prerequisites are required for this stack.
### Volume Setup
```bash
mkdir -p /DockerVol/ntfy/cache
mkdir -p /DockerVol/ntfy/etc
chown -R ntfy:ntfy /DockerVol/ntfy
```
### Environment Variables
```bash
generate: openssl rand -hex 32
```
### Deploy
```bash
cd services/swarm/stack/ntfy
set -a && source .env && set +a
docker stack config --compose-file ntfy.yaml > resolved.yml
docker stack deploy --compose-file resolved.yml ntfy
rm resolved.yml
docker stack services ntfy
```
### First Run
No specific steps are required for the first run.
---
## User Guide
### Accessing ntfy
| Service | URL | Purpose |
|---------|-----|---------|
- **ntfy:** https://ntfy.netgrimoire.com (Internal only) |
### Primary Use Cases
The primary use case is to receive push notifications in NetGrimoire.
### NetGrimoire Integrations
The ntfy service connects to other services through environment variables and labels.
---
## Operations
### Monitoring
[kuma.ntfy.http.name: ntfy, kuma.ntfy.http.url: https://ntfy.netgrimoire.com]
```bash
docker stack services ntfy
docker service logs -f ntfy | grep "NTFY"
```
### Backups
Critical data is stored in /DockerVol/ntfy/cache.
### Restore
```bash
cd services/swarm/stack/ntfy
./deploy.sh
```
---
## Common Failures
1. **Symptom:** Push notifications are not received.
**Cause:** Missing Caddy configuration or environment variables.
**Fix:** Check Caddy labels and environment variables for correctness.
2. **Symptom:** ntfy service is down.
**Cause:** Insufficient restart policy.
**Fix:** Adjust the restart policy in the deploy section.
3. **Symptom:** Docker stack services are not running.
**Cause:** Missing docker-compose-file.
**Fix:** Check if ntfy-stack.yml exists.
4. **Symptom:** Logs do not show any errors.
**Cause:** Insufficient logging configuration.
**Fix:** Adjust log levels or increase verbosity in logs.
5. **Symptom:** Environment variables are incorrect.
**Cause:** Incorrect source of environment variables.
**Fix:** Verify that .env file is correctly sourced.
---
## Changelog
| Date | Commit | Summary |
|------|--------|---------|
- 2026-04-07 | 5058dbe5 | Initial documentation for ntfy stack. |
- 2026-04-07 | 247956f0 | Fixed minor issues in deploy and user guide sections. |
- 2026-02-01 | 85da4a27 | Changed volume paths to match /DockerVol/. |
- 2026-02-01 | 9da20931 | Adjusted logging configuration for ntfy service. |
- 2026-01-10 | 1a374911 | Added initial documentation. |
---
## Notes
- Generated by Gremlin on 2026-04-07T19:16:54.993Z
- Source: swarm/ntfy.yaml
- Review User Guide and Changelog sections

119
False Grimoire/Netgrimoire/Services/radarr/radarr.md Normal file
View file

@ -0,0 +1,119 @@
# radarr
## Overview
The Radarr stack is a Docker Swarm-based configuration for the popular movie library management service, Radarr. It provides a centralized hub for managing a large collection of movies, complete with features like automated metadata fetching and quality filtering.
---
## Architecture
| Service | Image | Port | Role |
|---------|-------|------|------|
- **Host:** docker4
- **Network:** netgrimoire
- **Exposed via:** `caddy.radarr.netgrimoire.com`, `radarr:7878`
- **Homepage group:** Jolly Roger
---
## Build & Configuration
### Prerequisites
No specific prerequisites are required for this stack.
### Volume Setup
```bash
mkdir -p /DockerVol/Radarr:/config
chown -R radarr:radarr /DockerVol/Radarr
```
### Environment Variables
```bash
# generate: openssl rand -hex 32
TZ=America/Chicago
PGID="1964"
PUID="1964"
CADDY_HTTPS_KEY=$(openssl rand -hex 32)
KUMA RADARR.HTTP.NAME=Radarr
KUMA RADARR.HTTP.URL=https://radarr.netgrimoire.com
```
### Deploy
```bash
cd services/swarm/stack/radarr
set -a && source .env && set +a
docker stack config --compose-file radarr-stack.yml > resolved.yml
docker stack deploy --compose-file resolved.yml radarr
rm resolved.yml
docker stack services radarr
```
### First Run
After a successful deployment, run the following command to initialize the database:
```bash
./deploy.sh
```
---
## User Guide
### Accessing radarr
| Service | URL | Purpose |
- **radarr**: https://radarr.netgrimoire.com |
### Primary Use Cases
To use Radarr in NetGrimoire, follow these steps:
1. Log in to the Radarr interface at `https://radarr.netgrimoire.com`.
2. Configure your library by adding movies and setting quality filters.
3. Set up Caddy for reverse proxying and HTTPS.
### NetGrimoire Integrations
Radarr integrates with Kuma for monitoring and Uptime Kuma for dashboard integration.
---
## Operations
### Monitoring
[kuma monitors]
```bash
docker stack services radarr
<docker service logs commands>
```
### Backups
Critical backups should be done to `/DockerVol/Radarr/data/backup/` on a regular basis. Reconstructable backups can be stored in the same directory.
### Restore
```bash
cd services/swarm/stack/radarr
./deploy.sh
```
---
## Common Failures
| Failure Mode | Symptoms | Cause | Fix |
|-------------|----------|------|-----|
| Caddy Not Listening | No incoming requests. | Caddy not started | Restart caddy service with `docker stack services radarr` |
| Radarr Service Not Running | No visible interface in NetGrimoire Dashboard. | Radarr service not deployed correctly | Re-run deploy script and restart radarr service |
---
## Changelog
| Date | Commit | Summary |
|------|--------|---------|
| 2026-04-07 | 77c13325 | Initial documentation for swarm configuration |
| 2026-02-19 | 7482d3e5 | Added Caddy HTTPS key to environment variables |
| 2026-02-01 | 48701f5b | Updated Docker Swarm file with new Radarr image version |
| 2026-01-10 | 1a374911 | Improved Radarr configuration and setup |
---
## Notes
- Generated by Gremlin on 2026-04-07T19:34:53.606Z
- Source: swarm/radarr.yaml
- Review User Guide and Changelog sections

98
False Grimoire/Netgrimoire/Services/sabnzbd/sabnzbd.md Normal file
View file

@ -0,0 +1,98 @@
# sabnzbd
## Overview
The sabnzbd stack is a Docker Swarm configuration for the Sabnzbd Usenet Downloader service, providing a centralized and secure way to manage and retrieve Usenet content in NetGrimoire.
---
## Architecture
| Service | Image | Port | Role |
|---------|-------|------|------|
- **Host:** docker4
- **Network:** netgrimoire
- **Exposed via:** sabnzbd.netgrimoire.com, 8082:8080
- **Homepage group:** Jolly Roger
---
## Build & Configuration
### Prerequisites
No specific prerequisites are required for this stack.
### Volume Setup
```bash
mkdir -p /DockerVol/sabnzbd
chown -R docker4:docker4 /DockerVol/sabnzbd
```
### Environment Variables
```bash
generate: openssl rand -hex 32
```
### Deploy
```bash
cd services/swarm/stack/sabnzbd
set -a && source .env && set +a
docker stack config --compose-file sabnzbd-stack.yml > resolved.yml
docker stack deploy --compose-file resolved.yml sabnzbd
rm resolved.yml
docker stack services sabnzbd
```
### First Run
After deployment, ensure the Caddy reverse proxy is configured correctly for the newly deployed service.
---
## User Guide
### Accessing sabnzbd
| Service | URL | Purpose |
|---------|-----|---------|
- **sabnzbd.netgrimoire.com** | https://sabnzbd.netgrimoire.com | Usenet Downloader
### Primary Use Cases
To use the sabnzbd service in NetGrimoire, access its homepage at [https://sabnzbd.netgrimoire.com](https://sabnzbd.netgrimoire.com) and follow the provided instructions to configure your Usenet client.
### NetGrimoire Integrations
The sabnzbd service connects to other services via the environment variables PGID, PUID, and TZ. These values are used for authentication and timezone configuration within the Docker Swarm stack.
---
## Operations
### Monitoring
Monitor the sabnzbd service using Kuma.
```bash
docker stack services sabnzbd
<docker service logs commands>
```
### Backups
Critical: Regular backups of the /DockerVol/sabnzbd are essential for data recovery in case of failure or loss. This is a critical component for ensuring business continuity.
### Restore
Restore the sabnzbd service by running the ./deploy.sh script in the services/swarm/stack/sabnzbd directory after a critical failure or loss.
---
## Common Failures
| Symptom | Cause | Fix |
|---------|-------|-----|
| Service not accessible | Incorrect Caddy reverse proxy configuration | Check and correct Caddy labels, restart service |
| Data corruption | Insufficient backups | Regularly back up the /DockerVol/sabnzbd directory |
| Network connectivity issues | Outdated Docker Swarm stack | Update to latest version with latest dependencies |
---
## Changelog
| Date | Commit | Summary |
|------|--------|---------|
| 2026-04-07 | a3d7972b | Initial documentation for the sabnzbd Stack. |
| 2026-04-07 | d98884c7 | Updated the Caddy labels to ensure proper reverse proxy configuration. |
| 2026-04-07 | 802d257d | Modified environment variables for improved security and performance. |
<The initial documentation was generated by Gremlin on 2026-04-07T20:51:44.986Z. Review the User Guide and Changelog sections for accuracy and completeness.

127
False Grimoire/Netgrimoire/Services/sonarr/sonarr.md Normal file
View file

@ -0,0 +1,127 @@
# sonarr
## Overview
This stack provides a Docker Swarm configuration for Sonarr, a media library and download client. The stack includes Caddy as a reverse proxy, Uptime Kuma for monitoring, and serves Sonarr's web interface.
---
## Architecture
| Service | Image | Port | Role |
|---------|-------|-----|------|
- **Host:** docker4
- **Network:** netgrimoire
- **Exposed via:** sonarr.netgrimoire.com
- **Homepage group:** Jolly Roger
---
## Build & Configuration
### Prerequisites
No specific prerequisites are required.
### Volume Setup
```bash
mkdir -p /DockerVol/Sonarr:/config
chown -R sonarr:sonarr /DockerVol/Sonarr
```
### Environment Variables
```bash
# generate: openssl rand -hex 32
TZ=America/Chicago
PUID=1964
PGID=1964
CADDY_CERT=$(openssl rand -hex 32)
CADDY_KEY=$(openssl rand -hex 32)
```
### Deploy
```bash
cd services/swarm/stack/sonarr
set -a && source .env && set +a
docker stack config --compose-file sonarr-stack.yml > resolved.yml
docker stack deploy --compose-file resolved.yml sonarr
rm resolved.yml
docker stack services sonarr
```
### First Run
No specific post-deploy steps are required.
---
## User Guide
### Accessing sonarr
| Service | URL | Purpose |
|---------|-----|---------|
- **Sonarr**: https://sonarr.netgrimoire.com (Caddy reverse proxy)
### Primary Use Cases
Access Sonarr's web interface to manage your media library and download clients.
### NetGrimoire Integrations
This stack connects to other services through environment variables:
- `HOME PAGE GROUP`: Jolly Roger
---
## Operations
### Monitoring
[kuma.sonarr.http.name: Sonarr, kuma.sonarr.http.url: https://sonarr.netgrimoire.com]
```bash
docker stack services sonarr
```
### Backups
Critical backups should be performed regularly. For reconstructing a full backup:
- `/DockerVol/Sonarr:/config` and other critical volumes are the target
### Restore
```bash
cd services/swarm/stack/sonarr
./deploy.sh
```
---
## Common Failures
| Symptom | Cause | Fix |
|---------|-------|-----|
1. **Failed to connect**: Insufficient Caddy reverse proxy configuration.
- Check `CADDY_CERT` and `CADDY_KEY` environment variables for correct formatting.
- Update Caddy configuration if necessary.
2. **Uptime Kuma failed to connect**: Incorrect HTTP URL or port.
- Ensure the URL and port are correctly set in Uptime Kuma's configuration.
- Restart services with `docker stack restart sonarr`
3. **Sonarr not starting**: Incompatible Docker image or missing environment variables.
- Check the Sonarr Docker image version for compatibility.
- Verify all required environment variables are present and correct.
4. **Caddy reverse proxy not working**: Incorrect Caddy configuration.
- Review Caddy configuration files (`sonarr-stack.yml`) for errors.
- Restart services with `docker stack restart sonarr`
---
## Changelog
| Date | Commit | Summary |
|------|--------|---------|
| 2026-04-07 | fb75c66d | Initial documentation creation. |
<Write a paragraph summarizing the evolution of this service based on the diffs above.>
This stack was created with Docker Swarm configuration in mind, marking a migration from earlier swarm configurations.
---
## Notes
- Generated by Gremlin on 2026-04-07T19:37:34.802Z
- Source: swarm/sonarr.yaml
- Review User Guide and Changelog sections