traveler/Netgrimoire
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

New Grimoire

Browse source
This commit is contained in:
traveler 2026-04-12 09:53:51 -05:00
parent 77d589a13d
commit cc574f8aed
157 changed files with 29420 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

841
Vault-Grimoire/Backups/Immich-Backup.md Normal file
View file

@ -0,0 +1,841 @@
---
title: Immich Backup and Restore
description: Immich backup with Kopia
published: true
date: 2026-02-20T04:11:52.181Z
tags:
editor: markdown
dateCreated: 2026-02-14T03:14:32.594Z
---
# Immich Backup and Recovery Guide
## Overview
This document provides comprehensive backup and recovery procedures for Immich photo server. Since Immich's data is stored on standard filesystems (not ZFS or BTRFS), snapshots are not available and we rely on Immich's native backup approach combined with Kopia for offsite storage in vaults.
## Quick Reference
### Common Backup Commands
```bash
# Run a manual backup (all components)
/opt/scripts/backup-immich.sh
# Backup just the database
docker exec -t immich_postgres pg_dump --clean --if-exists \
--dbname=immich --username=postgres | gzip > "/opt/immich-backups/dump.sql.gz"
# List Kopia snapshots
kopia snapshot list --tags immich
# View backup logs
tail -f /var/log/immich-backup.log
```
### Common Restore Commands
```bash
# Restore database from backup
gunzip < /opt/immich-backups/immich-YYYYMMDD_HHMMSS/dump.sql.gz | \
docker exec -i immich_postgres psql --username=postgres --dbname=immich
# Restore from Kopia to new server
kopia snapshot list --tags tier1-backup
kopia restore <snapshot-id> /opt/immich-backups/
# Check container status after restore
docker compose ps
docker compose logs -f
```
## Critical Components to Backup
### 1. Docker Compose File
- **Location**: `/opt/immich/docker-compose.yml` (or your installation path)
- **Purpose**: Defines all containers, networks, and volumes
- **Importance**: Critical for recreating the exact container configuration
### 2. Configuration Files
- **Primary Config**: `/opt/immich/.env`
- **Purpose**: Database credentials, upload locations, timezone settings
- **Importance**: Required for proper service initialization
### 3. Database
- **PostgreSQL Data**: Contains all metadata, user accounts, albums, sharing settings, face recognition data, timeline information
- **Container**: `immich_postgres`
- **Database Name**: `immich` (default)
- **User**: `postgres` (default)
- **Backup Method**: `pg_dump` (official Immich recommendation)
### 4. Photo/Video Library
- **Upload Storage**: All original photos and videos uploaded by users
- **Location**: `/srv/immich/library` (per your .env UPLOAD_LOCATION)
- **Size**: Typically the largest component
- **Critical**: This is your actual data - photos cannot be recreated
### 5. Additional Important Data
- **Model Cache**: Docker volume `immich_model-cache` (machine learning models, can be re-downloaded)
- **External Paths**: `/export/photos` and `/srv/NextCloud-AIO` (mounted as read-only in your setup)
## Backup Strategy
### Two-Tier Backup Approach
We use a **two-tier approach** combining Immich's native backup method with Kopia for offsite storage:
1. **Tier 1 (Local)**: Immich database dump + library backup creates consistent, component-level backups
2. **Tier 2 (Offsite)**: Kopia snapshots the local backups and syncs to vaults
#### Why This Approach?
- **Best of both worlds**: Native database dump ensures Immich-specific consistency, Kopia provides deduplication and offsite protection
- **Component-level restore**: Can restore individual components (just database, just library, etc.)
- **Disaster recovery**: Full system restore from Kopia backups on new server
- **Efficient storage**: Kopia's deduplication reduces storage needs for offsite copies
#### Backup Frequency
- **Daily**: Immich backup runs at 2 AM
- **Daily**: Kopia snapshot of backups runs at 3 AM
- **Retention (Local)**: 7 days of Immich backups (managed by script)
- **Retention (Kopia/Offsite)**: 30 daily, 12 weekly, 12 monthly
### Immich Native Backup Method
Immich's official backup approach uses `pg_dump` for the database:
- Uses `pg_dump` with `--clean --if-exists` flags for consistent database dumps
- Hot backup without stopping PostgreSQL
- Produces compressed `.sql.gz` files
- Database remains available during backup
For the photo/video library, we use a **hybrid approach**:
- **Database**: Backed up locally as `dump.sql.gz` for fast component-level restore
- **Library**: Backed up directly by Kopia (no tar) for optimal deduplication and incremental backups
**Why not tar the library?**
- Kopia deduplicates at the file level - adding 1 photo shouldn't require backing up the entire library again
- Individual file access for selective restore
- Better compression and faster incremental backups
- Lower risk - corrupted tar loses everything, corrupted file only affects that file
**Key Features:**
- No downtime required
- Consistent point-in-time snapshot
- Standard PostgreSQL format (portable across systems)
- Efficient incremental backups of photo library
## Setting Up Immich Backups
### Prereq:
Make sure you are connected to the repository,
```bash
sudo kopia repository connect server \
--url=https://192.168.5.10:51516 \
--override-username=admin \
--server-cert-fingerprint=696a4999f594b5273a174fd7cab677d8dd1628f9b9d27e557daa87103ee064b2
```
#### Step 1: Configure Backup Location
Set the backup destination:
```bash
# Create the backup directory
mkdir -p /opt/immich-backups
chown -R root:root /opt/immich-backups
chmod 755 /opt/immich-backups
```
#### Step 2: Manual Backup Commands
```bash
cd /opt/immich
# Backup database using Immich's recommended method
docker exec -t immich_postgres pg_dump \
--clean \
--if-exists \
--dbname=immich \
--username=postgres \
| gzip > "/opt/immich-backups/dump.sql.gz"
# Backup configuration files
cp docker-compose.yml /opt/immich-backups/
cp .env /opt/immich-backups/
# Backup library with Kopia (no tar - better deduplication)
kopia snapshot create /srv/immich/library \
--tags immich,library,photos \
--description "Immich library manual backup"
```
**What gets created:**
- Local backup directory: `/opt/immich-backups/immich-YYYY-MM-DD-HH-MM-SS/`
- Contains: `dump.sql.gz` (database), config files
- Kopia snapshots:
- `/opt/immich-backups` (database + config)
- `/srv/immich/library` (photos/videos, no tar)
- `/opt/immich` (installation directory)
#### Step 3: Automated Backup Script
Create `/opt/scripts/backup-immich.sh`:
```bash
#!/bin/bash
# Immich Automated Backup Script
# This creates Immich backups, then snapshots them with Kopia for offsite storage
set -e
BACKUP_DATE=$(date +%Y%m%d_%H%M%S)
LOG_FILE="/var/log/immich-backup.log"
IMMICH_DIR="/opt/immich"
BACKUP_DIR="/opt/immich-backups"
KEEP_DAYS=7
# Database credentials from .env
DB_USERNAME="postgres"
DB_DATABASE_NAME="immich"
POSTGRES_CONTAINER="immich_postgres"
echo "[${BACKUP_DATE}] ========================================" | tee -a "$LOG_FILE"
echo "[${BACKUP_DATE}] Starting Immich backup process" | tee -a "$LOG_FILE"
# Step 1: Run Immich database backup using official method
echo "[${BACKUP_DATE}] Running Immich database backup..." | tee -a "$LOG_FILE"
cd "$IMMICH_DIR"
# Create backup directory with timestamp
mkdir -p "${BACKUP_DIR}/immich-${BACKUP_DATE}"
# Backup database using Immich's recommended method
docker exec -t ${POSTGRES_CONTAINER} pg_dump \
--clean \
--if-exists \
--dbname=${DB_DATABASE_NAME} \
--username=${DB_USERNAME} \
| gzip > "${BACKUP_DIR}/immich-${BACKUP_DATE}/dump.sql.gz"
BACKUP_EXIT=${PIPESTATUS[0]}
if [ $BACKUP_EXIT -ne 0 ]; then
echo "[${BACKUP_DATE}] ERROR: Immich database backup failed with exit code ${BACKUP_EXIT}" | tee -a "$LOG_FILE"
exit 1
fi
echo "[${BACKUP_DATE}] Immich database backup completed successfully" | tee -a "$LOG_FILE"
# Step 2: Verify library location exists (Kopia will backup directly, no tar needed)
echo "[${BACKUP_DATE}] Verifying library location..." | tee -a "$LOG_FILE"
# Get the upload location from docker-compose volumes
UPLOAD_LOCATION="/srv/immich/library"
if [ -d "${UPLOAD_LOCATION}" ]; then
#LIBRARY_SIZE=$(du -sh ${UPLOAD_LOCATION} | cut -f1)
echo "[${BACKUP_DATE}] Library location verified: ${UPLOAD_LOCATION} (${LIBRARY_SIZE})" | tee -a "$LOG_FILE"
echo "[${BACKUP_DATE}] Kopia will backup library files directly (no tar, better deduplication)" | tee -a "$LOG_FILE"
else
echo "[${BACKUP_DATE}] WARNING: Upload location not found at ${UPLOAD_LOCATION}" | tee -a "$LOG_FILE"
fi
# Step 3: Backup configuration files
echo "[${BACKUP_DATE}] Backing up configuration files..." | tee -a "$LOG_FILE"
cp "${IMMICH_DIR}/docker-compose.yml" "${BACKUP_DIR}/immich-${BACKUP_DATE}/"
cp "${IMMICH_DIR}/.env" "${BACKUP_DIR}/immich-${BACKUP_DATE}/"
echo "[${BACKUP_DATE}] Configuration backup completed" | tee -a "$LOG_FILE"
# Step 4: Clean up old backups
echo "[${BACKUP_DATE}] Cleaning up backups older than ${KEEP_DAYS} days..." | tee -a "$LOG_FILE"
find "${BACKUP_DIR}" -maxdepth 1 -type d -name "immich-*" -mtime +${KEEP_DAYS} -exec rm -rf {} \; 2>&1 | tee -a "$LOG_FILE"
echo "[${BACKUP_DATE}] Local backup cleanup completed" | tee -a "$LOG_FILE"
# Step 5: Create Kopia snapshot of backup directory
echo "[${BACKUP_DATE}] Creating Kopia snapshot..." | tee -a "$LOG_FILE"
kopia snapshot create "${BACKUP_DIR}" \
--tags immich:tier1-backup \
--description "Immich backup ${BACKUP_DATE}" \
2>&1 | tee -a "$LOG_FILE"
KOPIA_EXIT=${PIPESTATUS[0]}
if [ $KOPIA_EXIT -ne 0 ]; then
echo "[${BACKUP_DATE}] WARNING: Kopia snapshot failed with exit code ${KOPIA_EXIT}" | tee -a "$LOG_FILE"
echo "[${BACKUP_DATE}] Local Immich backup exists but offsite copy may be incomplete" | tee -a "$LOG_FILE"
exit 2
fi
echo "[${BACKUP_DATE}] Kopia snapshot completed successfully" | tee -a "$LOG_FILE"
# Step 6: Backup the library directly with Kopia (better deduplication than tar)
echo "[${BACKUP_DATE}] Creating Kopia snapshot of library..." | tee -a "$LOG_FILE"
if [ -d "${UPLOAD_LOCATION}" ]; then
kopia snapshot create "${UPLOAD_LOCATION}" \
--tags immich:library \
--description "Immich library ${BACKUP_DATE}" \
2>&1 | tee -a "$LOG_FILE"
KOPIA_LIB_EXIT=${PIPESTATUS[0]}
if [ $KOPIA_LIB_EXIT -ne 0 ]; then
echo "[${BACKUP_DATE}] WARNING: Kopia library snapshot failed" | tee -a "$LOG_FILE"
else
echo "[${BACKUP_DATE}] Library snapshot completed successfully" | tee -a "$LOG_FILE"
fi
fi
# Step 7: Also backup the Immich installation directory (configs, compose files)
#echo "[${BACKUP_DATE}] Backing up Immich installation directory..." | tee -a "$LOG_FILE"
#kopia snapshot create "${IMMICH_DIR}" \
# --tags immich,config,docker-compose \
# --description "Immich config ${BACKUP_DATE}" \
# 2>&1 | tee -a "$LOG_FILE"
echo "[${BACKUP_DATE}] Backup process completed successfully" | tee -a "$LOG_FILE"
echo "[${BACKUP_DATE}] ========================================" | tee -a "$LOG_FILE"
# Optional: Send notification on completion
# Add your notification method here (email, webhook, etc.)
```
Make it executable:
```bash
chmod +x /opt/scripts/backup-immich.sh
```
Add to crontab (daily at 2 AM):
```bash
# Edit root's crontab
crontab -e
# Add this line:
0 2 * * * /opt/scripts/backup-immich.sh 2>&1 | logger -t immich-backup
```
### Offsite Backup to Vaults
After local Kopia snapshots are created, they sync to your offsite vaults automatically through Kopia's repository configuration.
## Recovery Procedures
### Understanding Two Recovery Methods
We have **two restore methods** depending on the scenario:
1. **Local Restore** (Preferred): For component-level or same-server recovery
2. **Kopia Full Restore**: For complete disaster recovery to a new server
### Method 1: Local Restore (Recommended)
Use this method when:
- Restoring on the same/similar server
- Restoring specific components (just database, just library, etc.)
- Recovering from local Immich backups
#### Full System Restore
```bash
cd /opt/immich
# Stop Immich
docker compose down
# List available backups
ls -lh /opt/immich-backups/
# Choose a database backup
BACKUP_PATH="/opt/immich-backups/immich-YYYYMMDD_HHMMSS"
# Restore database
gunzip < ${BACKUP_PATH}/dump.sql.gz | \
docker compose exec -T database psql --username=postgres --dbname=immich
# Restore library from Kopia
kopia snapshot list --tags library
kopia restore <library-snapshot-id> /srv/immich/library
# Fix permissions
chown -R 1000:1000 /srv/immich/library
# Restore configuration (review changes first)
cp ${BACKUP_PATH}/.env .env.restored
cp ${BACKUP_PATH}/docker-compose.yml docker-compose.yml.restored
# Start Immich
docker compose up -d
# Monitor logs
docker compose logs -f
```
#### Example: Restore Only Database
```bash
cd /opt/immich
# Stop Immich
docker compose down
# Start only database
docker compose up -d database
sleep 10
# Restore database from backup
BACKUP_PATH="/opt/immich-backups/immich-YYYYMMDD_HHMMSS"
gunzip < ${BACKUP_PATH}/dump.sql.gz | \
docker compose exec -T database psql --username=postgres --dbname=immich
# Start all services
docker compose down
docker compose up -d
# Verify
docker compose logs -f
```
#### Example: Restore Only Library
```bash
cd /opt/immich
# Stop Immich
docker compose down
# Restore library from Kopia
kopia snapshot list --tags library
kopia restore <library-snapshot-id> /srv/immich/library
# Fix permissions
chown -R 1000:1000 /srv/immich/library
# Start Immich
docker compose up -d
```
### Method 2: Complete Server Rebuild (Kopia Restore)
Use this when recovering to a completely new server or when local backups are unavailable.
#### Step 1: Prepare New Server
```bash
# Update system
apt update && apt upgrade -y
# Install Docker
curl -fsSL https://get.docker.com | sh
systemctl enable docker
systemctl start docker
# Install Docker Compose
apt install docker-compose-plugin -y
# Install Kopia
curl -s https://kopia.io/signing-key | sudo gpg --dearmor -o /usr/share/keyrings/kopia-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/kopia-keyring.gpg] https://packages.kopia.io/apt/ stable main" | sudo tee /etc/apt/sources.list.d/kopia.list
apt update
apt install kopia -y
# Create directory structure
mkdir -p /opt/immich
mkdir -p /opt/immich-backups
mkdir -p /srv/immich/library
mkdir -p /srv/immich/postgres
```
#### Step 2: Restore Kopia Repository
```bash
# Connect to your offsite vault
kopia repository connect server \
--url=https://192.168.5.10:51516 \
--override-username=admin \
--server-cert-fingerprint=696a4999f594b5273a174fd7cab677d8dd1628f9b9d27e557daa87103ee064b2
# List available snapshots
kopia snapshot list --tags immich
```
#### Step 3: Restore Configuration
```bash
# Find and restore the config snapshot
kopia snapshot list --tags config
# Restore to the Immich directory
kopia restore <snapshot-id> /opt/immich/
# Verify critical files
ls -la /opt/immich/.env
ls -la /opt/immich/docker-compose.yml
```
#### Step 4: Restore Immich Backups Directory
```bash
# Restore the entire backup directory from Kopia
kopia snapshot list --tags tier1-backup
# Restore the most recent backup
kopia restore <snapshot-id> /opt/immich-backups/
# Verify backups were restored
ls -la /opt/immich-backups/
```
#### Step 5: Restore Database and Library
```bash
cd /opt/immich
# Find the most recent backup
LATEST_BACKUP=$(ls -td /opt/immich-backups/immich-* | head -1)
echo "Restoring from: $LATEST_BACKUP"
# Start database container
docker compose up -d database
sleep 30
# Restore database
gunzip < ${LATEST_BACKUP}/dump.sql.gz | \
docker compose exec -T database psql --username=postgres --dbname=immich
# Restore library from Kopia
kopia snapshot list --tags library
kopia restore <library-snapshot-id> /srv/immich/library
# Fix permissions
chown -R 1000:1000 /srv/immich/library
```
#### Step 6: Start and Verify Immich
```bash
cd /opt/immich
# Pull latest images (or use versions from backup if preferred)
docker compose pull
# Start all services
docker compose up -d
# Monitor logs
docker compose logs -f
```
#### Step 7: Post-Restore Verification
```bash
# Check container status
docker compose ps
# Test web interface
curl -I http://localhost:2283
# Verify database
docker compose exec database psql -U postgres -d immich -c "SELECT COUNT(*) FROM users;"
# Check library storage
ls -lah /srv/immich/library/
```
### Scenario 2: Restore Individual User's Photos
To restore a single user's library without affecting others:
**Option A: Using Kopia Mount (Recommended)**
```bash
# Mount the Kopia snapshot
kopia snapshot list --tags library
mkdir -p /mnt/kopia-library
kopia mount <library-snapshot-id> /mnt/kopia-library &
# Find the user's directory (using user ID from database)
# User libraries are typically in: library/{user-uuid}/
USER_UUID="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
# Copy user's data back
rsync -av /mnt/kopia-library/${USER_UUID}/ \
/srv/immich/library/${USER_UUID}/
# Fix permissions
chown -R 1000:1000 /srv/immich/library/${USER_UUID}/
# Unmount
kopia unmount /mnt/kopia-library
# Restart Immich to recognize changes
cd /opt/immich
docker compose restart immich-server
```
**Option B: Selective Kopia Restore**
```bash
cd /opt/immich
docker compose down
# Restore just the specific user's directory
kopia snapshot list --tags library
USER_UUID="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
# Restore with path filter
kopia restore <library-snapshot-id> /srv/immich/library \
--snapshot-path="${USER_UUID}"
# Fix permissions
chown -R 1000:1000 /srv/immich/library/${USER_UUID}/
# Start Immich
docker compose up -d
```
### Scenario 3: Database Recovery Only
If only the database is corrupted but library data is intact:
```bash
cd /opt/immich
# Stop Immich
docker compose down
# Start only database
docker compose up -d database
sleep 30
# Restore from most recent backup
LATEST_BACKUP=$(ls -td /opt/immich-backups/immich-* | head -1)
gunzip < ${LATEST_BACKUP}/dump.sql.gz | \
docker compose exec -T database psql --username=postgres --dbname=immich
# Start all services
docker compose down
docker compose up -d
# Verify
docker compose logs -f
```
### Scenario 4: Configuration Recovery Only
If you only need to restore configuration files:
```bash
cd /opt/immich
# Find the most recent backup
LATEST_BACKUP=$(ls -td /opt/immich-backups/immich-* | head -1)
# Stop Immich
docker compose down
# Backup current config (just in case)
cp .env .env.pre-restore
cp docker-compose.yml docker-compose.yml.pre-restore
# Restore config from backup
cp ${LATEST_BACKUP}/.env ./
cp ${LATEST_BACKUP}/docker-compose.yml ./
# Restart
docker compose up -d
```
## Verification and Testing
### Regular Backup Verification
Perform monthly restore tests to ensure backups are valid:
```bash
# Test restore to temporary location
mkdir -p /tmp/backup-test
kopia snapshot list --tags immich
kopia restore <snapshot-id> /tmp/backup-test/
# Verify files exist and are readable
ls -lah /tmp/backup-test/
gunzip < /tmp/backup-test/immich-*/dump.sql.gz | head -100
# Cleanup
rm -rf /tmp/backup-test/
```
### Backup Monitoring Script
Create `/opt/scripts/check-immich-backup.sh`:
```bash
#!/bin/bash
# Check last backup age
LAST_BACKUP=$(ls -td /opt/immich-backups/immich-* 2>/dev/null | head -1)
if [ -z "$LAST_BACKUP" ]; then
echo "WARNING: No Immich backups found"
exit 1
fi
BACKUP_DATE=$(basename "$LAST_BACKUP" | sed 's/immich-//')
BACKUP_EPOCH=$(date -d "${BACKUP_DATE:0:8} ${BACKUP_DATE:9:2}:${BACKUP_DATE:11:2}:${BACKUP_DATE:13:2}" +%s 2>/dev/null)
if [ -z "$BACKUP_EPOCH" ]; then
echo "WARNING: Cannot parse backup date"
exit 1
fi
NOW=$(date +%s)
AGE_HOURS=$(( ($NOW - $BACKUP_EPOCH) / 3600 ))
if [ $AGE_HOURS -gt 26 ]; then
echo "WARNING: Last Immich backup is $AGE_HOURS hours old"
# Send alert (email, Slack, etc.)
exit 1
else
echo "OK: Last backup $AGE_HOURS hours ago"
fi
# Check Kopia snapshots
KOPIA_LAST=$(kopia snapshot list --tags immich --json 2>/dev/null | jq -r '.[0].startTime' 2>/dev/null)
if [ -n "$KOPIA_LAST" ]; then
echo "Last Kopia snapshot: $KOPIA_LAST"
else
echo "WARNING: Cannot verify Kopia snapshots"
fi
```
## Disaster Recovery Checklist
When disaster strikes, follow this checklist:
- [ ] Confirm scope of failure (server, storage, specific component)
- [ ] Gather server information (hostname, IP, DNS records)
- [ ] Access offsite backup vault
- [ ] Provision new server (if needed)
- [ ] Install Docker and dependencies
- [ ] Connect to Kopia repository
- [ ] Restore configurations first
- [ ] Restore database
- [ ] Restore library data
- [ ] Start services and verify
- [ ] Test photo viewing and uploads
- [ ] Verify user accounts and albums
- [ ] Update DNS records if needed
- [ ] Document any issues encountered
- [ ] Update recovery procedures based on experience
## Important Notes
1. **External Mounts**: Your setup has `/export/photos` and `/srv/NextCloud-AIO` mounted as external read-only sources. These are not backed up by this script - ensure they have their own backup strategy.
2. **Database Password**: The default database password in your .env is `postgres`. Change this to a secure random password for production use.
3. **Permissions**: Library files should be owned by UID 1000:1000 for Immich to access them properly:
```bash
chown -R 1000:1000 /srv/immich/library
```
4. **Testing**: Always test recovery procedures in a lab environment before trusting them in production.
5. **Documentation**: Keep this guide and server details in a separate location (printed copy, password manager, etc.).
6. **Retention Policy**: Review Kopia retention settings periodically to balance storage costs with recovery needs.
## Backup Architecture Notes
### Why Two Backup Layers?
**Immich Native Backups** (Tier 1):
- ✅ Uses official Immich backup method (`pg_dump`)
- ✅ Fast, component-aware backups
- ✅ Selective restore (can restore just database or just library)
- ✅ Standard PostgreSQL format (portable)
- ❌ No deduplication (full copies each time)
- ❌ Limited to local storage initially
**Kopia Snapshots** (Tier 2):
- ✅ Deduplication and compression
- ✅ Efficient offsite replication to vaults
- ✅ Point-in-time recovery across multiple versions
- ✅ Disaster recovery to completely new infrastructure
- ❌ Less component-aware (treats as files)
- ❌ Slower for granular component restore
### Storage Efficiency
Using this two-tier approach:
- **Local**: Database backups (~7 days retention, relatively small)
- **Kopia**: Database backups + library (efficient deduplication)
**Why library goes directly to Kopia without tar:**
Example with 500GB library, adding 10GB photos/month:
**With tar approach:**
- Month 1: Backup 500GB tar
- Month 2: Add 10GB photos → Entire 510GB tar changes → Backup 510GB
- Month 3: Add 10GB photos → Entire 520GB tar changes → Backup 520GB
- **Total storage needed**: 500 + 510 + 520 = 1,530GB
**Without tar (Kopia direct):**
- Month 1: Backup 500GB
- Month 2: Add 10GB photos → Kopia only backs up the 10GB new files
- Month 3: Add 10GB photos → Kopia only backs up the 10GB new files
- **Total storage needed**: 500 + 10 + 10 = 520GB
**Savings**: ~66% reduction in storage and backup time!
This is why we:
- Keep database dumps local (small, fast component restore)
- Let Kopia handle library directly (efficient, incremental, deduplicated)
### Compression and Deduplication
**Database backups** use `gzip` compression:
- Typically 80-90% compression ratio for SQL dumps
- Small enough to keep local copies
**Library backups** use Kopia's built-in compression and deduplication:
- Photos (JPEG/HEIC): Already compressed, Kopia skips re-compression
- Videos: Already compressed, minimal additional compression
- RAW files: Some compression possible
- **Deduplication**: If you upload the same photo twice, Kopia stores it once
- **Block-level dedup**: Even modified photos share unchanged blocks
This is far more efficient than tar + gzip, which would:
- Compress already-compressed photos (wasted CPU, minimal benefit)
- Store entire archive even if only 1 file changed
- Prevent deduplication across backups
## Additional Resources
- [Immich Official Backup Documentation](https://immich.app/docs/administration/backup-and-restore)
- [Kopia Documentation](https://kopia.io/docs/)
- [Docker Volume Backup Best Practices](https://docs.docker.com/storage/volumes/#back-up-restore-or-migrate-data-volumes)
- [PostgreSQL pg_dump Documentation](https://www.postgresql.org/docs/current/app-pgdump.html)
## Revision History
| Date | Version | Changes |
|------|---------|---------|
| 2026-02-13 | 1.0 | Initial documentation - two-tier backup strategy using Immich's native backup method |
---
**Last Updated**: February 13, 2026
**Maintained By**: System Administrator
**Review Schedule**: Quarterly

879
Vault-Grimoire/Backups/MailCow-Backup.md Normal file
View file

@ -0,0 +1,879 @@
---
title: Mailcow Backup and Restore Strategy
description: Mailcow backup
published: true
date: 2026-02-20T04:15:25.924Z
tags:
editor: markdown
dateCreated: 2026-02-11T01:20:59.127Z
---
# Mailcow Backup and Recovery Guide
## Overview
This document provides comprehensive backup and recovery procedures for Mailcow email server. Since Mailcow is **not running on ZFS or BTRFS**, snapshots are not available and we rely on Mailcow's native backup script combined with Kopia for offsite storage in vaults.
## Quick Reference
### Common Backup Commands
```bash
# Run a manual backup (all components)
cd /opt/mailcow-dockerized
MAILCOW_BACKUP_LOCATION=/opt/mailcow-backups \
./helper-scripts/backup_and_restore.sh backup all --delete-days 7
# Backup with multithreading (faster)
THREADS=4 MAILCOW_BACKUP_LOCATION=/opt/mailcow-backups \
./helper-scripts/backup_and_restore.sh backup all --delete-days 7
# List Kopia snapshots
kopia snapshot list --tags mailcow
# View backup logs
tail -f /var/log/mailcow-backup.log
```
### Common Restore Commands
```bash
# Restore using mailcow native script (interactive)
cd /opt/mailcow-dockerized
./helper-scripts/backup_and_restore.sh restore
# Restore from Kopia to new server
kopia snapshot list --tags tier1-backup
kopia restore <snapshot-id> /opt/mailcow-backups/
# Check container status after restore
docker compose ps
docker compose logs -f
```
## Critical Components to Backup
### 1. Docker Compose File
- **Location**: `/opt/mailcow-dockerized/docker-compose.yml` (or your installation path)
- **Purpose**: Defines all containers, networks, and volumes
- **Importance**: Critical for recreating the exact container configuration
### 2. Configuration Files
- **Primary Config**: `/opt/mailcow-dockerized/mailcow.conf`
- **Additional Configs**:
- `/opt/mailcow-dockerized/data/conf/` (all subdirectories)
- Custom SSL certificates if not using Let's Encrypt
- Any override files (e.g., `docker-compose.override.yml`)
### 3. Database
- **MySQL/MariaDB Data**: Contains all mailbox configurations, users, domains, aliases, settings
- **Docker Volume**: `mailcowdockerized_mysql-vol`
- **Container Path**: `/var/lib/mysql`
### 4. Email Data
- **Maildir Storage**: All actual email messages
- **Docker Volume**: `mailcowdockerized_vmail-vol`
- **Container Path**: `/var/vmail`
- **Size**: Typically the largest component
### 5. Additional Important Data
- **Redis Data**: `mailcowdockerized_redis-vol` (cache and sessions)
- **Rspamd Data**: `mailcowdockerized_rspamd-vol` (spam learning)
- **Crypt Data**: `mailcowdockerized_crypt-vol` (if using mailbox encryption)
- **Postfix Queue**: `mailcowdockerized_postfix-vol` (queued/deferred mail)
## Backup Strategy
### Two-Tier Backup Approach
We use a **two-tier approach** combining Mailcow's native backup script with Kopia for offsite storage:
1. **Tier 1 (Local)**: Mailcow's `backup_and_restore.sh` script creates consistent, component-level backups
2. **Tier 2 (Offsite)**: Kopia snapshots the local backups and syncs to vaults
#### Why This Approach?
- **Best of both worlds**: Native script ensures mailcow-specific consistency, Kopia provides deduplication and offsite protection
- **Component-level restore**: Can restore individual components (just vmail, just mysql, etc.) using mailcow script
- **Disaster recovery**: Full system restore from Kopia backups on new server
- **Efficient storage**: Kopia's deduplication reduces storage needs for offsite copies
#### Backup Frequency
- **Daily**: Mailcow native backup runs at 2 AM
- **Daily**: Kopia snapshot of backups runs at 3 AM
- **Retention (Local)**: 7 days of mailcow backups (managed by script)
- **Retention (Kopia/Offsite)**: 30 daily, 12 weekly, 12 monthly
### Mailcow Native Backup Script
Mailcow includes `/opt/mailcow-dockerized/helper-scripts/backup_and_restore.sh` which handles:
- **vmail**: Email data (mailboxes)
- **mysql**: Database (using mariabackup for consistency)
- **redis**: Redis database
- **rspamd**: Spam filter learning data
- **crypt**: Encryption data
- **postfix**: Mail queue
**Key Features:**
- Uses `mariabackup` (hot backup without stopping MySQL)
- Supports multithreading for faster backups
- Architecture-aware (handles x86/ARM differences)
- Built-in cleanup with `--delete-days` parameter
- Creates compressed archives (.tar.zst or .tar.gz)
### Setting Up Mailcow Backups
#### Prereq:
Make sure you are connected to the repository,
```bash
sudo kopia repository connect server --url=https://192.168.5.10:51516 --override-username=admin --server-cert-fingerprint=696a4999f594b5273a174fd7cab677d8dd1628f9b9d27e557daa87103ee064b2
```
#### Step 1: Configure Backup Location
Set the backup destination via environment variable or in mailcow.conf:
```bash
# Option 1: Set environment variable (preferred for automation)
export MAILCOW_BACKUP_LOCATION="/opt/mailcow-backups"
# Option 2: Add to cron job directly (shown in automated script below)
```
Create the backup directory:
```bash
mkdir -p /opt/mailcow-backups
chown -R root:root /opt/mailcow-backups
chmod 777 /opt/mailcow-backups
```
#### Step 2: Manual Backup Commands
```bash
cd /opt/mailcow-dockerized
# Backup all components, delete backups older than 7 days
MAILCOW_BACKUP_LOCATION=/opt/mailcow-backups \
./helper-scripts/backup_and_restore.sh backup all --delete-days 7
# Backup with multithreading (faster for large mailboxes)
THREADS=4 MAILCOW_BACKUP_LOCATION=/opt/mailcow-backups \
./helper-scripts/backup_and_restore.sh backup all --delete-days 7
# Backup specific components only
MAILCOW_BACKUP_LOCATION=/opt/mailcow-backups \
./helper-scripts/backup_and_restore.sh backup vmail mysql --delete-days 7
```
**What gets created:**
- Backup directory: `/opt/mailcow-backups/mailcow-YYYY-MM-DD-HH-MM-SS/`
- Contains: `.tar.zst` compressed archives for each component
- Plus: `mailcow.conf` copy for restore reference
#### Step 3: Automated Backup Script
Create `/opt/scripts/backup-mailcow.sh`:
```bash
#!/bin/bash
# Mailcow Automated Backup Script
# This creates mailcow native backups, then snapshots them with Kopia for offsite storage
set -e
BACKUP_DATE=$(date +%Y%m%d_%H%M%S)
LOG_FILE="/var/log/mailcow-backup.log"
MAILCOW_DIR="/opt/mailcow-dockerized"
BACKUP_DIR="/opt/mailcow-backups"
THREADS=4 # Adjust based on your CPU cores
KEEP_DAYS=7 # Keep local mailcow backups for 7 days
echo "[${BACKUP_DATE}] ========================================" | tee -a "$LOG_FILE"
echo "[${BACKUP_DATE}] Starting Mailcow backup process" | tee -a "$LOG_FILE"
# Step 1: Run mailcow's native backup script
echo "[${BACKUP_DATE}] Running mailcow native backup..." | tee -a "$LOG_FILE"
cd "$MAILCOW_DIR"
# Run the backup with multithreading
THREADS=${THREADS} MAILCOW_BACKUP_LOCATION=${BACKUP_DIR} \
./helper-scripts/backup_and_restore.sh backup all --delete-days ${KEEP_DAYS} \
2>&1 | tee -a "$LOG_FILE"
BACKUP_EXIT=${PIPESTATUS[0]}
if [ $BACKUP_EXIT -ne 0 ]; then
echo "[${BACKUP_DATE}] ERROR: Mailcow backup failed with exit code ${BACKUP_EXIT}" | tee -a "$LOG_FILE"
exit 1
fi
echo "[${BACKUP_DATE}] Mailcow native backup completed successfully" | tee -a "$LOG_FILE"
# Step 2: Create Kopia snapshot of backup directory
echo "[${BACKUP_DATE}] Creating Kopia snapshot..." | tee -a "$LOG_FILE"
kopia snapshot create "${BACKUP_DIR}" \
--tags mailcow:tier1-backup \
--description "Mailcow backup ${BACKUP_DATE}" \
2>&1 | tee -a "$LOG_FILE"
KOPIA_EXIT=${PIPESTATUS[0]}
if [ $KOPIA_EXIT -ne 0 ]; then
echo "[${BACKUP_DATE}] WARNING: Kopia snapshot failed with exit code ${KOPIA_EXIT}" | tee -a "$LOG_FILE"
echo "[${BACKUP_DATE}] Local mailcow backup exists but offsite copy may be incomplete" | tee -a "$LOG_FILE"
exit 2
fi
echo "[${BACKUP_DATE}] Kopia snapshot completed successfully" | tee -a "$LOG_FILE"
# Step 3: Also backup the mailcow installation directory (configs, compose files)
echo "[${BACKUP_DATE}] Backing up mailcow installation directory..." | tee -a "$LOG_FILE"
kopia snapshot create "${MAILCOW_DIR}" \
--tags mailcow,config,docker-compose \
--description "Mailcow config ${BACKUP_DATE}" \
2>&1 | tee -a "$LOG_FILE"
echo "[${BACKUP_DATE}] Backup process completed successfully" | tee -a "$LOG_FILE"
echo "[${BACKUP_DATE}] ========================================" | tee -a "$LOG_FILE"
# Optional: Send notification on completion
# Add your notification method here (email, webhook, etc.)
```
Make it executable:
```bash
chmod +x /opt/scripts/backup-mailcow.sh
```
Add to crontab (daily at 2 AM):
```bash
# Edit root's crontab
crontab -e
# Add this line:
0 2 * * * /opt/scripts/backup-mailcow.sh 2>&1 | logger -t mailcow-backup
```
### Offsite Backup to Vaults
After local Kopia snapshots are created, sync to your offsite vaults:
```bash
# Option 1: Kopia repository sync (if using multiple Kopia repos)
kopia repository sync-to filesystem --path /mnt/vault/mailcow-backup
# Option 2: Rsync to vault
rsync -avz --delete /backup/kopia-repo/ /mnt/vault/mailcow-backup/
# Option 3: Rclone to remote vault
rclone sync /backup/kopia-repo/ vault:mailcow-backup/
```
## Recovery Procedures
### Understanding Two Recovery Methods
We have **two restore methods** depending on the scenario:
1. **Mailcow Native Restore** (Preferred): For component-level or same-server recovery
2. **Kopia Full Restore**: For complete disaster recovery to a new server
### Method 1: Mailcow Native Restore (Recommended)
Use this method when:
- Restoring on the same/similar server
- Restoring specific components (just email, just database, etc.)
- Recovering from local mailcow backups
#### Step 1: List Available Backups
```bash
cd /opt/mailcow-dockerized
# Run the restore script
./helper-scripts/backup_and_restore.sh restore
```
The script will prompt:
```
Backup location (absolute path, starting with /): /opt/mailcow-backups
```
#### Step 2: Select Backup
The script displays available backups:
```
Found project name mailcowdockerized
[ 1 ] - /opt/mailcow-backups/mailcow-2026-02-09-02-00-14/
[ 2 ] - /opt/mailcow-backups/mailcow-2026-02-10-02-00-08/
```
Enter the number of the backup to restore.
#### Step 3: Select Components
Choose what to restore:
```
[ 0 ] - all
[ 1 ] - Crypt data
[ 2 ] - Rspamd data
[ 3 ] - Mail directory (/var/vmail)
[ 4 ] - Redis DB
[ 5 ] - Postfix data
[ 6 ] - SQL DB
```
**Important**: The script will:
- Stop mailcow containers automatically
- Restore selected components
- Handle permissions correctly
- Restart containers when done
#### Example: Restore Only Email Data
```bash
cd /opt/mailcow-dockerized
./helper-scripts/backup_and_restore.sh restore
# When prompted:
# - Backup location: /opt/mailcow-backups
# - Select backup: 2 (most recent)
# - Select component: 3 (Mail directory)
```
#### Example: Restore Database Only
```bash
cd /opt/mailcow-dockerized
./helper-scripts/backup_and_restore.sh restore
# When prompted:
# - Backup location: /opt/mailcow-backups
# - Select backup: 2 (most recent)
# - Select component: 6 (SQL DB)
```
**Note**: For database restore, the script will modify `mailcow.conf` with the database credentials from the backup. Review the changes after restore.
### Method 2: Complete Server Rebuild (Kopia Restore)
Use this when recovering to a completely new server or when local backups are unavailable.
#### Step 1: Prepare New Server
```bash
# Update system
apt update && apt upgrade -y
# Install Docker
curl -fsSL https://get.docker.com | sh
systemctl enable docker
systemctl start docker
# Install Docker Compose
apt install docker-compose-plugin -y
# Install Kopia
curl -s https://kopia.io/signing-key | apt-key add -
echo "deb https://packages.kopia.io/apt/ stable main" | tee /etc/apt/sources.list.d/kopia.list
apt update
apt install kopia -y
# Create directory structure
mkdir -p /opt/mailcow-dockerized
mkdir -p /opt/mailcow-backups/database
```
#### Step 2: Restore Kopia Repository
```bash
# Connect to your offsite vault
# If vault is mounted:
kopia repository connect filesystem --path /mnt/vault/mailcow-backup
# If vault is remote:
kopia repository connect s3 --bucket=your-bucket --access-key=xxx --secret-access-key=xxx
# List available snapshots
kopia snapshot list --tags mailcow
```
#### Step 3: Restore Configuration
```bash
# Find and restore the config snapshot
kopia snapshot list --tags config
# Restore to the Mailcow directory
kopia restore <snapshot-id> /opt/mailcow-dockerized/
# Verify critical files
ls -la /opt/mailcow-dockerized/mailcow.conf
ls -la /opt/mailcow-dockerized/docker-compose.yml
```
#### Step 4: Restore Mailcow Backups Directory
```bash
# Restore the entire backup directory from Kopia
kopia snapshot list --tags tier1-backup
# Restore the most recent backup
kopia restore <snapshot-id> /opt/mailcow-backups/
# Verify backups were restored
ls -la /opt/mailcow-backups/
```
#### Step 5: Run Mailcow Native Restore
Now use mailcow's built-in restore script:
```bash
cd /opt/mailcow-dockerized
# Run the restore script
./helper-scripts/backup_and_restore.sh restore
# When prompted:
# - Backup location: /opt/mailcow-backups
# - Select the most recent backup
# - Select [ 0 ] - all (to restore everything)
```
The script will:
1. Stop all mailcow containers
2. Restore all components (vmail, mysql, redis, rspamd, postfix, crypt)
3. Update mailcow.conf with restored database credentials
4. Restart all containers
**Alternative: Manual Restore** (if you prefer more control)
```bash
cd /opt/mailcow-dockerized
# Start containers to create volumes
docker compose up -d --no-start
docker compose down
# Find the most recent backup directory
LATEST_BACKUP=$(ls -td /opt/mailcow-backups/mailcow-* | head -1)
echo "Restoring from: $LATEST_BACKUP"
# Extract each component manually
cd "$LATEST_BACKUP"
# Restore vmail (email data)
docker run --rm \
-v mailcowdockerized_vmail-vol:/backup \
-v "$PWD":/restore \
debian:bookworm-slim \
tar --use-compress-program='zstd -d' -xvf /restore/backup_vmail.tar.zst
# Restore MySQL
docker run --rm \
-v mailcowdockerized_mysql-vol:/backup \
-v "$PWD":/restore \
mariadb:10.11 \
tar --use-compress-program='zstd -d' -xvf /restore/backup_mysql.tar.zst
# Restore Redis
docker run --rm \
-v mailcowdockerized_redis-vol:/backup \
-v "$PWD":/restore \
debian:bookworm-slim \
tar --use-compress-program='zstd -d' -xvf /restore/backup_redis.tar.zst
# Restore other components similarly (rspamd, postfix, crypt)
# ...
# Copy mailcow.conf from backup
cp "$LATEST_BACKUP/mailcow.conf" /opt/mailcow-dockerized/mailcow.conf
```
#### Step 6: Start and Verify Mailcow
```bash
cd /opt/mailcow-dockerized
# Pull latest images (or use versions from backup if preferred)
docker compose pull
# Start all services
docker compose up -d
# Monitor logs
docker compose logs -f
```
#### Step 7: Post-Restore Verification
```bash
# Check container status
docker compose ps
# Test web interface
curl -I https://mail.yourdomain.com
# Check mail log
docker compose logs -f postfix-mailcow
# Verify database
docker compose exec mysql-mailcow mysql -u root -p$(grep DBROOT mailcow.conf | cut -d'=' -f2) -e "SHOW DATABASES;"
# Check email storage
docker compose exec dovecot-mailcow ls -lah /var/vmail/
```
### Scenario 2: Restore Individual Mailbox
To restore a single user's mailbox without affecting others:
#### Option A: Using Mailcow Backups (If Available)
```bash
cd /opt/mailcow-dockerized
# Temporarily mount the backup
BACKUP_DIR="/opt/mailcow-backups/mailcow-YYYY-MM-DD-HH-MM-SS"
# Extract just the vmail archive to a temporary location
mkdir -p /tmp/vmail-restore
cd "$BACKUP_DIR"
tar --use-compress-program='zstd -d' -xvf backup_vmail.tar.zst -C /tmp/vmail-restore
# Find the user's mailbox
# Structure: /tmp/vmail-restore/var/vmail/domain.com/user/
ls -la /tmp/vmail-restore/var/vmail/yourdomain.com/
# Copy specific mailbox
rsync -av /tmp/vmail-restore/var/vmail/yourdomain.com/user@domain.com/ \
/var/lib/docker/volumes/mailcowdockerized_vmail-vol/_data/yourdomain.com/user@domain.com/
# Fix permissions
docker run --rm \
-v mailcowdockerized_vmail-vol:/vmail \
debian:bookworm-slim \
chown -R 5000:5000 /vmail/yourdomain.com/user@domain.com/
# Cleanup
rm -rf /tmp/vmail-restore
# Restart Dovecot to recognize changes
docker compose restart dovecot-mailcow
```
#### Option B: Using Kopia Snapshot (If Local Backups Unavailable)
```bash
# Mount the vmail snapshot temporarily
mkdir -p /mnt/restore
kopia mount <vmail-snapshot-id> /mnt/restore
# Find the user's mailbox
# Structure: /mnt/restore/domain.com/user/
ls -la /mnt/restore/yourdomain.com/
# Copy specific mailbox
rsync -av /mnt/restore/yourdomain.com/user@domain.com/ \
/var/lib/docker/volumes/mailcowdockerized_vmail-vol/_data/yourdomain.com/user@domain.com/
# Fix permissions
chown -R 5000:5000 /var/lib/docker/volumes/mailcowdockerized_vmail-vol/_data/yourdomain.com/user@domain.com/
# Unmount
kopia unmount /mnt/restore
# Restart Dovecot to recognize changes
docker compose restart dovecot-mailcow
```
### Scenario 3: Database Recovery Only
If only the database is corrupted but email data is intact:
#### Option A: Using Mailcow Native Restore (Recommended)
```bash
cd /opt/mailcow-dockerized
# Run the restore script
./helper-scripts/backup_and_restore.sh restore
# When prompted:
# - Backup location: /opt/mailcow-backups
# - Select the most recent backup
# - Select [ 6 ] - SQL DB (database only)
```
The script will:
1. Stop mailcow
2. Restore the MySQL database from the mariabackup archive
3. Update mailcow.conf with the restored database credentials
4. Restart mailcow
#### Option B: Manual Database Restore from Kopia
If local backups are unavailable:
```bash
cd /opt/mailcow-dockerized
# Stop Mailcow
docker compose down
# Start only MySQL
docker compose up -d mysql-mailcow
# Wait for MySQL
sleep 30
# Restore from Kopia database dump
kopia snapshot list --tags database
kopia restore <snapshot-id> /tmp/db-restore/
# Import the dump
LATEST_DUMP=$(ls -t /tmp/db-restore/mailcow_*.sql | head -1)
docker compose exec -T mysql-mailcow mysql -u root -p$(grep DBROOT mailcow.conf | cut -d'=' -f2) < "$LATEST_DUMP"
# Start all services
docker compose down
docker compose up -d
# Verify
docker compose logs -f
```
### Scenario 4: Configuration Recovery Only
If you only need to restore configuration files:
#### Option A: From Mailcow Backup
```bash
# Find the most recent backup
LATEST_BACKUP=$(ls -td /opt/mailcow-backups/mailcow-* | head -1)
# Stop Mailcow
cd /opt/mailcow-dockerized
docker compose down
# Backup current config (just in case)
cp mailcow.conf mailcow.conf.pre-restore
cp docker-compose.yml docker-compose.yml.pre-restore
# Restore mailcow.conf from backup
cp "$LATEST_BACKUP/mailcow.conf" ./mailcow.conf
# If you also need other config files from data/conf/,
# you would need to extract them from the backup archives
# Restart
docker compose up -d
```
#### Option B: From Kopia Snapshot
```bash
# Restore config snapshot to temporary location
kopia restore <config-snapshot-id> /tmp/mailcow-restore/
# Stop Mailcow
cd /opt/mailcow-dockerized
docker compose down
# Backup current config (just in case)
cp mailcow.conf mailcow.conf.pre-restore
cp docker-compose.yml docker-compose.yml.pre-restore
# Restore specific files
cp /tmp/mailcow-restore/mailcow.conf ./
cp /tmp/mailcow-restore/docker-compose.yml ./
cp -r /tmp/mailcow-restore/data/conf/* ./data/conf/
# Restart
docker compose up -d
```
## Verification and Testing
### Regular Backup Verification
Perform monthly restore tests to ensure backups are valid:
```bash
# Test restore to temporary location
mkdir -p /tmp/backup-test
kopia snapshot list --tags mailcow
kopia restore <snapshot-id> /tmp/backup-test/
# Verify files exist and are readable
ls -lah /tmp/backup-test/
cat /tmp/backup-test/mailcow.conf
# Cleanup
rm -rf /tmp/backup-test/
```
### Backup Monitoring Script
Create `/opt/scripts/check-mailcow-backup.sh`:
```bash
#!/bin/bash
# Check last backup age
LAST_BACKUP=$(kopia snapshot list --tags mailcow --json | jq -r '.[0].startTime')
LAST_BACKUP_EPOCH=$(date -d "$LAST_BACKUP" +%s)
NOW=$(date +%s)
AGE_HOURS=$(( ($NOW - $LAST_BACKUP_EPOCH) / 3600 ))
if [ $AGE_HOURS -gt 26 ]; then
echo "WARNING: Last Mailcow backup is $AGE_HOURS hours old"
# Send alert (email, Slack, etc.)
exit 1
else
echo "OK: Last backup $AGE_HOURS hours ago"
fi
```
## Disaster Recovery Checklist
When disaster strikes, follow this checklist:
- [ ] Confirm scope of failure (server, storage, specific component)
- [ ] Gather server information (hostname, IP, DNS records)
- [ ] Access offsite backup vault
- [ ] Provision new server (if needed)
- [ ] Install Docker and dependencies
- [ ] Connect to Kopia repository
- [ ] Restore configurations first
- [ ] Restore database
- [ ] Restore email data
- [ ] Start services and verify
- [ ] Test email sending/receiving
- [ ] Verify webmail access
- [ ] Check DNS records and update if needed
- [ ] Document any issues encountered
- [ ] Update recovery procedures based on experience
## Important Notes
1. **DNS**: Keep DNS records documented separately. Recovery includes updating DNS if server IP changes.
2. **SSL Certificates**: Let's Encrypt certificates are in the backup but may need renewal. Mailcow will handle this automatically.
3. **Permissions**: Docker volumes have specific UID/GID requirements:
- vmail: `5000:5000`
- mysql: `999:999`
4. **Testing**: Always test recovery procedures in a lab environment before trusting them in production.
5. **Documentation**: Keep this guide and server details in a separate location (printed copy, password manager, etc.).
6. **Retention Policy**: Review Kopia retention settings periodically to balance storage costs with recovery needs.
## Backup Architecture Notes
### Why Two Backup Layers?
**Mailcow Native Backups** (Tier 1):
- ✅ Component-aware (knows about mailcow's structure)
- ✅ Uses mariabackup for consistent MySQL hot backups
- ✅ Fast, selective restore (can restore just one component)
- ✅ Architecture-aware (handles x86/ARM differences)
- ❌ No deduplication (full copies each time)
- ❌ Limited to local storage initially
**Kopia Snapshots** (Tier 2):
- ✅ Deduplication and compression
- ✅ Efficient offsite replication to vaults
- ✅ Point-in-time recovery across multiple versions
- ✅ Disaster recovery to completely new infrastructure
- ❌ Less component-aware (treats as files)
- ❌ Slower for granular component restore
### Storage Efficiency
Using this two-tier approach:
- **Local**: Mailcow creates ~7 days of native backups (may be large, but short retention)
- **Offsite**: Kopia deduplicates these backups for long-term vault storage (much smaller)
Example storage calculation (10GB mailbox):
- Local: 7 days × 10GB = ~70GB (before compression)
- Kopia (offsite): First backup ~10GB, subsequent backups only store changes (might be <1GB/day after dedup)
### Compression Formats
Mailcow's script creates `.tar.zst` (Zstandard) or `.tar.gz` (gzip) files:
- **Zstandard** (modern): Better compression ratio, faster (recommended)
- **Gzip** (legacy): Wider compatibility with older systems
Verify your backup compression:
```bash
ls -lh /opt/mailcow-backups/mailcow-*/
# Look for .tar.zst (preferred) or .tar.gz
```
### Cross-Architecture Considerations
**Important for ARM/x86 Migration**:
Mailcow's backup script is architecture-aware. When restoring:
- **Rspamd data** cannot be restored across different architectures (x86 ↔ ARM)
- **All other components** (vmail, mysql, redis, postfix, crypt) are architecture-independent
If migrating between architectures:
```bash
# Restore everything EXCEPT rspamd
# Select components individually: vmail, mysql, redis, postfix, crypt
# Skip rspamd - it will rebuild its learning database over time
```
### Testing Your Backups
**Monthly Test Protocol**:
1. **Verify local backups exist**:
```bash
ls -lh /opt/mailcow-backups/
# Should see recent dated directories
```
2. **Verify Kopia snapshots**:
```bash
kopia snapshot list --tags mailcow
# Should see recent snapshots
```
3. **Test restore in lab** (recommended quarterly):
- Spin up a test VM
- Restore from Kopia
- Run mailcow native restore
- Verify email delivery and webmail access
## Additional Resources
- [Mailcow Official Backup Documentation](https://docs.mailcow.email/backup_restore/b_n_r-backup/)
- [Kopia Documentation](https://kopia.io/docs/)
- [Docker Volume Backup Best Practices](https://docs.docker.com/storage/volumes/#back-up-restore-or-migrate-data-volumes)
## Revision History
| Date | Version | Changes |
|------|---------|---------|
| 2026-02-10 | 1.1 | Integrated mailcow native backup_and_restore.sh script as primary backup method |
| 2026-02-10 | 1.0 | Initial documentation |
---
**Last Updated**: February 10, 2026
**Maintained By**: System Administrator
**Review Schedule**: Quarterly

1151
Vault-Grimoire/Backups/Nextcloud-Backup.md Normal file
View file

File diff suppressed because it is too large Load diff

19
Vault-Grimoire/Backups/Services-Backup.md Normal file
View file

@ -0,0 +1,19 @@
---
title: Services Backup
description:
published: true
date: 2026-02-20T04:08:15.923Z
tags:
editor: markdown
dateCreated: 2026-02-05T21:28:23.152Z
---
- [Mailcow](/backup-mailcow)
- [Immich](/immich_backup)
- [Nextcloud](/nextcloud_backup)
- kopia
- forgejo
- bitwarden
- wiki
- journalv

567
Vault-Grimoire/Backups/Wiki-Backup.md Normal file
View file

@ -0,0 +1,567 @@
---
title: Wikijs Backup
description: Backup Wikijs
published: true
date: 2026-02-23T04:35:32.870Z
tags:
editor: markdown
dateCreated: 2026-02-23T04:35:24.121Z
---
# Wiki.js Backup & Recovery
**Service:** Wiki.js (Netgrimoire)
**Stack:** Docker Compose — Wiki.js + PostgreSQL
**Backup Targets:** PostgreSQL database dump, Git content repository, Docker Compose config
**Backup Destinations:** Local vault path → Kopia → offsite vaults
---
## Overview
Wiki.js data lives in two separate places that must be backed up independently:
**PostgreSQL database** — stores page metadata, navigation, user accounts, permissions, page history, assets, and all configuration. This is the critical component for a portable restore. Without it, a new instance has no knowledge of your wiki structure.
**Git content repository** — stores the actual page content in markdown files, synced from Forgejo. This is already mirrored on the VAULT SSD at `/vault/repos/wiki/`. It is inherently redundant as long as Forgejo is healthy, but is included in backups for completeness and offline portability.
**Docker Compose config** — the `docker-compose.yml` and `.env` files needed to recreate the stack.
---
## What Gets Backed Up
| Component | Location | Method | Critical? |
|---|---|---|---|
| PostgreSQL database | Docker volume | `pg_dump` → SQL file | Yes — primary restore target |
| Git content repo | `/vault/repos/wiki/` | Already on VAULT SSD | Yes — page content |
| Docker Compose files | `/opt/stacks/wikijs/` | rsync copy | Yes — stack config |
| Wiki.js data volume | Docker volume | Optional rsync | No — DB + Git covers this |
---
## Backup Strategy
### Tier 1 — Daily Dump to Vault Path
A script runs daily via systemd timer. It produces a portable `pg_dump` SQL file written to `/vault/backups/wiki/`. These local dumps are retained for 14 days.
**Key choices:**
- `--format=plain` — plain SQL, portable to any PostgreSQL version and any host
- `--no-owner` — strips role ownership, so the dump restores cleanly on a new instance with a different postgres user (critical for Pocket Grimoire restores)
- `--no-acl` — strips GRANT/REVOKE statements for the same reason
- No application downtime required — PostgreSQL handles consistent dumps natively
### Tier 2 — Kopia Snapshot to Offsite Vaults
After the daily dump completes, Kopia snapshots the entire `/vault/backups/wiki/` directory and replicates to your offsite vaults. Kopia deduplication means only changed blocks are transferred after the first run.
---
## Setup
### Step 0 — Confirm Kopia Repository Exists
If Kopia is not yet initialized on this host, initialize it first. If you already initialized Kopia for Mailcow or another service, skip this step — all services share the same Kopia repository.
```bash
# Check if repository already exists
kopia repository status
# If not initialized, create it against your vault path
kopia repository create filesystem --path=/vault/kopia
# Connect on subsequent logins if disconnected
kopia repository connect filesystem --path=/vault/kopia
```
### Step 1 — Create Backup Directories
```bash
sudo mkdir -p /vault/backups/wiki
sudo chown $(whoami):$(whoami) /vault/backups/wiki
```
### Step 2 — Create the Backup Script
```bash
sudo nano /usr/local/sbin/wikijs-backup.sh
```
```bash
#!/usr/bin/env bash
# wikijs-backup.sh — Daily Wiki.js backup: pg_dump + git repo + config
# Writes to /vault/backups/wiki/, then snapshots with Kopia
set -euo pipefail
# ── Configuration ─────────────────────────────────────────────────────────────
BACKUP_DIR="/vault/backups/wiki"
DATE=$(date +%Y%m%d_%H%M%S)
CONTAINER_DB="wikijs_db" # Adjust to your actual container name
PG_USER="wikijs"
PG_DB="wikijs"
WIKI_STACK_DIR="/opt/stacks/wikijs" # Location of docker-compose.yml and .env
GIT_REPO_DIR="/vault/repos/wiki" # Git content mirror (already on vault SSD)
RETAIN_DAYS=14 # Local dump retention
LOG="/var/log/wikijs-backup.log"
touch "$LOG"
log() { echo "$(date -Is) $*" | tee -a "$LOG"; }
# ── Step 1: PostgreSQL dump ────────────────────────────────────────────────────
log "Starting Wiki.js PostgreSQL dump..."
docker exec "$CONTAINER_DB" pg_dump \
-U "$PG_USER" \
"$PG_DB" \
--format=plain \
--no-owner \
--no-acl \
> "${BACKUP_DIR}/wikijs-db-${DATE}.sql"
gzip "${BACKUP_DIR}/wikijs-db-${DATE}.sql"
log "PostgreSQL dump complete: wikijs-db-${DATE}.sql.gz"
# ── Step 2: Docker Compose config backup ──────────────────────────────────────
log "Backing up Docker Compose config..."
CONFIG_BACKUP="${BACKUP_DIR}/wikijs-config-${DATE}.tar.gz"
tar -czf "$CONFIG_BACKUP" \
-C "$(dirname "$WIKI_STACK_DIR")" \
"$(basename "$WIKI_STACK_DIR")"
log "Config backup complete: wikijs-config-${DATE}.tar.gz"
# ── Step 3: Git repo snapshot (content mirror) ────────────────────────────────
# The git repo lives on the VAULT SSD and is already versioned.
# We record the current HEAD commit for reference.
if [ -d "${GIT_REPO_DIR}/.git" ]; then
GIT_HEAD=$(git -C "$GIT_REPO_DIR" rev-parse HEAD 2>/dev/null || echo "unknown")
echo "Git HEAD at backup time: ${GIT_HEAD}" \
> "${BACKUP_DIR}/wikijs-git-ref-${DATE}.txt"
log "Git content repo HEAD: ${GIT_HEAD}"
else
log "WARNING: Git repo not found at ${GIT_REPO_DIR} — skipping git ref"
fi
# ── Step 4: Cleanup old local dumps ───────────────────────────────────────────
log "Cleaning up dumps older than ${RETAIN_DAYS} days..."
find "$BACKUP_DIR" -name "wikijs-db-*.sql.gz" -mtime +"$RETAIN_DAYS" -delete
find "$BACKUP_DIR" -name "wikijs-config-*.tar.gz" -mtime +"$RETAIN_DAYS" -delete
find "$BACKUP_DIR" -name "wikijs-git-ref-*.txt" -mtime +"$RETAIN_DAYS" -delete
# ── Step 5: Kopia snapshot ────────────────────────────────────────────────────
log "Running Kopia snapshot of /vault/backups/wiki/..."
kopia snapshot create "$BACKUP_DIR" \
--tags "service:wikijs,host:$(hostname -s)"
log "Kopia snapshot complete."
# ── Done ──────────────────────────────────────────────────────────────────────
log "Wiki.js backup finished successfully."
```
```bash
sudo chmod +x /usr/local/sbin/wikijs-backup.sh
```
### Step 3 — Create systemd Service and Timer
```bash
sudo nano /etc/systemd/system/wikijs-backup.service
```
```ini
[Unit]
Description=Wiki.js daily backup (pg_dump + config + Kopia snapshot)
After=docker.service
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/wikijs-backup.sh
```
```bash
sudo nano /etc/systemd/system/wikijs-backup.timer
```
```ini
[Unit]
Description=Run Wiki.js backup daily at 02:00
[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true
[Install]
WantedBy=timers.target
```
```bash
sudo systemctl daemon-reload
sudo systemctl enable wikijs-backup.timer
sudo systemctl start wikijs-backup.timer
# Verify
systemctl list-timers | grep wikijs
```
### Step 4 — Configure Kopia Retention Policy
```bash
# Set retention policy for wiki backups
kopia policy set /vault/backups/wiki \
--keep-daily 14 \
--keep-weekly 8 \
--keep-monthly 12 \
--compression zstd
# Verify policy
kopia policy show /vault/backups/wiki
```
### Step 5 — Test the Backup
```bash
# Run manually first time
sudo /usr/local/sbin/wikijs-backup.sh
# Verify output
ls -lh /vault/backups/wiki/
# Should show: wikijs-db-YYYYMMDD_HHMMSS.sql.gz
# wikijs-config-YYYYMMDD_HHMMSS.tar.gz
# wikijs-git-ref-YYYYMMDD_HHMMSS.txt
# Verify Kopia snapshot was created
kopia snapshot list /vault/backups/wiki
# Check backup log
tail -n 30 /var/log/wikijs-backup.log
```
---
## Verifying Backups
### Check dump is readable
```bash
# Inspect the SQL dump without extracting
zcat /vault/backups/wiki/wikijs-db-YYYYMMDD_HHMMSS.sql.gz | head -50
# Should show PostgreSQL header, version info, and CREATE TABLE statements
```
### Verify Kopia snapshots
```bash
# List recent snapshots
kopia snapshot list /vault/backups/wiki
# Show snapshot details
kopia snapshot list /vault/backups/wiki --all
# Verify snapshot integrity
kopia snapshot verify
```
### Test restore to a temporary database (non-destructive)
```bash
# Start a temporary Postgres container
docker run --rm -d \
--name wikijs-restore-test \
-e POSTGRES_USER=wikijs \
-e POSTGRES_PASSWORD=testpassword \
-e POSTGRES_DB=wikijs_test \
postgres:16-alpine
# Wait for Postgres to be ready
sleep 5
# Restore dump into test container
zcat /vault/backups/wiki/wikijs-db-YYYYMMDD_HHMMSS.sql.gz | \
docker exec -i wikijs-restore-test psql -U wikijs -d wikijs_test
# Verify tables exist
docker exec wikijs-restore-test psql -U wikijs -d wikijs_test -c "\dt"
# Expected output: List of tables (pages, users, pageHistory, assets, etc.)
# Cleanup test container
docker stop wikijs-restore-test
```
---
## Recovery Procedures
### Scenario A — Restore to a New Wiki.js Instance (Any Host)
This covers full disaster recovery to a fresh server, including Pocket Grimoire.
**Requirements on the destination host:**
- Docker and Docker Compose installed
- A `docker-compose.yml` and `.env` ready (from backup or Pocket Grimoire stack)
- Sufficient disk space
**Step 1: Locate the backup**
```bash
# On Netgrimoire, find the dump to restore
ls -lh /vault/backups/wiki/
# Or restore from Kopia
kopia snapshot list /vault/backups/wiki
kopia restore SNAPSHOT_ID /tmp/wiki-restore/
ls /tmp/wiki-restore/
```
**Step 2: Copy dump to the destination host**
```bash
# From Netgrimoire, copy to the destination server
scp /vault/backups/wiki/wikijs-db-YYYYMMDD_HHMMSS.sql.gz \
user@destination-host:/tmp/
# Or to Pocket Grimoire
scp /vault/backups/wiki/wikijs-db-YYYYMMDD_HHMMSS.sql.gz \
user@pocket-grimoire.local:/tmp/
```
**Step 3: Start the database container only**
On the destination host, start just the database — do not start Wiki.js yet:
```bash
cd /srv/pocket-grimoire/stacks/wikijs # Adjust path as needed
# Start only the database container
docker compose up -d db
# Wait for healthy status
docker compose ps
# db should show: healthy
```
**Step 4: Restore the dump**
```bash
# Restore the dump into the running database container
zcat /tmp/wikijs-db-YYYYMMDD_HHMMSS.sql.gz | \
docker exec -i pocketgrimoire_db psql \
-U wikijs \
-d wikijs
# Verify tables restored
docker exec pocketgrimoire_db psql -U wikijs -d wikijs -c "\dt"
```
**Step 5: Start Wiki.js**
```bash
docker compose up -d
# Watch startup logs
docker logs -f pocketgrimoire_wikijs
# Wait for: "HTTP Server started successfully"
```
**Step 6: Verify**
Open `http://pocket-grimoire.local:3000` and confirm:
- Pages load correctly
- Navigation structure is intact
- User accounts are present (if you had multiple users)
**Step 7: Re-sync Git content (if needed)**
The database knows the page structure, but if the Git content repo isn't present on the new host, import it:
```bash
# In Wiki.js admin panel:
# Administration → Storage → Git
# Click "Force Sync" or "Import Content"
# Or copy the repo from VAULT SSD
rsync -avP /vault/repos/wiki/ /srv/pocket-grimoire/repos/wiki/
```
---
### Scenario B — Restore on Existing Netgrimoire Instance
Use this when the Wiki.js database is corrupted but the host is otherwise healthy.
**Step 1: Stop Wiki.js (leave database running)**
```bash
cd /opt/stacks/wikijs
docker compose stop wikijs
```
**Step 2: Drop and recreate the database**
```bash
docker exec -it wikijs_db psql -U postgres -c "DROP DATABASE wikijs;"
docker exec -it wikijs_db psql -U postgres -c "CREATE DATABASE wikijs OWNER wikijs;"
```
**Step 3: Restore**
```bash
zcat /vault/backups/wiki/wikijs-db-YYYYMMDD_HHMMSS.sql.gz | \
docker exec -i wikijs_db psql -U wikijs -d wikijs
```
**Step 4: Restart Wiki.js**
```bash
docker compose start wikijs
docker logs -f wikijs
```
---
### Scenario C — Restore Config Only
If the stack config was lost but the database volume is intact:
```bash
# Extract config from backup
tar -xzf /vault/backups/wiki/wikijs-config-YYYYMMDD_HHMMSS.tar.gz \
-C /opt/stacks/
# Verify
ls /opt/stacks/wikijs/
# Should show: docker-compose.yml .env
# Restart stack
cd /opt/stacks/wikijs
docker compose up -d
```
---
### Restore from Kopia (Offsite)
When local vault files are unavailable, restore the backup directory from Kopia first:
```bash
# List available snapshots
kopia snapshot list /vault/backups/wiki
# Restore snapshot to temp directory
kopia restore SNAPSHOT_ID /tmp/wiki-restore/
# Then proceed with the appropriate scenario above
# using files from /tmp/wiki-restore/ instead of /vault/backups/wiki/
```
---
## Pocket Grimoire Specifics
When restoring to Pocket Grimoire, note the following differences from a full Netgrimoire instance:
**Container names** differ — use `pocketgrimoire_db` instead of `wikijs_db`.
**Stack path** is `/srv/pocket-grimoire/stacks/wikijs/` instead of `/opt/stacks/wikijs/`.
**The database is already initialized** when Pocket Grimoire is first set up. Restoring a Netgrimoire dump overwrites it entirely, which is the intended behavior — Pocket Grimoire becomes a mirror of Netgrimoire's wiki state.
**Git content repo** is located at `/srv/pocket-grimoire/repos/wiki/` and is populated via the sync script (`pocketgrimoire-sync.sh`). A database restore alone is sufficient if the Git repo is already in place.
**Recommended restore workflow for Pocket Grimoire:**
```bash
# 1. Copy dump from VAULT SSD (already available on Pocket Grimoire)
ls /srv/vaultpg/backups/wiki/
# 2. Start db container only
cd /srv/pocket-grimoire/stacks/wikijs && docker compose up -d db
# 3. Restore
zcat /srv/vaultpg/backups/wiki/wikijs-db-LATEST.sql.gz | \
docker exec -i pocketgrimoire_db psql -U wikijs -d wikijs
# 4. Start full stack
docker compose up -d
```
Because the VAULT SSD is always connected to Pocket Grimoire, no file transfer is needed — the dumps are already there.
---
## Monitoring & Alerts
Add the following to your existing ntfy/monitoring setup to alert on backup failures. Wrap the backup script call in an error trap:
```bash
# Add to wikijs-backup.sh after set -euo pipefail:
NTFY_URL="https://ntfy.YOUR_DOMAIN/wikijs-backup"
on_error() {
curl -fsS -X POST "$NTFY_URL" \
-H "Title: Wiki.js backup FAILED ($(hostname -s))" \
-H "Priority: high" \
-H "Tags: rotating_light" \
-d "Backup failed at $(date -Is). Check /var/log/wikijs-backup.log"
}
trap on_error ERR
```
### Check backup age manually
```bash
# Find most recent dump
ls -lt /vault/backups/wiki/wikijs-db-*.sql.gz | head -3
# Check Kopia last snapshot time
kopia snapshot list /vault/backups/wiki | tail -5
```
---
## Quick Reference
```bash
# Run backup manually
sudo /usr/local/sbin/wikijs-backup.sh
# Watch backup log
tail -f /var/log/wikijs-backup.log
# Check timer status
systemctl status wikijs-backup.timer
# List local dumps
ls -lh /vault/backups/wiki/
# List Kopia snapshots
kopia snapshot list /vault/backups/wiki
# Restore dump (generic)
zcat /vault/backups/wiki/wikijs-db-YYYYMMDD_HHMMSS.sql.gz | \
docker exec -i CONTAINER_NAME psql -U wikijs -d wikijs
# Test dump is readable
zcat /vault/backups/wiki/wikijs-db-YYYYMMDD_HHMMSS.sql.gz | head -50
```
---
## Revision History
| Version | Date | Notes |
|---|---|---|
| 1.0 | 2026-02-22 | Initial release — pg_dump + Kopia + Pocket Grimoire restore procedures |

940
Vault-Grimoire/Kopia/Kopia-Overview.md Normal file
View file

@ -0,0 +1,940 @@
---
title: Setting Up Kopia
description:
published: true
date: 2026-02-20T04:27:59.823Z
tags:
editor: markdown
dateCreated: 2026-01-23T22:14:17.009Z
---
# Kopia Backup System Documentation
## Overview
This system implements a two-tier backup strategy using **two separate Kopia Server instances**:
1. **Primary Repository** (`/srv/vault/kopia_repository`) - Full backups of all clients, served on port 51515
2. **Vault Repository** (`/srv/vault/backup`) - Targeted critical data backups, served on port 51516, replicated offsite via ZFS send/receive
The Vault repository sits on its own ZFS dataset to enable clean replication to offsite Pi systems. Running two separate Kopia servers allows independent management of each repository while maintaining the same HTTPS-based client connection model for both.
---
## Architecture
```
Clients (docker2, cindy's desktop, etc.)
↓
├─→ Primary Backup → Kopia Server Primary (port 51515)
│ → /srv/vault/kopia_repository (all data)
│
└─→ Vault Backup → Kopia Server Vault (port 51516)
→ /srv/vault/backup (critical data only)
↓
ZFS Send/Receive
↓
┌───────┴───────┐
↓ ↓
Pi Vault 1 Pi Vault 2
(offsite) (offsite)
```
---
## Initial Setup on ZNAS
### Prerequisites
- Docker installed on ZNAS
- ZFS pool available
### 1. Create ZFS Datasets
```bash
# Primary repository dataset (if not already created)
zfs create -o mountpoint=/srv/vault zpool/vault
zfs create zpool/vault/kopia_repository
# Vault repository dataset (for offsite replication)
zfs create zpool/vault/backup
```
### 2. Install Kopia Servers (Docker)
We run **two separate Kopia Server containers** - one for primary backups, one for vault backups.
```bash
# Primary repository server (port 51515)
docker run -d \
--name kopia-server-primary \
--restart unless-stopped \
-p 51515:51515 \
-v /srv/vault/kopia_repository:/app/repository \
-v /srv/vault/config-primary:/app/config \
-v /srv/vault/logs-primary:/app/logs \
kopia/kopia:latest server start \
--address=0.0.0.0:51515 \
--tls-generate-cert
# Vault repository server (port 51516)
docker run -d \
--name kopia-server-vault \
--restart unless-stopped \
-p 51516:51516 \
-v /srv/vault/backup:/app/repository \
-v /srv/vault/config-vault:/app/config \
-v /srv/vault/logs-vault:/app/logs \
kopia/kopia:latest server start \
--address=0.0.0.0:51516 \
--tls-generate-cert
```
**Get the certificate fingerprints:**
```bash
# Primary server fingerprint
docker exec kopia-server-primary kopia server status
# Vault server fingerprint
docker exec kopia-server-vault kopia server status
```
**Note:** Record both certificate fingerprints - you'll need them for client connections.
- **Primary server cert SHA256:** `696a4999f594b5273a174fd7cab677d8dd1628f9b9d27e557daa87103ee064b2`
- **Vault server cert SHA256:** *(get from command above)*
### 3. Create Kopia Repositories
Each server manages its own repository. These are created during first server start, but you can initialize them manually if needed.
```bash
# Primary repository (usually created via GUI on first use)
docker exec -it kopia-server-primary kopia repository create filesystem \
--path=/app/repository \
--description="Primary backup repository"
# Vault repository
docker exec -it kopia-server-vault kopia repository create filesystem \
--path=/app/repository \
--description="Vault backup repository for offsite replication"
```
**Note:** If you created the primary repository via the Kopia UI, you don't need to run the first command.
### 4. Create User Accounts
Create users on each server separately.
**Primary repository users:**
```bash
# Enter primary server container
docker exec -it kopia-server-primary /bin/sh
# Create users
kopia server users add admin@docker2
kopia server users add cindy@DESKTOP-QLSVD8P
# Password for cindy: LucyDog123
# Exit container
exit
```
**Vault repository users:**
```bash
# Enter vault server container
docker exec -it kopia-server-vault /bin/sh
# Create users
kopia server users add admin@docker2-vault
kopia server users add cindy@DESKTOP-QLSVD8P-vault
# Use same passwords or different based on security requirements
# Exit container
exit
```
---
## Client Configuration
### Linux Client (docker2)
#### Primary Backup Setup
1. **Install Kopia**
```bash
# Download and install kopia .deb package
wget https://github.com/kopia/kopia/releases/download/v0.XX.X/kopia_0.XX.X_amd64.deb
sudo dpkg -i kopia_0.XX.X_amd64.deb
```
2. **Remove old repository (if exists)**
```bash
sudo kopia repository disconnect || true
sudo rm -rf /root/.config/kopia
```
3. **Connect to primary repository**
```bash
sudo kopia repository connect server \
--url=https://192.168.5.10:51515 \
--override-username=admin@docker2 \
--server-cert-fingerprint=696a4999f594b5273a174fd7cab677d8dd1628f9b9d27e557daa87103ee064b2
```
4. **Create initial snapshot**
```bash
sudo kopia snapshot create /DockerVol/
```
5. **Set up cron job for primary backups**
```bash
sudo crontab -e
# Add this line (runs every 3 hours)
*/180 * * * * /usr/bin/kopia snapshot create /DockerVol >> /var/log/kopia-primary-cron.log 2>&1
```
#### Vault Backup Setup (Critical Data)
1. **Create secondary kopia config directory**
```bash
sudo mkdir -p /root/.config/kopia-vault
```
2. **Connect to vault repository**
```bash
sudo kopia --config-file=/root/.config/kopia-vault/repository.config \
repository connect server \
--url=https://192.168.5.10:51516 \
--override-username=admin@docker2-vault \
--server-cert-fingerprint=<VAULT_SERVER_CERT_FINGERPRINT>
```
**Note:** Replace `<VAULT_SERVER_CERT_FINGERPRINT>` with the actual fingerprint from the vault server (see setup section).
3. **Create vault backup script**
```bash
sudo nano /usr/local/bin/kopia-vault-backup.sh
```
Add this content:
```bash
#!/bin/bash
# Kopia Vault Backup Script
# Backs up critical data to vault repository for offsite replication
KOPIA_CONFIG="/root/.config/kopia-vault/repository.config"
LOG_FILE="/var/log/kopia-vault-cron.log"
# Add your critical directories here
VAULT_DIRS=(
"/DockerVol/critical-app1"
"/DockerVol/critical-app2"
"/home/admin/documents"
)
echo "=== Vault backup started at $(date) ===" >> "$LOG_FILE"
for dir in "${VAULT_DIRS[@]}"; do
if [ -d "$dir" ]; then
echo "Backing up: $dir" >> "$LOG_FILE"
/usr/bin/kopia --config-file="$KOPIA_CONFIG" snapshot create "$dir" >> "$LOG_FILE" 2>&1
else
echo "Directory not found: $dir" >> "$LOG_FILE"
fi
done
echo "=== Vault backup completed at $(date) ===" >> "$LOG_FILE"
echo "" >> "$LOG_FILE"
```
4. **Make script executable**
```bash
sudo chmod +x /usr/local/bin/kopia-vault-backup.sh
```
5. **Set up cron job for vault backups**
```bash
sudo crontab -e
# Add this line (runs daily at 3 AM)
0 3 * * * /usr/local/bin/kopia-vault-backup.sh
```
---
### Windows Client (Cindy's Desktop)
#### Primary Backup Setup
1. **Install Kopia**
```powershell
# Using winget
winget install kopia
```
2. **Connect to primary repository**
```powershell
kopia repository connect server `
--url=https://192.168.5.10:51515 `
--override-username=cindy@DESKTOP-QLSVD8P `
--server-cert-fingerprint=696a4999f594b5273a174fd7cab677d8dd1628f9b9d27e557daa87103ee064b2
```
3. **Create initial snapshot**
```powershell
kopia snapshot create C:\Users\cindy
```
4. **Set exclusion policy**
```powershell
kopia policy set `
--global `
--add-ignore "**\AppData\Local\Temp\**" `
--add-ignore "**\AppData\Local\Packages\**"
```
5. **Create primary backup script**
```powershell
# Create scripts folder
New-Item -ItemType Directory -Force -Path C:\Scripts
# Create backup script
New-Item -ItemType File -Path C:\Scripts\kopia-primary-nightly.ps1
```
Add this content to `C:\Scripts\kopia-primary-nightly.ps1`:
```powershell
# Kopia Primary Backup Script
# Repository password
$env:KOPIA_PASSWORD = "LucyDog123"
# Run backup with logging
kopia snapshot create C:\Users\cindy `
--progress `
| Tee-Object -FilePath C:\Logs\kopia-primary.log -Append
# Log completion
Add-Content -Path C:\Logs\kopia-primary.log -Value "Backup completed at $(Get-Date)"
Add-Content -Path C:\Logs\kopia-primary.log -Value "---"
```
6. **Secure the script**
- Right-click `C:\Scripts\kopia-primary-nightly.ps1` → Properties → Security
- Ensure only Cindy's user account has read access
7. **Create scheduled task for primary backup**
- Press `Win + R` → type `taskschd.msc`
- Click "Create Task" (not "Basic Task")
**General tab:**
- Name: `Kopia Primary Nightly Backup`
- ✔ Run whether user is logged on or not
- ✔ Run with highest privileges
- Configure for: Windows 10/11
**Triggers tab:**
- New → Daily at 2:00 AM
- ✔ Enabled
**Actions tab:**
- Program: `powershell.exe`
- Arguments: `-ExecutionPolicy Bypass -File C:\Scripts\kopia-primary-nightly.ps1`
- Start in: `C:\Scripts`
**Conditions tab:**
- ✔ Wake the computer to run this task
- ✔ Start only if on AC power (recommended for laptops)
**Settings tab:**
- ✔ Allow task to be run on demand
- ✔ Run task as soon as possible after scheduled start is missed
- ❌ Stop the task if it runs longer than...
**Note:** When creating the task, use PIN (not Windows password) when prompted. For scheduled task credential: use password Harvey123= (MS account password)
#### Vault Backup Setup (Critical Data)
1. **Create vault config directory**
```powershell
New-Item -ItemType Directory -Force -Path C:\Users\cindy\.config\kopia-vault
```
2. **Connect to vault repository**
```powershell
kopia --config-file="C:\Users\cindy\.config\kopia-vault\repository.config" `
repository connect server `
--url=https://192.168.5.10:51516 `
--override-username=cindy@DESKTOP-QLSVD8P-vault `
--server-cert-fingerprint=<VAULT_SERVER_CERT_FINGERPRINT>
```
**Note:** Replace `<VAULT_SERVER_CERT_FINGERPRINT>` with the actual fingerprint from the vault server.
3. **Create vault backup script**
```powershell
New-Item -ItemType File -Path C:\Scripts\kopia-vault-nightly.ps1
```
Add this content to `C:\Scripts\kopia-vault-nightly.ps1`:
```powershell
# Kopia Vault Backup Script
# Backs up critical data to vault repository for offsite replication
$env:KOPIA_PASSWORD = "LucyDog123"
$KOPIA_CONFIG = "C:\Users\cindy\.config\kopia-vault\repository.config"
# Define critical directories to back up
$VaultDirs = @(
"C:\Users\cindy\Documents",
"C:\Users\cindy\Pictures",
"C:\Users\cindy\Desktop\Important"
)
# Log header
Add-Content -Path C:\Logs\kopia-vault.log -Value "=== Vault backup started at $(Get-Date) ==="
# Backup each directory
foreach ($dir in $VaultDirs) {
if (Test-Path $dir) {
Add-Content -Path C:\Logs\kopia-vault.log -Value "Backing up: $dir"
kopia --config-file="$KOPIA_CONFIG" snapshot create $dir `
| Tee-Object -FilePath C:\Logs\kopia-vault.log -Append
} else {
Add-Content -Path C:\Logs\kopia-vault.log -Value "Directory not found: $dir"
}
}
# Log completion
Add-Content -Path C:\Logs\kopia-vault.log -Value "=== Vault backup completed at $(Get-Date) ==="
Add-Content -Path C:\Logs\kopia-vault.log -Value ""
```
4. **Create log directory**
```powershell
New-Item -ItemType Directory -Force -Path C:\Logs
```
5. **Create scheduled task for vault backup**
- Press `Win + R` → type `taskschd.msc`
- Click "Create Task"
**General tab:**
- Name: `Kopia Vault Nightly Backup`
- ✔ Run whether user is logged on or not
- ✔ Run with highest privileges
**Triggers tab:**
- New → Daily at 3:00 AM (after primary backup)
- ✔ Enabled
**Actions tab:**
- Program: `powershell.exe`
- Arguments: `-ExecutionPolicy Bypass -File C:\Scripts\kopia-vault-nightly.ps1`
- Start in: `C:\Scripts`
**Conditions/Settings:** Same as primary backup task
---
## ZFS Replication to Offsite Pi Vaults
### Setup on ZNAS (Source)
1. **Create snapshot script**
```bash
sudo nano /usr/local/bin/vault-snapshot.sh
```
Add this content:
```bash
#!/bin/bash
# Create ZFS snapshot of vault dataset for replication
DATASET="zpool/vault/backup"
SNAPSHOT_NAME="vault-$(date +%Y%m%d-%H%M%S)"
# Create snapshot
zfs snapshot "${DATASET}@${SNAPSHOT_NAME}"
# Keep only last 7 days of snapshots on source
zfs list -t snapshot -o name -s creation | grep "^${DATASET}@vault-" | head -n -7 | xargs -r -n 1 zfs destroy
echo "Created snapshot: ${DATASET}@${SNAPSHOT_NAME}"
```
2. **Make executable**
```bash
sudo chmod +x /usr/local/bin/vault-snapshot.sh
```
3. **Schedule snapshot creation**
```bash
sudo crontab -e
# Add this line (create snapshot daily at 4 AM, after vault backups complete)
0 4 * * * /usr/local/bin/vault-snapshot.sh >> /var/log/vault-snapshot.log 2>&1
```
4. **Create replication script**
```bash
sudo nano /usr/local/bin/vault-replicate.sh
```
Add this content:
```bash
#!/bin/bash
# Replicate vault dataset to offsite Pi systems
DATASET="zpool/vault/backup"
PI1_HOST="pi-vault-1.local" # Update with actual hostname/IP
PI2_HOST="pi-vault-2.local" # Update with actual hostname/IP
PI_USER="admin"
REMOTE_DATASET="tank/vault-backup" # Update with actual dataset on Pi
# Get the latest snapshot
LATEST_SNAP=$(zfs list -t snapshot -o name -s creation | grep "^${DATASET}@vault-" | tail -n 1)
if [ -z "$LATEST_SNAP" ]; then
echo "No snapshots found for replication"
exit 1
fi
echo "Replicating snapshot: $LATEST_SNAP"
# Function to replicate to a target
replicate_to_target() {
local TARGET_HOST=$1
echo "=== Replicating to $TARGET_HOST ==="
# Get the last snapshot on remote (if any)
LAST_REMOTE=$(ssh ${PI_USER}@${TARGET_HOST} "zfs list -t snapshot -o name -s creation 2>/dev/null | grep '^${REMOTE_DATASET}@vault-' | tail -n 1" || echo "")
if [ -z "$LAST_REMOTE" ]; then
# Initial replication (full send)
echo "Performing initial full replication to $TARGET_HOST"
zfs send -c $LATEST_SNAP | ssh ${PI_USER}@${TARGET_HOST} "zfs receive -F ${REMOTE_DATASET}"
else
# Incremental replication
echo "Performing incremental replication to $TARGET_HOST"
LAST_SNAP_NAME=$(echo $LAST_REMOTE | cut -d'@' -f2)
zfs send -c -i ${DATASET}@${LAST_SNAP_NAME} $LATEST_SNAP | ssh ${PI_USER}@${TARGET_HOST} "zfs receive -F ${REMOTE_DATASET}"
fi
# Clean up old snapshots on remote (keep last 30 days)
ssh ${PI_USER}@${TARGET_HOST} "zfs list -t snapshot -o name -s creation | grep '^${REMOTE_DATASET}@vault-' | head -n -30 | xargs -r -n 1 zfs destroy"
echo "Replication to $TARGET_HOST completed"
}
# Replicate to both Pi systems
replicate_to_target $PI1_HOST
replicate_to_target $PI2_HOST
echo "All replications completed at $(date)"
```
5. **Make executable**
```bash
sudo chmod +x /usr/local/bin/vault-replicate.sh
```
6. **Set up SSH keys for passwordless replication**
```bash
# Generate SSH key if needed
ssh-keygen -t ed25519 -C "znas-replication"
# Copy to both Pi systems
ssh-copy-id admin@pi-vault-1.local
ssh-copy-id admin@pi-vault-2.local
```
7. **Schedule replication**
```bash
sudo crontab -e
# Add this line (replicate daily at 5 AM, after snapshot creation)
0 5 * * * /usr/local/bin/vault-replicate.sh >> /var/log/vault-replicate.log 2>&1
```
### Setup on Pi Vault Systems (Targets)
Repeat these steps on both Pi Vault 1 and Pi Vault 2:
1. **Create ZFS pool on SSD** (if not already done)
```bash
# Assuming SSD is /dev/sda
sudo zpool create tank /dev/sda
```
2. **Create dataset for receiving backups**
```bash
sudo zfs create tank/vault-backup
```
3. **Set appropriate permissions**
```bash
# Allow the replication user to receive snapshots
sudo zfs allow admin receive,create,mount,destroy tank/vault-backup
```
4. **Verify replication** (after first run)
```bash
zfs list -t snapshot | grep vault-
```
---
## Maintenance and Monitoring
### Regular Health Checks
**On Clients:**
```bash
# Linux
sudo kopia snapshot list
sudo kopia snapshot verify --file-parallelism=8
sudo kopia repository status
# Windows (PowerShell)
kopia snapshot list
kopia snapshot verify --file-parallelism=8
kopia repository status
```
**On ZNAS:**
```bash
# Check ZFS health
zpool status
# Check both Kopia servers are running
docker ps | grep kopia
# Check vault snapshots
zfs list -t snapshot | grep "vault/backup"
# Check replication logs
tail -f /var/log/vault-replicate.log
# View server statuses
docker exec kopia-server-primary kopia server status
docker exec kopia-server-vault kopia server status
```
**On Pi Vaults:**
```bash
# Check received snapshots
zfs list -t snapshot | grep vault-backup
# Check available space
zfs list tank/vault-backup
```
### Monthly Maintenance Tasks
1. **Verify vault backups are replicating**
```bash
# On ZNAS
cat /var/log/vault-replicate.log | grep "completed"
# On Pi systems
zfs list -t snapshot -o name,creation | grep vault-backup | tail
```
2. **Test restore from vault repository**
```bash
# Connect to vault repo and verify a random snapshot
kopia --config-file=/path/to/vault/config repository connect server --url=...
kopia snapshot list
kopia snapshot verify --file-parallelism=8
```
3. **Check disk space on all systems**
4. **Review backup logs for errors**
### Backup Policy Recommendations
**Primary Repository:**
- Retention: 7 daily, 4 weekly, 6 monthly
- Compression: enabled
- All data from clients
**Vault Repository:**
- Retention: 14 daily, 8 weekly, 12 monthly, 3 yearly
- Compression: enabled
- Only critical data for offsite protection
**ZFS Snapshots:**
- Keep 7 days on ZNAS (source)
- Keep 30 days on Pi vaults (targets)
---
## Disaster Recovery Procedures
### Scenario 1: Restore from Primary Repository
```bash
# Linux
sudo kopia snapshot list
sudo kopia snapshot restore <snapshot-id> /restore/location
# Windows
kopia snapshot list
kopia snapshot restore <snapshot-id> C:\restore\location
```
### Scenario 2: Restore from Vault Repository (Offsite)
If ZNAS is unavailable, restore directly from Pi vault:
1. **On Pi vault:**
```bash
# Mount the latest snapshot
LATEST=$(zfs list -t snapshot -o name | grep vault-backup | tail -n 1)
zfs clone $LATEST tank/vault-backup-restore
```
2. **Access Kopia repository directly:**
```bash
kopia repository connect filesystem --path=/tank/vault-backup-restore
kopia snapshot list
kopia snapshot restore <snapshot-id> /restore/location
```
3. **Clean up after restore:**
```bash
zfs destroy tank/vault-backup-restore
```
### Scenario 3: Complete System Rebuild
1. Rebuild ZNAS and restore vault dataset from Pi
2. Reinstall Kopia server in Docker
3. Point server to restored vault repository
4. Reconnect clients to primary and vault repositories
5. Resume scheduled backups
---
## Troubleshooting
### Client can't connect to repository
```bash
# Check both servers are running
docker ps | grep kopia
# Should see both kopia-server-primary and kopia-server-vault
# Check firewall
sudo ufw status | grep 51515
sudo ufw status | grep 51516
# Verify certificate fingerprints
docker exec kopia-server-primary kopia server status
docker exec kopia-server-vault kopia server status
# Check server logs
docker logs kopia-server-primary
docker logs kopia-server-vault
```
### Vault replication failing
```bash
# Check SSH connectivity
ssh admin@pi-vault-1.local "echo Connected"
# Check ZFS pool health
zpool status
# Check remote dataset exists
ssh admin@pi-vault-1.local "zfs list tank/vault-backup"
# Manual test send
zfs send -n -v zpool/vault/backup@latest | ssh admin@pi-vault-1.local "cat > /dev/null"
```
### Windows scheduled task not running
- Check Task Scheduler → Task History
- Verify PIN/password authentication (use password Harvey123= for task credential)
- Check that computer is awake at scheduled time
- Review power settings (prevent sleep, wake for tasks)
- Check log files: `C:\Logs\kopia-primary.log` and `C:\Logs\kopia-vault.log`
### Snapshot cleanup not working
```bash
# Manually clean old snapshots
zfs list -t snapshot -o name,used,creation | grep vault-backup
# Remove specific snapshot
zfs destroy zpool/vault/backup@vault-YYYYMMDD-HHMMSS
```
---
## Security Notes
1. **Passwords in scripts:** Current implementation stores passwords in plaintext in scripts. For production, consider:
- Windows Credential Manager
- Linux keyring or encrypted credential storage
- Environment variables set at system level
2. **SSH keys:** Replication uses SSH keys. Keep private keys secure and use passphrase protection where possible.
3. **Network security:** Kopia server uses HTTPS with certificate validation. Ensure certificate fingerprint is verified on first connection.
4. **Physical security:** Offsite Pi vaults should be stored in secure locations with different risk profiles (fire, flood, theft).
---
## Quick Reference Commands
### Kopia Client Commands
```bash
# List snapshots
kopia snapshot list
# Create snapshot
kopia snapshot create /path/to/backup
# Verify integrity
kopia snapshot verify --file-parallelism=8
# Check repository status
kopia repository status
# View policies
kopia policy list
# Mount snapshot (Linux)
kopia mount <snapshot-id> /mnt/snapshot
# Use alternate config (for vault repository)
kopia --config-file=/path/to/vault/repository.config snapshot list
```
### ZFS Commands
```bash
# List snapshots
zfs list -t snapshot
# Create manual snapshot
zfs snapshot zpool/vault/backup@manual-$(date +%Y%m%d)
# Send full snapshot
zfs send zpool/vault/backup@snapshot | ssh user@host zfs receive tank/backup
# Send incremental
zfs send -i @old @new zpool/vault/backup | ssh user@host zfs receive tank/backup
# List replication progress
zpool status -v
# Check dataset size
zfs list -o space zpool/vault/backup
```
---
## Appendix: System Specifications
**ZNAS:**
- ZFS fileserver
- Docker running **two** Kopia servers:
- **kopia-server-primary** on port 51515
- **kopia-server-vault** on port 51516
- IP: 192.168.5.10
- Datasets:
- `/srv/vault/kopia_repository` (zpool/vault/kopia_repository) - Primary repository
- `/srv/vault/backup` (zpool/vault/backup) - Vault repository (replicated)
**Clients:**
- **docker2** (Linux) - Backs up /DockerVol/
- Primary: Every 3 hours → port 51515
- Vault: Daily at 3 AM (critical directories only) → port 51516
- **DESKTOP-QLSVD8P** (Windows - Cindy's desktop) - Backs up C:\Users\cindy
- Primary: Daily at 2 AM → port 51515
- Vault: Daily at 3 AM (Documents, Pictures, Important files) → port 51516
- Kopia password: LucyDog123
- Task Scheduler credential: Harvey123=
**Offsite Vaults:**
- **Pi Vault 1** - Raspberry Pi with SSD (tank/vault-backup)
- **Pi Vault 2** - Raspberry Pi with SSD (tank/vault-backup)
**Server Certificates:**
- Primary server SHA256: `696a4999f594b5273a174fd7cab677d8dd1628f9b9d27e557daa87103ee064b2`
- Vault server SHA256: *(get from `docker exec kopia-server-vault kopia server status`)*
---
## Workflow Summary
### Daily Backup Flow
**2:00 AM** - Cindy's desktop primary backup runs
**3:00 AM** - docker2 vault backup runs
**3:00 AM** - Cindy's desktop vault backup runs
**4:00 AM** - ZNAS creates ZFS snapshot of vault dataset
**5:00 AM** - ZNAS replicates vault snapshot to both Pi systems
**Every 3 hours** - docker2 primary backup runs
### What Gets Backed Up Where
**Primary Repository (Full Backups):**
- docker2: /DockerVol/ (all Docker volumes)
- Cindy: C:\Users\cindy (entire user profile, minus temp files)
**Vault Repository (Critical Data for Offsite):**
- docker2: Selected critical Docker volumes
- Cindy: Documents, Pictures, Important desktop files
**Offsite (Via ZFS Send):**
- Entire vault repository (all clients' critical data)
- Replicated to 2 separate Pi systems
---
## Future Enhancements
Consider adding:
- Email notifications on backup failures
- Monitoring dashboard (Grafana/Prometheus)
- Backup validation automation
- Additional retention policies per client
- Encrypted credentials storage
- Remote monitoring of Pi vault systems
- Automated restore testing
- Bandwidth throttling for replication
- Multiple ZFS snapshot retention policies
---
## Change Log
- **2025-02-11** - Initial comprehensive documentation created
- Added two-tier backup strategy (primary + vault)
- Added ZFS replication procedures for offsite backup
- Added Pi vault setup instructions
- Added disaster recovery procedures
- Consolidated all client configurations
- Added workflow diagrams and timing
---
## Support and Feedback
For issues or improvements to this documentation, contact the system administrator.
**Useful Resources:**
- Kopia Documentation: https://kopia.io/docs/
- ZFS Administration Guide: https://openzfs.github.io/openzfs-docs/
- Kopia GitHub: https://github.com/kopia/kopia

113
Vault-Grimoire/Kopia/Kopia-Service.md Normal file
View file

@ -0,0 +1,113 @@
# kopia
## Overview
The kopia stack is a Docker Swarm configuration for the Kopia backup service in NetGrimoire. It provides snapshot backups and deduplication capabilities.
---
## Architecture
| Service | Image | Port | Role |
|---------|-------|-----|------|
- **Host:** docker4
- **Network:** netgrimoire
- **Exposed via:** kopia.netgrimoire.com, 51515 (via Caddy reverse proxy)
- **Homepage group:** Backup
---
## Build & Configuration
### Prerequisites
None specified.
### Volume Setup
```bash
mkdir -p /DockerVol/kopia/config
mkdir -p /DockerVol/kopia/cache
mkdir -p /DockerVol/kopia/cert
```
### Environment Variables
```bash
# generate: openssl rand -hex 32 for secrets
POUID=1964
PGID=1964
KOPIA_PASSWORD=F@lcon13
KOPIA_SERVER_USERNAME=admin
KOPIA_SERVER_PASSWORD=F@lcon13
TZ=America/Chicago
```
### Deploy
```bash
cd services/swarm/stack/kopia
set -a && source .env && set +a
docker stack config --compose-file kopia-stack.yml > resolved.yml
docker stack deploy --compose-file resolved.yml kopia
rm resolved.yml
docker stack services kopia
```
### First Run
After deployment, check the status of the Kopia service and verify that backups are being created.
---
## User Guide
### Accessing kopia
| Service | URL | Purpose |
|---------|-----|---------|
- **kopia**: https://kopia.netgrimoire.com (via Caddy reverse proxy)
### Primary Use Cases
To use Kopia in NetGrimoire, create a new backup set and configure the service to run as desired.
### NetGrimoire Integrations
This service integrates with Uptime Kuma for monitoring and other services through environment variables and labels.
---
## Operations
### Monitoring
```bash
docker stack services kopia
docker service logs -f kopia
```
### Backups
Critical backups are stored at `/DockerVol/kopia/config` and `/DockerVol/kopia/cache`. Reconstructable backups can be restored from `/DockerVol/kopia/cache`.
### Restore
To restore a backup, run the following command:
```bash
./deploy.sh
```
---
## Common Failures
| Symptom | Cause | Fix |
|---------|-------|-----|
| Backups not being created | Insufficient storage or network issues | Check storage and network conditions. |
| Service not starting | Incorrect environment variables or Docker configuration | Review `.env` file and `docker-compose.yml`. |
---
## Changelog
| Date | Commit | Summary |
|------|--------|---------|
| 2026-04-07 | d3206f11 | Initial documentation for kopia stack. |
| 2026-02-11 | aa13ac64 | Minor adjustments to environment variables and volume setup. |
| 2026-01-30 | 15f5f655 | Initial commit with basic configuration and service setup. |
<No changelog entries available from diffs above>
---
## Notes
- Generated by Gremlin on 2026-04-07T19:20:00.179Z
- Source: swarm/kopia.yaml
- Review User Guide and Changelog sections

44
Vault-Grimoire/Offsite/Vault-Architecture.md Normal file
View file

@ -0,0 +1,44 @@
---
title: Offsite Vault Architecture
description: Two Pi vault nodes — ZFS raw send, syncoid, Pocket Grimoire
published: true
date: 2026-04-12T00:00:00.000Z
tags: vault, offsite, zfs, kopia
editor: markdown
dateCreated: 2026-04-12T00:00:00.000Z
---
# Offsite Vault Architecture
## Overview
Two offsite nodes receive ZFS replication from `znas`:
| Node | Location | Role |
|------|----------|------|
| Vault Pi (dedicated) | Offsite / home shelf | Kopia offsite server, ZFS vault pool |
| Pocket Grimoire | Travel / portable | Portable vault + media, also a vault node |
## Replication Method
ZFS raw send via `syncoid` with `-w` flag (raw/encrypted mode):
```bash
# Dedicated vault Pi
syncoid -w znas:vault/data vault-pi:vault/data
# Pocket Grimoire pre-travel
syncoid znas:vault/Green/Pocket pocket:/srv/greenpg/Green
```
The `-w` flag sends encrypted ZFS streams. The receiving node stores data in its encrypted form — no decryption keys are needed on the vault nodes. Keys stay exclusively on `znas`.
## Kopia Offsite Server
The vault container (`vault.yaml`) runs a Kopia server on port 51516 that serves as the remote endpoint for the dedicated Pi vault. Accessible at `vault.netgrimoire.com`.
## Pocket Grimoire as Vault Node
Pocket Grimoire's ZFS pool (`pocket-green` at `/srv/greenpg/`) receives a `syncoid` push from `znas` before each trip. This makes Pocket Grimoire an offsite backup node whenever it leaves the house.
See [Pocket Grimoire Sync](/Pocket-Grimoire/Sync/Pre-Travel-Sync) for the pre-travel checklist.

60
Vault-Grimoire/Overview.md Normal file
View file

@ -0,0 +1,60 @@
---
title: Vault Grimoire
description: Storage and backup — the dragon guards the data hoard
published: true
date: 2026-04-12T00:00:00.000Z
tags: vault, storage, backup
editor: markdown
dateCreated: 2026-04-12T00:00:00.000Z
---
# Vault Grimoire
![vault-badge](/images/vault-badge.png)
The Vault Grimoire covers all storage and backup infrastructure. Data starts at `znas`, is deduplicated and encrypted by Kopia, and replicates offsite to two Pi vault nodes — one dedicated vault Pi and one inside Pocket Grimoire.
---
## Sections
| Section | Contents |
|---------|----------|
| [ZFS](/Vault-Grimoire/ZFS/Storage-Layout) | ZFS pools, datasets, NFS exports, commands reference |
| [Kopia](/Vault-Grimoire/Kopia/Kopia-Overview) | Backup repos, retention, restore, two-repo architecture |
| [Backups](/Vault-Grimoire/Backups/Services-Backup) | Per-service backup runbooks (Immich, MailCow, Nextcloud, Wiki, services) |
| [Offsite](/Vault-Grimoire/Offsite/Vault-Architecture) | Pi vault nodes, ZFS raw send, syncoid workflow |
---
## Offsite Vault Architecture
```
znas (primary)
└── ZFS pool → Kopia dedup → encrypted repo
├── syncoid -w → Pi Vault (dedicated offsite)
└── syncoid → Pocket Grimoire (portable vault node)
```
Both offsite nodes receive ZFS raw send with the `-w` flag. Encryption keys stay on `znas`. The vault nodes store encrypted data only — no keys needed there.
---
## Two-Repo Architecture
Kopia uses two separate containers on different ports:
| Container | Repo | URL | Purpose |
|-----------|------|-----|---------|
| kopia | Primary vault | `kopia.netgrimoire.com` | Main backup, dedup, retention |
| vault | Offsite server | `vault.netgrimoire.com` (port 51516) | Replication target for Pi vaults |
One Kopia server instance per repository. They cannot share.
---
## Key Rules
- ZFS encryption cannot be done in-place. Migration requires `rsync` to a new encrypted dataset, then ZFS raw send with `-w` to vaults (no key exposure on vault side).
- ZFS must fully mount before NFS starts on znas. Systemd override required: `After=zfs-import.target zfs-mount.service`.
- Loopback NFS mount needs `x-systemd.after=nfs-server.service` in fstab.

393
Vault-Grimoire/ZFS/NFS-Exports.md Normal file
View file

@ -0,0 +1,393 @@
---
title: ZFS-NFS-Exports
description: Exporting NFS shares from ZFS datasets
published: true
date: 2026-02-23T21:58:20.626Z
tags:
editor: markdown
dateCreated: 2026-02-01T20:45:40.210Z
---
# NFS Configuration
## Overview
ZNAS exports storage via NFSv4. All exports are ZFS datasets mounted directly to `/export/*` — no bind mounts. NFS is configured to wait for ZFS at boot via a systemd override.
ZNAS also mounts its own NFS exports back to itself at `/data/nfs/znas`. This is intentional: Docker Swarm containers scheduled to ZNAS need to access NAS storage at the same paths as containers running on other swarm members. The loopback mount provides a consistent NFS-backed path regardless of which node a container lands on.
All other clients are Linux systems using autofs.
---
## Server Configuration
### ZFS Mountpoints
ZFS datasets mount directly to `/export/*`. No bind mounts are used.
```
vault → /export
vault/Common → /export/Common
vault/Data → /export/Data
vault/Data/media_books → /export/Data/media/books
vault/Data/media_comics → /export/Data/media/comics
vault/Docker → /export/Docker
vault/Green → /export/Green
vault/Green/Pocket → /export/Green/Pocket
vault/Photos → /export/Photos
```
Verify at any time:
```bash
mount | grep export
```
### /etc/exports
```
# NFSv4 - pseudo filesystem root
/export *(ro,fsid=0,no_root_squash,no_subtree_check,crossmnt)
# Shares beneath the NFSv4 root
/export/Common *(fsid=4,rw,no_subtree_check,insecure)
/export/Data *(fsid=5,rw,no_subtree_check,insecure,crossmnt)
/export/Data/media/books *(fsid=51,rw,no_subtree_check,insecure,nohide)
/export/Data/media/comics *(fsid=52,rw,no_subtree_check,insecure,nohide)
/export/Docker *(fsid=29,rw,no_root_squash,sync,no_subtree_check,insecure)
/export/Green *(fsid=30,rw,no_root_squash,no_subtree_check,insecure)
/export/photos *(fsid=31,rw,no_root_squash,no_subtree_check,insecure)
```
**Key options:**
- `fsid=0` on `/export` — required for NFSv4 pseudo-root. Clients enumerate all exports from here.
- `crossmnt` — allows NFS to cross ZFS dataset boundaries when traversing the tree.
- `nohide` — required on `media/books` and `media/comics` because they are separate ZFS datasets mounted beneath the `vault/Data` export path. Without it clients see empty directories.
- `no_root_squash` — Docker and Green exports allow root writes. Required for container volume mounts.
- `insecure` — permits connections from unprivileged ports (>1024). Required for some Linux NFS clients and all macOS clients.
- `sync` on Docker — forces synchronous writes for container volume safety.
### systemd Boot Order Override
NFS is configured to wait for ZFS to fully mount before starting.
`/etc/systemd/system/nfs-server.service.d/override.conf`:
```ini
[Unit]
After=zfs-import.target zfs-mount.service local-fs.target
Requires=zfs-import.target zfs-mount.service
```
Apply after any changes:
```bash
sudo systemctl daemon-reload
sudo systemctl restart nfs-server
```
### Autofs Disabled on Server
Autofs is disabled on ZNAS itself. It must only run on NFS clients. Running autofs on the server creates recursive mount loops.
```bash
sudo systemctl stop autofs
sudo systemctl disable autofs
```
---
## Loopback Mount (Docker Swarm)
ZNAS mounts its own NFS exports back to itself at `/data/nfs/znas`. This ensures containers scheduled to ZNAS by Docker Swarm access storage at the same NFS-backed paths as containers running on any other swarm member — consistent regardless of which node a service lands on.
Swarm container volume mounts reference paths under `/data/nfs/znas/` rather than `/export/` directly.
### The Timing Problem
Getting this mount to survive reboots reliably was non-trivial. The loopback has a chicken-and-egg dependency chain:
1. ZFS must import and mount pools before NFS server can export anything
2. NFS server must be fully started before the loopback mount can succeed
3. The loopback mount must be established before Docker Swarm containers start
A plain `_netdev` fstab entry is not sufficient — `_netdev` only guarantees the network is up, not that the NFS server is ready. The mount would race against NFS startup and fail silently or hang.
### Solution — fstab with x-systemd.after
The loopback is established via `/etc/fstab` using the `x-systemd.after` option to explicitly declare the dependency on `nfs-server.service`:
```
localhost:/ /data/nfs/znas nfs4 defaults,_netdev,x-systemd.after=nfs-server.service 0 0
```
`x-systemd.after=nfs-server.service` causes systemd-fstab-generator to automatically create a mount unit (`data-nfs-znas.mount`) with `After=nfs-server.service` in its `[Unit]` block. This guarantees the full dependency chain:
```
zfs-import.target
→ zfs-mount.service
→ nfs-server.service (via nfs-server override.conf)
→ data-nfs-znas.mount (via x-systemd.after in fstab)
→ remote-fs.target
→ Docker Swarm containers
```
The generated unit (created automatically at runtime by systemd-fstab-generator — not a file on disk):
```ini
# /run/systemd/generator/data-nfs-znas.mount
[Unit]
Documentation=man:fstab(5) man:systemd-fstab-generator(8)
SourcePath=/etc/fstab
After=nfs-server.service
Before=remote-fs.target
[Mount]
What=localhost:/
Where=/data/nfs/znas
Type=nfs4
Options=defaults,_netdev,x-systemd.after=nfs-server.service
```
**Do not create a hand-written systemd mount unit for this.** systemd-fstab-generator handles it automatically from the fstab entry. A manual unit would conflict.
### Verify Loopback is Active
```bash
mount | grep data/nfs/znas
# Should show: localhost:/ on /data/nfs/znas type nfs4 (...)
systemctl status data-nfs-znas.mount
# Should show: active (mounted)
```
---
## Client Configuration
All non-Swarm clients are Linux systems using autofs.
### Autofs Configuration
`/etc/auto.master` (relevant entry):
```
/data/nfs /etc/auto.nfs
```
`/etc/auto.nfs`:
```
znas -fstype=nfs4 192.168.5.10:/
```
This mounts the full NFSv4 tree from ZNAS at `/data/nfs/znas` on demand — the same path used by the loopback mount on ZNAS itself. All swarm nodes (including ZNAS) access NAS storage via `/data/nfs/znas/`.
**Note:** Autofs must be enabled on clients and disabled on the NFS server. Running autofs on the server creates recursive mount loops.
### Adding a New Client
```bash
# Install autofs if not present
sudo apt install autofs
# Add to /etc/auto.master if not already present
echo "/data/nfs /etc/auto.nfs" | sudo tee -a /etc/auto.master
# Create or update /etc/auto.nfs
echo "znas -fstype=nfs4 192.168.5.10:/" | sudo tee -a /etc/auto.nfs
# Reload autofs
sudo systemctl reload autofs
# Trigger mount by accessing the path
ls /data/nfs/znas/
```
### Manual Mount (testing only)
```bash
# Verify exports are visible from client
showmount -e 192.168.5.10
# Test manual mount
sudo mkdir -p /mnt/znas
sudo mount -t nfs4 192.168.5.10:/ /mnt/znas
# Verify tree is accessible
ls /mnt/znas/Data/media/books/
# Unmount after testing
sudo umount /mnt/znas
```
---
## Adding New Datasets
When creating a new ZFS dataset that needs to be NFS-accessible:
```bash
# Create with the correct mountpoint from the start
sudo zfs create -o mountpoint=/export/Data/new_folder vault/Data/new_folder
```
The dataset will be automatically visible via NFS due to `crossmnt` and `nohide` on the parent — no changes to `/etc/exports` needed unless the new dataset requires different access controls.
If different permissions are required, add an explicit entry to `/etc/exports` and reload:
```bash
sudo exportfs -ra
```
---
## Current Export List
Verified via `showmount -e 127.0.0.1`:
```
/export/photos *
/export/Green *
/export/Docker *
/export/Data/media/comics *
/export/Data/media/books *
/export/Data *
/export/Common *
/export *
```
---
## Known Gotchas
**Loopback mount races NFS at boot** — This was the hardest problem to solve. A plain `_netdev` fstab entry only guarantees the network interface is up, not that the NFS server is ready to accept connections. The loopback mount would attempt before NFS finished starting and fail silently or hang. The fix is `x-systemd.after=nfs-server.service` in the fstab options, which causes systemd-fstab-generator to emit an `After=nfs-server.service` dependency in the generated mount unit. The full required boot chain is: `zfs-import.target` → `zfs-mount.service` → `nfs-server.service` → `data-nfs-znas.mount`. Each link must be explicit.
**Do not hand-write a systemd mount unit for the loopback** — systemd-fstab-generator creates `data-nfs-znas.mount` automatically from the fstab entry at runtime (in `/run/systemd/generator/`, not `/etc/systemd/system/`). Creating a manual unit in `/etc/systemd/system/` will conflict with the generated one.
**Autofs must be disabled on the server** — Running autofs on ZNAS itself creates a recursive mount loop. Autofs belongs on clients only. If autofs is accidentally re-enabled on ZNAS it will fight with the fstab loopback mount.
**NFSv4 pseudo-root is required** — The `/export` entry with `fsid=0` is mandatory for NFSv4 clients. Without it clients cannot enumerate the export tree. Do not remove it even though it looks redundant.
**`nohide` on sub-datasets** — `vault/Data/media_books` and `vault/Data/media_comics` are separate ZFS datasets mounted beneath the `vault/Data` export path. NFS does not cross filesystem boundaries by default. Without `nohide` clients see empty directories at those paths even though the data is present.
**Do not use bind mounts for ZFS datasets** — Configure ZFS mountpoints directly to `/export/*`. Bind mounts in fstab for ZFS datasets cause ordering problems and are unnecessary.
**Always set mountpoints when creating new datasets** — If a dataset is created without an explicit mountpoint it will inherit the parent's path and may not be visible or exportable correctly. Set `mountpoint=` at creation time.
---
## Troubleshooting
### Datasets not visible via NFS
```bash
# Verify dataset is mounted
zfs list | grep dataset_name
# Check NFS can read it
sudo -u nobody ls -la /export/path/to/dataset/
# Reload exports
sudo exportfs -ra
sudo systemctl restart nfs-server
```
### Client shows empty directories
```bash
# Clear NFS cache and remount
sudo umount -f /mnt/znas
sudo mount -t nfs4 192.168.5.10:/ /mnt/znas
# Test without caching to isolate the problem
sudo mount -t nfs4 -o noac,lookupcache=none 192.168.5.10:/ /mnt/znas
```
### After reboot, exports are empty
```bash
# Confirm ZFS mounted before NFS started
systemctl status zfs-mount.service
systemctl status nfs-server.service
# Confirm override is in place
systemctl cat nfs-server.service | grep -A5 "\[Unit\]"
```
### Loopback mount not working for Swarm containers
```bash
# Check mount unit status
systemctl status data-nfs-znas.mount
# Verify full dependency chain is satisfied
systemctl status zfs-mount.service
systemctl status nfs-server.service
systemctl status data-nfs-znas.mount
# Verify loopback is mounted
mount | grep data/nfs/znas
# If missing, mount manually to test
sudo mount -t nfs4 127.0.0.1:/ /data/nfs/znas
# Check container can see the path
docker run --rm -v /data/nfs/znas/Data:/data alpine ls /data
```
If the unit fails at boot, confirm the fstab entry includes `x-systemd.after=nfs-server.service` — without this the mount races against NFS startup and loses. A plain `_netdev` entry is not sufficient.
---
## Configuration Files Reference
### /etc/exports
```
/export *(ro,fsid=0,no_root_squash,no_subtree_check,crossmnt)
/export/Common *(fsid=4,rw,no_subtree_check,insecure)
/export/Data *(fsid=5,rw,no_subtree_check,insecure,crossmnt)
/export/Data/media/books *(fsid=51,rw,no_subtree_check,insecure,nohide)
/export/Data/media/comics *(fsid=52,rw,no_subtree_check,insecure,nohide)
/export/Docker *(fsid=29,rw,no_root_squash,sync,no_subtree_check,insecure)
/export/Green *(fsid=30,rw,no_root_squash,no_subtree_check,insecure)
/export/photos *(fsid=31,rw,no_root_squash,no_subtree_check,insecure)
```
### /etc/systemd/system/nfs-server.service.d/override.conf
```ini
[Unit]
After=zfs-import.target zfs-mount.service local-fs.target
Requires=zfs-import.target zfs-mount.service
```
### /etc/fstab (ZNAS system mounts only)
ZFS datasets are not listed here — ZFS handles its own mounting. Only system partitions appear:
```
# / - btrfs on nvme0n1p2
/dev/disk/by-uuid/40c60952-0340-4a78-81f9-5b2193da26c6 / btrfs defaults 0 1
# /boot - ext4 on nvme0n1p3
/dev/disk/by-uuid/4abb4efa-0b2b-4e4a-bcaf-78227db4628f /boot ext4 defaults 0 1
# swap
/dev/disk/by-uuid/d07437a0-3d0e-417a-a88e-438c603c2237 none swap sw 0 0
# /srv - btrfs on nvme0n1p5
/dev/disk/by-uuid/c66e81ff-436e-4d6f-980b-6f4875ea7c8e /srv btrfs defaults 0 1
```
---
## Command Reference
- Show active exports: `sudo exportfs -v`
- Reload exports: `sudo exportfs -ra`
- Show available exports (from any host): `showmount -e 192.168.5.10`
- Restart NFS: `sudo systemctl restart nfs-server`
- Check NFS status: `systemctl status nfs-server`
- Verify ZFS mounts: `mount | grep export`
- Verify loopback: `mount | grep data/nfs`

239
Vault-Grimoire/ZFS/Storage-Layout.md Normal file
View file

@ -0,0 +1,239 @@
---
title: Netgrimoire Storage
description: Where is it at
published: true
date: 2026-02-23T18:38:27.621Z
tags:
editor: markdown
dateCreated: 2026-01-22T21:10:37.035Z
---
# NAS Storage Layout
## Overview
ZNAS is the primary NAS for Netgrimoire. It runs Ubuntu with OpenZFS and serves as the source of truth for all storage, including datasets that replicate out to the Pocket Grimoire portable system.
The system mounts everything under `/export/` for NFS sharing, with select datasets mounted under `/srv/` for local service consumption (Immich, NextCloud-AIO, Kopia, backup).
## ZFS Pools
- `vault` — primary NAS storage, RAIDZ1×2, 8 drives
- `greenpg` — Pocket Grimoire GREEN SSD (Kanguru UltraLock), docked for sync when present
## Zpool Architecture
```
pool: vault
state: ONLINE
scan: scrub repaired 0B in 2 days 10:24:08 with 0 errors on Tue Feb 10 10:48:10 2026
config:
NAME STATE READ WRITE CKSUM
vault ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
ata-ST24000DM001-3Y7103_ZXA06K45 ONLINE 0 0 0
ata-ST24000DM001-3Y7103_ZXA08CVY ONLINE 0 0 0
ata-ST24000DM001-3Y7103_ZXA0FP10 ONLINE 0 0 0
raidz1-1 ONLINE 0 0 0
ata-ST16000NE000-2RW103_ZL2Q3275 ONLINE 0 0 0
ata-ST16000NM001G-2KK103_ZL26R5XW ONLINE 0 0 0
ata-ST16000NT001-3LV101_ZRS0KVQW ONLINE 0 0 0
ata-WDC_WD140EDFZ-11A0VA0_9MG81N0J ONLINE 0 0 0
ata-WDC_WD140EDFZ-11A0VA0_Y5J35Z6C ONLINE 0 0 0
errors: No known data errors
```
`raidz1-0` is 3× Seagate 24TB (~48TB usable). `raidz1-1` is 3× Seagate 16TB + 2× WD 14TB (~56TB usable — the 14TB drives are the limiting factor per stripe, leaving ~2TB/drive unused on the 16TB drives). Total pool: ~94TB raw, 39TB currently available.
```
pool: greenpg
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
greenpg ONLINE 0 0 0
scsi-1Kanguru_UltraLock_DB090722NC10001 ONLINE 0 0 0
errors: No known data errors
```
`greenpg` is a portable pool. Export it before physically moving to Pocket Grimoire.
## ZFS Datasets
| Dataset | Mountpoint | Used | Avail | Refer | Quota | Compression | Purpose |
|---------|-----------|------|-------|-------|-------|-------------|---------|
| `vault` | `/export` | 55.3T | 39.0T | 771G | none | 1.00x | Pool root / NFSv4 pseudo-root |
| `vault/Common` | `/export/Common` | 214G | 39.0T | 214G | none | 1.06x | General shared storage |
| `vault/Data` | `/export/Data` | 38.4T | 39.0T | 36.4T | none | 1.00x | Primary data — 36.4T lives directly in dataset root |
| `vault/Data/media_books` | `/export/Data/media/books` | 925G | 39.0T | 925G | none | 1.03x | Book library |
| `vault/Data/media_comics` | `/export/Data/media/comics` | 1.15T | 39.0T | 1.15T | none | 1.00x | Comic library |
| `vault/Green` | `/export/Green` | 14.7T | 5.31T | 9.66T | 20T | 1.00x | Personal media — 9.66T direct, 5.02T in Pocket child |
| `vault/Green/Pocket` | `/export/Green/Pocket` | 5.02T | 2.48T | 5.02T | 7.5T | 1.00x | Pocket Grimoire replication source |
| `vault/Kopia` | `/srv/vault/kopia_repository` | 349G | 39.0T | 349G | none | 1.02x | Kopia backup repository |
| `vault/NextCloud-AIO` | `/srv/NextCloud-AIO` | 341G | 39.0T | 341G | none | 1.01x | NextCloud data |
| `vault/Photos` | `/export/Photos` | 135K | 39.0T | 135K | none | 1.00x | Photos (sparse — see notes) |
| `vault/backup` | `/srv/vault/backup` | 442G | 582G | 442G | 1T | 1.00x | Local system backups |
| `vault/docker` | `/export/Docker` | 22.2G | 39.0T | 22.2G | none | 1.13x | Docker volumes |
| `vault/immich` | `/srv/immich` | 117G | 39.0T | 117G | none | 1.03x | Immich photo service data |
| `greenpg` | `/greenpg` | 2.94T | 4.20T | 96K | — | 1.00x | GREEN SSD pool root (portable) |
| `greenpg/Pocket` | `/greenpg/Pocket` | 2.94T | 4.20T | 2.94T | — | 1.00x | Personal media + Stash data |
**Notes on specific datasets:**
`vault/Data` — 36.4T lives directly in the dataset root at `/export/Data/`. `media_books` and `media_comics` are the only child datasets and account for ~2T combined. The remaining ~36T is general data stored directly under the parent.
`vault/Green` — 9.66T lives directly in `/export/Green/` with the remaining 5.02T in the `Pocket` child dataset. The 20T quota caps total Green growth. `vault/Green/Pocket` has its own 7.5T sub-quota.
`vault/Photos` — nearly empty (135K). Photos are primarily managed through Immich at `vault/immich`. This dataset may be vestigial or reserved for future use.
`vault/backup` — has a hard 1T quota. Unlike other vault datasets which draw from the full 39T pool availability, this dataset is capped. Current usage is 442G with 582G remaining.
Compression ratios are near 1.00x across most datasets because content is already compressed (media files, binary data). `vault/docker` (1.13x) and `vault/Common` (1.06x) see modest gains from compressible config and text data.
## NFS Exports
All exports use NFSv4 with `/export` as the pseudo-filesystem root (`fsid=0`).
| Export | fsid | Options | Notes |
|--------|------|---------|-------|
| `/export` | 0 | `ro, no_root_squash, no_subtree_check, crossmnt` | NFSv4 pseudo-root — required for v4 clients |
| `/export/Common` | 4 | `rw, no_subtree_check, insecure` | General access |
| `/export/Data` | 5 | `rw, no_subtree_check, insecure, crossmnt` | Data root |
| `/export/Data/media/books` | 51 | `rw, no_subtree_check, insecure, nohide` | Separate ZFS dataset — needs `nohide` |
| `/export/Data/media/comics` | 52 | `rw, no_subtree_check, insecure, nohide` | Separate ZFS dataset — needs `nohide` |
| `/export/Docker` | 29 | `rw, no_root_squash, sync, no_subtree_check, insecure` | Container volumes |
| `/export/Green` | 30 | `rw, no_root_squash, no_subtree_check, insecure` | Personal media + Pocket Grimoire source |
| `/export/photos` | 31 | `rw, no_root_squash, no_subtree_check, insecure` | Photos |
Current `/etc/exports`:
```
/export *(ro,fsid=0,no_root_squash,no_subtree_check,crossmnt)
/export/Common *(fsid=4,rw,no_subtree_check,insecure)
/export/Data *(fsid=5,rw,no_subtree_check,insecure,crossmnt)
/export/Data/media/books *(fsid=51,rw,no_subtree_check,insecure,nohide)
/export/Data/media/comics *(fsid=52,rw,no_subtree_check,insecure,nohide)
/export/Docker *(fsid=29,rw,no_root_squash,sync,no_subtree_check,insecure)
/export/Green *(fsid=30,rw,no_root_squash,no_subtree_check,insecure)
/export/photos *(fsid=31,rw,no_root_squash,no_subtree_check,insecure)
```
There is also an active loopback NFSv4 mount on the system itself:
```
localhost:/ → /data/nfs/znas (NFSv4.2, rsize/wsize=1M)
```
## SMB Shares
*(To be documented.)*
## Standard Paths
- `/export/` — NFS root (vault pool root)
- `/export/Data/` — primary data
- `/export/Data/media/books/` — book library
- `/export/Data/media/comics/` — comic library
- `/export/Green/` — personal media
- `/export/Green/Pocket/` — Pocket Grimoire replication source
- `/export/Docker/` — container volumes
- `/export/Photos/` — photos
- `/srv/immich/` — Immich service data
- `/srv/NextCloud-AIO/` — NextCloud data
- `/srv/vault/kopia_repository/` — Kopia backup repo
- `/srv/vault/backup/` — local system backups
- `/greenpg/Pocket/` — GREEN SSD when docked for sync
## Permissions & UID/GID Model
*(To be documented — dockhand UID 1964, container access rules.)*
## Services Using Local Mounts
These datasets are consumed directly by services on ZNAS and are not NFS-exported:
| Service | Dataset | Mountpoint |
|---------|---------|-----------|
| Immich | `vault/immich` | `/srv/immich` |
| NextCloud-AIO | `vault/NextCloud-AIO` | `/srv/NextCloud-AIO` |
| Kopia | `vault/Kopia` | `/srv/vault/kopia_repository` |
| Local backup | `vault/backup` | `/srv/vault/backup` |
## Pocket Grimoire Integration
`vault/Green/Pocket` is the replication source for the Pocket Grimoire GREEN SSD (`greenpg`). It contains personal media and Stash application data (database, previews, blobs). See the Pocket Grimoire deployment guide for full procedures.
**Fast resync when GREEN SSD is physically docked on ZNAS:**
```bash
# Check pool name (retains whatever name it had when last exported)
zpool list | grep greenpg
# Import if needed
sudo zpool import greenpg
sudo zfs load-key greenpg
sudo zfs mount -a
# Sync
sudo syncoid vault/Green/Pocket greenpg/Pocket
# Export before physically disconnecting — always do this
sudo zfs unmount greenpg/Pocket
sudo zfs unmount greenpg
sudo zpool export greenpg
```
**Network sync** runs automatically on Pocket Grimoire via a 6-hour syncoid systemd timer when connected over the network.
## Backup & Snapshot Strategy
**Snapshots:**
```bash
# Manual pre-change snapshot
zfs snapshot vault/Docker@before-upgrade
# List all snapshots
zfs list -t snapshot
# List snapshots for a specific dataset
zfs list -t snapshot -r vault/Green
```
**Kopia:** Repository at `vault/Kopia` → `/srv/vault/kopia_repository`. *(Document snapshot policy and sources.)*
**Replication:** `vault/Green/Pocket` → `greenpg/Pocket` via syncoid. See Pocket Grimoire Integration above.
## Known Gotchas
**NFSv4 pseudo-root** — The `/export` entry with `fsid=0` is required for NFSv4 clients to enumerate subdirectories. Do not remove it even if it appears redundant.
**`nohide` on sub-datasets** — `vault/Data/media_books` and `vault/Data/media_comics` are separate ZFS datasets mounted beneath the `vault/Data` export path. NFS does not cross filesystem boundaries by default. Without `nohide` clients see empty directories at those paths.
**`vault/backup` quota** — This dataset has a hard 1T quota and does not share the general pool availability. Current headroom is ~582G. Monitor before large backup operations.
**`vault/Green` quota** — Capped at 20T total with a 7.5T sub-quota on `vault/Green/Pocket`. The GREEN SSD itself is ~7TB, so the sub-quota is the effective ceiling for the Pocket sync.
**raidz1-1 mixed drive sizes** — The three 16TB drives in raidz1-1 have ~2TB/drive going unused because RAIDZ1 stripes are limited by the smallest drive in the VDEV (14TB WDs). This capacity is permanently unavailable unless the VDEV is rebuilt.
**Kanguru UltraLock hardware encryption** — The GREEN SSD has hardware-level PIN protection in addition to ZFS encryption. The drive must be hardware-unlocked before `zpool import` will see it.
**Always export `greenpg` before disconnecting** — Export flushes writes and marks the pool clean. Pulling the drive without exporting risks a dirty import on next use.
**`vault/Data` root usage** — 36.4T lives directly in `/export/Data/` rather than in child datasets. This is normal for this setup but means `zfs list` on the parent alone shows the full usage without a breakdown.
## Command Reference
- Health: `zpool status`
- Space available to pool: `zpool list`
- Space available to datasets: `zfs list`
- Dataset configuration: `zfs get -r compression,dedup,recordsize,atime,quota,reservation vault`
- Create a snapshot: `zfs snapshot vault/Docker@before-upgrade`
- List snapshots: `zfs list -t snapshot`
- Reload NFS exports: `sudo exportfs -ra`
- Show active NFS exports: `sudo exportfs -v`
- Run a scrub: `sudo zpool scrub vault`
- Sync GREEN SSD: `sudo syncoid vault/Green/Pocket greenpg/Pocket`

168
Vault-Grimoire/ZFS/ZFS-Commands.md Normal file
View file

@ -0,0 +1,168 @@
---
title: ZFS Common Commands
description: ZFS Commands
published: true
date: 2026-02-20T04:26:23.798Z
tags: zfs commands
editor: markdown
dateCreated: 2026-01-31T15:23:07.585Z
---
# ZFS Essential Commands Cheat Sheet
---
## Pool Health & Status
zpool status
zpool status -v
zpool list
## Dataset Space & Usage
zfs list
zfs list -r vault
zfs list -o name,used,avail,refer,logicalused,compressratio
zfs list -r -o name,used,avail,refer,quota,reservation vault
## Dataset Properties & Settings
zfs get all vault/dataset
zfs get -r compression,dedup,recordsize,atime,quota,reservation vault
zfs get -r compression,dedup,recordsize,encryption,keylocation,keyformat,snapdir vault
zfs get -s local -r all vault
zfs get quota,refquota,reservation,refreservation -r vault
## Mount Encrypted Dataset
zfs load-key vault/Green/Pocket
zfs mount vault/Green/Pocket
## Pool I/O & Performance Monitoring
zpool iostat -v 1
arcstat 1
cat /proc/spl/kstat/zfs/arcstats
## Scrubs & Data Integrity
zpool scrub vault
zpool scrub -s vault
zpool status
## Snapshots
zfs snapshot vault/dataset@snapname
zfs list -t snapshot
zfs rollback vault/dataset@snapname
zfs clone vault/dataset@snapname vault/dataset-clone
## Replication (Send / Receive)
zfs send vault/dataset@snap1 | zfs receive backup/dataset
zfs send -i snap1 vault/dataset@snap2 | zfs receive backup/dataset
zfs send -nv vault/dataset@snap1
## Dataset Tuning (Live-Safe Changes)
zfs set compression=lz4 vault/dataset
zfs set recordsize=1M vault/dataset
zfs set atime=off vault/dataset
zfs set dedup=on vault/dataset
## Encryption Management
zfs get encryption,keylocation,keystatus vault/dataset
zfs unload-key vault/dataset
zfs load-key vault/dataset
## Disk Preparation & Cleanup
wipefs /dev/sdX
wipefs -a /dev/sdX
zpool labelclear -f /dev/sdX
sgdisk --zap-all /dev/sdX
lsblk -f /dev/sdX
## Pool Expansion (Add VDEV)
zpool add vault raidz2 \
/dev/disk/by-id/disk1 \
/dev/disk/by-id/disk2 \
/dev/disk/by-id/disk3 \
/dev/disk/by-id/disk4 \
/dev/disk/by-id/disk5
## Pool Import / Recovery
zpool import
zpool import vault
zpool import -f vault
zpool import -o readonly=on vault
## Locks, Holds & History
zfs holds -r vault
zpool history
zfs diff vault/dataset@snap1 vault/dataset@snap2
## Deduplication & Compression Stats
zpool list -v
zdb -DD vault
## Inventory / Documentation Dumps
zpool status > zpool-status.txt
zfs list -r > zfs-layout.txt
zfs get -r all vault > zfs-settings.txt
## Top 10 Must-Know Commands
zpool status
zpool list
zpool iostat -v 1
zpool scrub vault
zfs list
zfs get all vault/dataset
zfs snapshot vault/dataset@snap
zfs rollback vault/dataset@snap
zfs send | zfs receive
arcstat 1