New Grimoire

This commit is contained in:
traveler 2026-04-12 09:53:51 -05:00
parent 77d589a13d
commit cc574f8aed
157 changed files with 29420 additions and 0 deletions

View file

@ -0,0 +1,841 @@
---
title: Immich Backup and Restore
description: Immich backup with Kopia
published: true
date: 2026-02-20T04:11:52.181Z
tags:
editor: markdown
dateCreated: 2026-02-14T03:14:32.594Z
---
# Immich Backup and Recovery Guide
## Overview
This document provides comprehensive backup and recovery procedures for Immich photo server. Since Immich's data is stored on standard filesystems (not ZFS or BTRFS), snapshots are not available and we rely on Immich's native backup approach combined with Kopia for offsite storage in vaults.
## Quick Reference
### Common Backup Commands
```bash
# Run a manual backup (all components)
/opt/scripts/backup-immich.sh
# Backup just the database
docker exec -t immich_postgres pg_dump --clean --if-exists \
--dbname=immich --username=postgres | gzip > "/opt/immich-backups/dump.sql.gz"
# List Kopia snapshots
kopia snapshot list --tags immich
# View backup logs
tail -f /var/log/immich-backup.log
```
### Common Restore Commands
```bash
# Restore database from backup
gunzip < /opt/immich-backups/immich-YYYYMMDD_HHMMSS/dump.sql.gz | \
docker exec -i immich_postgres psql --username=postgres --dbname=immich
# Restore from Kopia to new server
kopia snapshot list --tags tier1-backup
kopia restore <snapshot-id> /opt/immich-backups/
# Check container status after restore
docker compose ps
docker compose logs -f
```
## Critical Components to Backup
### 1. Docker Compose File
- **Location**: `/opt/immich/docker-compose.yml` (or your installation path)
- **Purpose**: Defines all containers, networks, and volumes
- **Importance**: Critical for recreating the exact container configuration
### 2. Configuration Files
- **Primary Config**: `/opt/immich/.env`
- **Purpose**: Database credentials, upload locations, timezone settings
- **Importance**: Required for proper service initialization
### 3. Database
- **PostgreSQL Data**: Contains all metadata, user accounts, albums, sharing settings, face recognition data, timeline information
- **Container**: `immich_postgres`
- **Database Name**: `immich` (default)
- **User**: `postgres` (default)
- **Backup Method**: `pg_dump` (official Immich recommendation)
### 4. Photo/Video Library
- **Upload Storage**: All original photos and videos uploaded by users
- **Location**: `/srv/immich/library` (per your .env UPLOAD_LOCATION)
- **Size**: Typically the largest component
- **Critical**: This is your actual data - photos cannot be recreated
### 5. Additional Important Data
- **Model Cache**: Docker volume `immich_model-cache` (machine learning models, can be re-downloaded)
- **External Paths**: `/export/photos` and `/srv/NextCloud-AIO` (mounted as read-only in your setup)
## Backup Strategy
### Two-Tier Backup Approach
We use a **two-tier approach** combining Immich's native backup method with Kopia for offsite storage:
1. **Tier 1 (Local)**: Immich database dump + library backup creates consistent, component-level backups
2. **Tier 2 (Offsite)**: Kopia snapshots the local backups and syncs to vaults
#### Why This Approach?
- **Best of both worlds**: Native database dump ensures Immich-specific consistency, Kopia provides deduplication and offsite protection
- **Component-level restore**: Can restore individual components (just database, just library, etc.)
- **Disaster recovery**: Full system restore from Kopia backups on new server
- **Efficient storage**: Kopia's deduplication reduces storage needs for offsite copies
#### Backup Frequency
- **Daily**: Immich backup runs at 2 AM
- **Daily**: Kopia snapshot of backups runs at 3 AM
- **Retention (Local)**: 7 days of Immich backups (managed by script)
- **Retention (Kopia/Offsite)**: 30 daily, 12 weekly, 12 monthly
### Immich Native Backup Method
Immich's official backup approach uses `pg_dump` for the database:
- Uses `pg_dump` with `--clean --if-exists` flags for consistent database dumps
- Hot backup without stopping PostgreSQL
- Produces compressed `.sql.gz` files
- Database remains available during backup
For the photo/video library, we use a **hybrid approach**:
- **Database**: Backed up locally as `dump.sql.gz` for fast component-level restore
- **Library**: Backed up directly by Kopia (no tar) for optimal deduplication and incremental backups
**Why not tar the library?**
- Kopia deduplicates at the file level - adding 1 photo shouldn't require backing up the entire library again
- Individual file access for selective restore
- Better compression and faster incremental backups
- Lower risk - corrupted tar loses everything, corrupted file only affects that file
**Key Features:**
- No downtime required
- Consistent point-in-time snapshot
- Standard PostgreSQL format (portable across systems)
- Efficient incremental backups of photo library
## Setting Up Immich Backups
### Prereq:
Make sure you are connected to the repository,
```bash
sudo kopia repository connect server \
--url=https://192.168.5.10:51516 \
--override-username=admin \
--server-cert-fingerprint=696a4999f594b5273a174fd7cab677d8dd1628f9b9d27e557daa87103ee064b2
```
#### Step 1: Configure Backup Location
Set the backup destination:
```bash
# Create the backup directory
mkdir -p /opt/immich-backups
chown -R root:root /opt/immich-backups
chmod 755 /opt/immich-backups
```
#### Step 2: Manual Backup Commands
```bash
cd /opt/immich
# Backup database using Immich's recommended method
docker exec -t immich_postgres pg_dump \
--clean \
--if-exists \
--dbname=immich \
--username=postgres \
| gzip > "/opt/immich-backups/dump.sql.gz"
# Backup configuration files
cp docker-compose.yml /opt/immich-backups/
cp .env /opt/immich-backups/
# Backup library with Kopia (no tar - better deduplication)
kopia snapshot create /srv/immich/library \
--tags immich,library,photos \
--description "Immich library manual backup"
```
**What gets created:**
- Local backup directory: `/opt/immich-backups/immich-YYYY-MM-DD-HH-MM-SS/`
- Contains: `dump.sql.gz` (database), config files
- Kopia snapshots:
- `/opt/immich-backups` (database + config)
- `/srv/immich/library` (photos/videos, no tar)
- `/opt/immich` (installation directory)
#### Step 3: Automated Backup Script
Create `/opt/scripts/backup-immich.sh`:
```bash
#!/bin/bash
# Immich Automated Backup Script
# This creates Immich backups, then snapshots them with Kopia for offsite storage
set -e
BACKUP_DATE=$(date +%Y%m%d_%H%M%S)
LOG_FILE="/var/log/immich-backup.log"
IMMICH_DIR="/opt/immich"
BACKUP_DIR="/opt/immich-backups"
KEEP_DAYS=7
# Database credentials from .env
DB_USERNAME="postgres"
DB_DATABASE_NAME="immich"
POSTGRES_CONTAINER="immich_postgres"
echo "[${BACKUP_DATE}] ========================================" | tee -a "$LOG_FILE"
echo "[${BACKUP_DATE}] Starting Immich backup process" | tee -a "$LOG_FILE"
# Step 1: Run Immich database backup using official method
echo "[${BACKUP_DATE}] Running Immich database backup..." | tee -a "$LOG_FILE"
cd "$IMMICH_DIR"
# Create backup directory with timestamp
mkdir -p "${BACKUP_DIR}/immich-${BACKUP_DATE}"
# Backup database using Immich's recommended method
docker exec -t ${POSTGRES_CONTAINER} pg_dump \
--clean \
--if-exists \
--dbname=${DB_DATABASE_NAME} \
--username=${DB_USERNAME} \
| gzip > "${BACKUP_DIR}/immich-${BACKUP_DATE}/dump.sql.gz"
BACKUP_EXIT=${PIPESTATUS[0]}
if [ $BACKUP_EXIT -ne 0 ]; then
echo "[${BACKUP_DATE}] ERROR: Immich database backup failed with exit code ${BACKUP_EXIT}" | tee -a "$LOG_FILE"
exit 1
fi
echo "[${BACKUP_DATE}] Immich database backup completed successfully" | tee -a "$LOG_FILE"
# Step 2: Verify library location exists (Kopia will backup directly, no tar needed)
echo "[${BACKUP_DATE}] Verifying library location..." | tee -a "$LOG_FILE"
# Get the upload location from docker-compose volumes
UPLOAD_LOCATION="/srv/immich/library"
if [ -d "${UPLOAD_LOCATION}" ]; then
#LIBRARY_SIZE=$(du -sh ${UPLOAD_LOCATION} | cut -f1)
echo "[${BACKUP_DATE}] Library location verified: ${UPLOAD_LOCATION} (${LIBRARY_SIZE})" | tee -a "$LOG_FILE"
echo "[${BACKUP_DATE}] Kopia will backup library files directly (no tar, better deduplication)" | tee -a "$LOG_FILE"
else
echo "[${BACKUP_DATE}] WARNING: Upload location not found at ${UPLOAD_LOCATION}" | tee -a "$LOG_FILE"
fi
# Step 3: Backup configuration files
echo "[${BACKUP_DATE}] Backing up configuration files..." | tee -a "$LOG_FILE"
cp "${IMMICH_DIR}/docker-compose.yml" "${BACKUP_DIR}/immich-${BACKUP_DATE}/"
cp "${IMMICH_DIR}/.env" "${BACKUP_DIR}/immich-${BACKUP_DATE}/"
echo "[${BACKUP_DATE}] Configuration backup completed" | tee -a "$LOG_FILE"
# Step 4: Clean up old backups
echo "[${BACKUP_DATE}] Cleaning up backups older than ${KEEP_DAYS} days..." | tee -a "$LOG_FILE"
find "${BACKUP_DIR}" -maxdepth 1 -type d -name "immich-*" -mtime +${KEEP_DAYS} -exec rm -rf {} \; 2>&1 | tee -a "$LOG_FILE"
echo "[${BACKUP_DATE}] Local backup cleanup completed" | tee -a "$LOG_FILE"
# Step 5: Create Kopia snapshot of backup directory
echo "[${BACKUP_DATE}] Creating Kopia snapshot..." | tee -a "$LOG_FILE"
kopia snapshot create "${BACKUP_DIR}" \
--tags immich:tier1-backup \
--description "Immich backup ${BACKUP_DATE}" \
2>&1 | tee -a "$LOG_FILE"
KOPIA_EXIT=${PIPESTATUS[0]}
if [ $KOPIA_EXIT -ne 0 ]; then
echo "[${BACKUP_DATE}] WARNING: Kopia snapshot failed with exit code ${KOPIA_EXIT}" | tee -a "$LOG_FILE"
echo "[${BACKUP_DATE}] Local Immich backup exists but offsite copy may be incomplete" | tee -a "$LOG_FILE"
exit 2
fi
echo "[${BACKUP_DATE}] Kopia snapshot completed successfully" | tee -a "$LOG_FILE"
# Step 6: Backup the library directly with Kopia (better deduplication than tar)
echo "[${BACKUP_DATE}] Creating Kopia snapshot of library..." | tee -a "$LOG_FILE"
if [ -d "${UPLOAD_LOCATION}" ]; then
kopia snapshot create "${UPLOAD_LOCATION}" \
--tags immich:library \
--description "Immich library ${BACKUP_DATE}" \
2>&1 | tee -a "$LOG_FILE"
KOPIA_LIB_EXIT=${PIPESTATUS[0]}
if [ $KOPIA_LIB_EXIT -ne 0 ]; then
echo "[${BACKUP_DATE}] WARNING: Kopia library snapshot failed" | tee -a "$LOG_FILE"
else
echo "[${BACKUP_DATE}] Library snapshot completed successfully" | tee -a "$LOG_FILE"
fi
fi
# Step 7: Also backup the Immich installation directory (configs, compose files)
#echo "[${BACKUP_DATE}] Backing up Immich installation directory..." | tee -a "$LOG_FILE"
#kopia snapshot create "${IMMICH_DIR}" \
# --tags immich,config,docker-compose \
# --description "Immich config ${BACKUP_DATE}" \
# 2>&1 | tee -a "$LOG_FILE"
echo "[${BACKUP_DATE}] Backup process completed successfully" | tee -a "$LOG_FILE"
echo "[${BACKUP_DATE}] ========================================" | tee -a "$LOG_FILE"
# Optional: Send notification on completion
# Add your notification method here (email, webhook, etc.)
```
Make it executable:
```bash
chmod +x /opt/scripts/backup-immich.sh
```
Add to crontab (daily at 2 AM):
```bash
# Edit root's crontab
crontab -e
# Add this line:
0 2 * * * /opt/scripts/backup-immich.sh 2>&1 | logger -t immich-backup
```
### Offsite Backup to Vaults
After local Kopia snapshots are created, they sync to your offsite vaults automatically through Kopia's repository configuration.
## Recovery Procedures
### Understanding Two Recovery Methods
We have **two restore methods** depending on the scenario:
1. **Local Restore** (Preferred): For component-level or same-server recovery
2. **Kopia Full Restore**: For complete disaster recovery to a new server
### Method 1: Local Restore (Recommended)
Use this method when:
- Restoring on the same/similar server
- Restoring specific components (just database, just library, etc.)
- Recovering from local Immich backups
#### Full System Restore
```bash
cd /opt/immich
# Stop Immich
docker compose down
# List available backups
ls -lh /opt/immich-backups/
# Choose a database backup
BACKUP_PATH="/opt/immich-backups/immich-YYYYMMDD_HHMMSS"
# Restore database
gunzip < ${BACKUP_PATH}/dump.sql.gz | \
docker compose exec -T database psql --username=postgres --dbname=immich
# Restore library from Kopia
kopia snapshot list --tags library
kopia restore <library-snapshot-id> /srv/immich/library
# Fix permissions
chown -R 1000:1000 /srv/immich/library
# Restore configuration (review changes first)
cp ${BACKUP_PATH}/.env .env.restored
cp ${BACKUP_PATH}/docker-compose.yml docker-compose.yml.restored
# Start Immich
docker compose up -d
# Monitor logs
docker compose logs -f
```
#### Example: Restore Only Database
```bash
cd /opt/immich
# Stop Immich
docker compose down
# Start only database
docker compose up -d database
sleep 10
# Restore database from backup
BACKUP_PATH="/opt/immich-backups/immich-YYYYMMDD_HHMMSS"
gunzip < ${BACKUP_PATH}/dump.sql.gz | \
docker compose exec -T database psql --username=postgres --dbname=immich
# Start all services
docker compose down
docker compose up -d
# Verify
docker compose logs -f
```
#### Example: Restore Only Library
```bash
cd /opt/immich
# Stop Immich
docker compose down
# Restore library from Kopia
kopia snapshot list --tags library
kopia restore <library-snapshot-id> /srv/immich/library
# Fix permissions
chown -R 1000:1000 /srv/immich/library
# Start Immich
docker compose up -d
```
### Method 2: Complete Server Rebuild (Kopia Restore)
Use this when recovering to a completely new server or when local backups are unavailable.
#### Step 1: Prepare New Server
```bash
# Update system
apt update && apt upgrade -y
# Install Docker
curl -fsSL https://get.docker.com | sh
systemctl enable docker
systemctl start docker
# Install Docker Compose
apt install docker-compose-plugin -y
# Install Kopia
curl -s https://kopia.io/signing-key | sudo gpg --dearmor -o /usr/share/keyrings/kopia-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/kopia-keyring.gpg] https://packages.kopia.io/apt/ stable main" | sudo tee /etc/apt/sources.list.d/kopia.list
apt update
apt install kopia -y
# Create directory structure
mkdir -p /opt/immich
mkdir -p /opt/immich-backups
mkdir -p /srv/immich/library
mkdir -p /srv/immich/postgres
```
#### Step 2: Restore Kopia Repository
```bash
# Connect to your offsite vault
kopia repository connect server \
--url=https://192.168.5.10:51516 \
--override-username=admin \
--server-cert-fingerprint=696a4999f594b5273a174fd7cab677d8dd1628f9b9d27e557daa87103ee064b2
# List available snapshots
kopia snapshot list --tags immich
```
#### Step 3: Restore Configuration
```bash
# Find and restore the config snapshot
kopia snapshot list --tags config
# Restore to the Immich directory
kopia restore <snapshot-id> /opt/immich/
# Verify critical files
ls -la /opt/immich/.env
ls -la /opt/immich/docker-compose.yml
```
#### Step 4: Restore Immich Backups Directory
```bash
# Restore the entire backup directory from Kopia
kopia snapshot list --tags tier1-backup
# Restore the most recent backup
kopia restore <snapshot-id> /opt/immich-backups/
# Verify backups were restored
ls -la /opt/immich-backups/
```
#### Step 5: Restore Database and Library
```bash
cd /opt/immich
# Find the most recent backup
LATEST_BACKUP=$(ls -td /opt/immich-backups/immich-* | head -1)
echo "Restoring from: $LATEST_BACKUP"
# Start database container
docker compose up -d database
sleep 30
# Restore database
gunzip < ${LATEST_BACKUP}/dump.sql.gz | \
docker compose exec -T database psql --username=postgres --dbname=immich
# Restore library from Kopia
kopia snapshot list --tags library
kopia restore <library-snapshot-id> /srv/immich/library
# Fix permissions
chown -R 1000:1000 /srv/immich/library
```
#### Step 6: Start and Verify Immich
```bash
cd /opt/immich
# Pull latest images (or use versions from backup if preferred)
docker compose pull
# Start all services
docker compose up -d
# Monitor logs
docker compose logs -f
```
#### Step 7: Post-Restore Verification
```bash
# Check container status
docker compose ps
# Test web interface
curl -I http://localhost:2283
# Verify database
docker compose exec database psql -U postgres -d immich -c "SELECT COUNT(*) FROM users;"
# Check library storage
ls -lah /srv/immich/library/
```
### Scenario 2: Restore Individual User's Photos
To restore a single user's library without affecting others:
**Option A: Using Kopia Mount (Recommended)**
```bash
# Mount the Kopia snapshot
kopia snapshot list --tags library
mkdir -p /mnt/kopia-library
kopia mount <library-snapshot-id> /mnt/kopia-library &
# Find the user's directory (using user ID from database)
# User libraries are typically in: library/{user-uuid}/
USER_UUID="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
# Copy user's data back
rsync -av /mnt/kopia-library/${USER_UUID}/ \
/srv/immich/library/${USER_UUID}/
# Fix permissions
chown -R 1000:1000 /srv/immich/library/${USER_UUID}/
# Unmount
kopia unmount /mnt/kopia-library
# Restart Immich to recognize changes
cd /opt/immich
docker compose restart immich-server
```
**Option B: Selective Kopia Restore**
```bash
cd /opt/immich
docker compose down
# Restore just the specific user's directory
kopia snapshot list --tags library
USER_UUID="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
# Restore with path filter
kopia restore <library-snapshot-id> /srv/immich/library \
--snapshot-path="${USER_UUID}"
# Fix permissions
chown -R 1000:1000 /srv/immich/library/${USER_UUID}/
# Start Immich
docker compose up -d
```
### Scenario 3: Database Recovery Only
If only the database is corrupted but library data is intact:
```bash
cd /opt/immich
# Stop Immich
docker compose down
# Start only database
docker compose up -d database
sleep 30
# Restore from most recent backup
LATEST_BACKUP=$(ls -td /opt/immich-backups/immich-* | head -1)
gunzip < ${LATEST_BACKUP}/dump.sql.gz | \
docker compose exec -T database psql --username=postgres --dbname=immich
# Start all services
docker compose down
docker compose up -d
# Verify
docker compose logs -f
```
### Scenario 4: Configuration Recovery Only
If you only need to restore configuration files:
```bash
cd /opt/immich
# Find the most recent backup
LATEST_BACKUP=$(ls -td /opt/immich-backups/immich-* | head -1)
# Stop Immich
docker compose down
# Backup current config (just in case)
cp .env .env.pre-restore
cp docker-compose.yml docker-compose.yml.pre-restore
# Restore config from backup
cp ${LATEST_BACKUP}/.env ./
cp ${LATEST_BACKUP}/docker-compose.yml ./
# Restart
docker compose up -d
```
## Verification and Testing
### Regular Backup Verification
Perform monthly restore tests to ensure backups are valid:
```bash
# Test restore to temporary location
mkdir -p /tmp/backup-test
kopia snapshot list --tags immich
kopia restore <snapshot-id> /tmp/backup-test/
# Verify files exist and are readable
ls -lah /tmp/backup-test/
gunzip < /tmp/backup-test/immich-*/dump.sql.gz | head -100
# Cleanup
rm -rf /tmp/backup-test/
```
### Backup Monitoring Script
Create `/opt/scripts/check-immich-backup.sh`:
```bash
#!/bin/bash
# Check last backup age
LAST_BACKUP=$(ls -td /opt/immich-backups/immich-* 2>/dev/null | head -1)
if [ -z "$LAST_BACKUP" ]; then
echo "WARNING: No Immich backups found"
exit 1
fi
BACKUP_DATE=$(basename "$LAST_BACKUP" | sed 's/immich-//')
BACKUP_EPOCH=$(date -d "${BACKUP_DATE:0:8} ${BACKUP_DATE:9:2}:${BACKUP_DATE:11:2}:${BACKUP_DATE:13:2}" +%s 2>/dev/null)
if [ -z "$BACKUP_EPOCH" ]; then
echo "WARNING: Cannot parse backup date"
exit 1
fi
NOW=$(date +%s)
AGE_HOURS=$(( ($NOW - $BACKUP_EPOCH) / 3600 ))
if [ $AGE_HOURS -gt 26 ]; then
echo "WARNING: Last Immich backup is $AGE_HOURS hours old"
# Send alert (email, Slack, etc.)
exit 1
else
echo "OK: Last backup $AGE_HOURS hours ago"
fi
# Check Kopia snapshots
KOPIA_LAST=$(kopia snapshot list --tags immich --json 2>/dev/null | jq -r '.[0].startTime' 2>/dev/null)
if [ -n "$KOPIA_LAST" ]; then
echo "Last Kopia snapshot: $KOPIA_LAST"
else
echo "WARNING: Cannot verify Kopia snapshots"
fi
```
## Disaster Recovery Checklist
When disaster strikes, follow this checklist:
- [ ] Confirm scope of failure (server, storage, specific component)
- [ ] Gather server information (hostname, IP, DNS records)
- [ ] Access offsite backup vault
- [ ] Provision new server (if needed)
- [ ] Install Docker and dependencies
- [ ] Connect to Kopia repository
- [ ] Restore configurations first
- [ ] Restore database
- [ ] Restore library data
- [ ] Start services and verify
- [ ] Test photo viewing and uploads
- [ ] Verify user accounts and albums
- [ ] Update DNS records if needed
- [ ] Document any issues encountered
- [ ] Update recovery procedures based on experience
## Important Notes
1. **External Mounts**: Your setup has `/export/photos` and `/srv/NextCloud-AIO` mounted as external read-only sources. These are not backed up by this script - ensure they have their own backup strategy.
2. **Database Password**: The default database password in your .env is `postgres`. Change this to a secure random password for production use.
3. **Permissions**: Library files should be owned by UID 1000:1000 for Immich to access them properly:
```bash
chown -R 1000:1000 /srv/immich/library
```
4. **Testing**: Always test recovery procedures in a lab environment before trusting them in production.
5. **Documentation**: Keep this guide and server details in a separate location (printed copy, password manager, etc.).
6. **Retention Policy**: Review Kopia retention settings periodically to balance storage costs with recovery needs.
## Backup Architecture Notes
### Why Two Backup Layers?
**Immich Native Backups** (Tier 1):
- ✅ Uses official Immich backup method (`pg_dump`)
- ✅ Fast, component-aware backups
- ✅ Selective restore (can restore just database or just library)
- ✅ Standard PostgreSQL format (portable)
- ❌ No deduplication (full copies each time)
- ❌ Limited to local storage initially
**Kopia Snapshots** (Tier 2):
- ✅ Deduplication and compression
- ✅ Efficient offsite replication to vaults
- ✅ Point-in-time recovery across multiple versions
- ✅ Disaster recovery to completely new infrastructure
- ❌ Less component-aware (treats as files)
- ❌ Slower for granular component restore
### Storage Efficiency
Using this two-tier approach:
- **Local**: Database backups (~7 days retention, relatively small)
- **Kopia**: Database backups + library (efficient deduplication)
**Why library goes directly to Kopia without tar:**
Example with 500GB library, adding 10GB photos/month:
**With tar approach:**
- Month 1: Backup 500GB tar
- Month 2: Add 10GB photos → Entire 510GB tar changes → Backup 510GB
- Month 3: Add 10GB photos → Entire 520GB tar changes → Backup 520GB
- **Total storage needed**: 500 + 510 + 520 = 1,530GB
**Without tar (Kopia direct):**
- Month 1: Backup 500GB
- Month 2: Add 10GB photos → Kopia only backs up the 10GB new files
- Month 3: Add 10GB photos → Kopia only backs up the 10GB new files
- **Total storage needed**: 500 + 10 + 10 = 520GB
**Savings**: ~66% reduction in storage and backup time!
This is why we:
- Keep database dumps local (small, fast component restore)
- Let Kopia handle library directly (efficient, incremental, deduplicated)
### Compression and Deduplication
**Database backups** use `gzip` compression:
- Typically 80-90% compression ratio for SQL dumps
- Small enough to keep local copies
**Library backups** use Kopia's built-in compression and deduplication:
- Photos (JPEG/HEIC): Already compressed, Kopia skips re-compression
- Videos: Already compressed, minimal additional compression
- RAW files: Some compression possible
- **Deduplication**: If you upload the same photo twice, Kopia stores it once
- **Block-level dedup**: Even modified photos share unchanged blocks
This is far more efficient than tar + gzip, which would:
- Compress already-compressed photos (wasted CPU, minimal benefit)
- Store entire archive even if only 1 file changed
- Prevent deduplication across backups
## Additional Resources
- [Immich Official Backup Documentation](https://immich.app/docs/administration/backup-and-restore)
- [Kopia Documentation](https://kopia.io/docs/)
- [Docker Volume Backup Best Practices](https://docs.docker.com/storage/volumes/#back-up-restore-or-migrate-data-volumes)
- [PostgreSQL pg_dump Documentation](https://www.postgresql.org/docs/current/app-pgdump.html)
## Revision History
| Date | Version | Changes |
|------|---------|---------|
| 2026-02-13 | 1.0 | Initial documentation - two-tier backup strategy using Immich's native backup method |
---
**Last Updated**: February 13, 2026
**Maintained By**: System Administrator
**Review Schedule**: Quarterly

View file

@ -0,0 +1,879 @@
---
title: Mailcow Backup and Restore Strategy
description: Mailcow backup
published: true
date: 2026-02-20T04:15:25.924Z
tags:
editor: markdown
dateCreated: 2026-02-11T01:20:59.127Z
---
# Mailcow Backup and Recovery Guide
## Overview
This document provides comprehensive backup and recovery procedures for Mailcow email server. Since Mailcow is **not running on ZFS or BTRFS**, snapshots are not available and we rely on Mailcow's native backup script combined with Kopia for offsite storage in vaults.
## Quick Reference
### Common Backup Commands
```bash
# Run a manual backup (all components)
cd /opt/mailcow-dockerized
MAILCOW_BACKUP_LOCATION=/opt/mailcow-backups \
./helper-scripts/backup_and_restore.sh backup all --delete-days 7
# Backup with multithreading (faster)
THREADS=4 MAILCOW_BACKUP_LOCATION=/opt/mailcow-backups \
./helper-scripts/backup_and_restore.sh backup all --delete-days 7
# List Kopia snapshots
kopia snapshot list --tags mailcow
# View backup logs
tail -f /var/log/mailcow-backup.log
```
### Common Restore Commands
```bash
# Restore using mailcow native script (interactive)
cd /opt/mailcow-dockerized
./helper-scripts/backup_and_restore.sh restore
# Restore from Kopia to new server
kopia snapshot list --tags tier1-backup
kopia restore <snapshot-id> /opt/mailcow-backups/
# Check container status after restore
docker compose ps
docker compose logs -f
```
## Critical Components to Backup
### 1. Docker Compose File
- **Location**: `/opt/mailcow-dockerized/docker-compose.yml` (or your installation path)
- **Purpose**: Defines all containers, networks, and volumes
- **Importance**: Critical for recreating the exact container configuration
### 2. Configuration Files
- **Primary Config**: `/opt/mailcow-dockerized/mailcow.conf`
- **Additional Configs**:
- `/opt/mailcow-dockerized/data/conf/` (all subdirectories)
- Custom SSL certificates if not using Let's Encrypt
- Any override files (e.g., `docker-compose.override.yml`)
### 3. Database
- **MySQL/MariaDB Data**: Contains all mailbox configurations, users, domains, aliases, settings
- **Docker Volume**: `mailcowdockerized_mysql-vol`
- **Container Path**: `/var/lib/mysql`
### 4. Email Data
- **Maildir Storage**: All actual email messages
- **Docker Volume**: `mailcowdockerized_vmail-vol`
- **Container Path**: `/var/vmail`
- **Size**: Typically the largest component
### 5. Additional Important Data
- **Redis Data**: `mailcowdockerized_redis-vol` (cache and sessions)
- **Rspamd Data**: `mailcowdockerized_rspamd-vol` (spam learning)
- **Crypt Data**: `mailcowdockerized_crypt-vol` (if using mailbox encryption)
- **Postfix Queue**: `mailcowdockerized_postfix-vol` (queued/deferred mail)
## Backup Strategy
### Two-Tier Backup Approach
We use a **two-tier approach** combining Mailcow's native backup script with Kopia for offsite storage:
1. **Tier 1 (Local)**: Mailcow's `backup_and_restore.sh` script creates consistent, component-level backups
2. **Tier 2 (Offsite)**: Kopia snapshots the local backups and syncs to vaults
#### Why This Approach?
- **Best of both worlds**: Native script ensures mailcow-specific consistency, Kopia provides deduplication and offsite protection
- **Component-level restore**: Can restore individual components (just vmail, just mysql, etc.) using mailcow script
- **Disaster recovery**: Full system restore from Kopia backups on new server
- **Efficient storage**: Kopia's deduplication reduces storage needs for offsite copies
#### Backup Frequency
- **Daily**: Mailcow native backup runs at 2 AM
- **Daily**: Kopia snapshot of backups runs at 3 AM
- **Retention (Local)**: 7 days of mailcow backups (managed by script)
- **Retention (Kopia/Offsite)**: 30 daily, 12 weekly, 12 monthly
### Mailcow Native Backup Script
Mailcow includes `/opt/mailcow-dockerized/helper-scripts/backup_and_restore.sh` which handles:
- **vmail**: Email data (mailboxes)
- **mysql**: Database (using mariabackup for consistency)
- **redis**: Redis database
- **rspamd**: Spam filter learning data
- **crypt**: Encryption data
- **postfix**: Mail queue
**Key Features:**
- Uses `mariabackup` (hot backup without stopping MySQL)
- Supports multithreading for faster backups
- Architecture-aware (handles x86/ARM differences)
- Built-in cleanup with `--delete-days` parameter
- Creates compressed archives (.tar.zst or .tar.gz)
### Setting Up Mailcow Backups
#### Prereq:
Make sure you are connected to the repository,
```bash
sudo kopia repository connect server --url=https://192.168.5.10:51516 --override-username=admin --server-cert-fingerprint=696a4999f594b5273a174fd7cab677d8dd1628f9b9d27e557daa87103ee064b2
```
#### Step 1: Configure Backup Location
Set the backup destination via environment variable or in mailcow.conf:
```bash
# Option 1: Set environment variable (preferred for automation)
export MAILCOW_BACKUP_LOCATION="/opt/mailcow-backups"
# Option 2: Add to cron job directly (shown in automated script below)
```
Create the backup directory:
```bash
mkdir -p /opt/mailcow-backups
chown -R root:root /opt/mailcow-backups
chmod 777 /opt/mailcow-backups
```
#### Step 2: Manual Backup Commands
```bash
cd /opt/mailcow-dockerized
# Backup all components, delete backups older than 7 days
MAILCOW_BACKUP_LOCATION=/opt/mailcow-backups \
./helper-scripts/backup_and_restore.sh backup all --delete-days 7
# Backup with multithreading (faster for large mailboxes)
THREADS=4 MAILCOW_BACKUP_LOCATION=/opt/mailcow-backups \
./helper-scripts/backup_and_restore.sh backup all --delete-days 7
# Backup specific components only
MAILCOW_BACKUP_LOCATION=/opt/mailcow-backups \
./helper-scripts/backup_and_restore.sh backup vmail mysql --delete-days 7
```
**What gets created:**
- Backup directory: `/opt/mailcow-backups/mailcow-YYYY-MM-DD-HH-MM-SS/`
- Contains: `.tar.zst` compressed archives for each component
- Plus: `mailcow.conf` copy for restore reference
#### Step 3: Automated Backup Script
Create `/opt/scripts/backup-mailcow.sh`:
```bash
#!/bin/bash
# Mailcow Automated Backup Script
# This creates mailcow native backups, then snapshots them with Kopia for offsite storage
set -e
BACKUP_DATE=$(date +%Y%m%d_%H%M%S)
LOG_FILE="/var/log/mailcow-backup.log"
MAILCOW_DIR="/opt/mailcow-dockerized"
BACKUP_DIR="/opt/mailcow-backups"
THREADS=4 # Adjust based on your CPU cores
KEEP_DAYS=7 # Keep local mailcow backups for 7 days
echo "[${BACKUP_DATE}] ========================================" | tee -a "$LOG_FILE"
echo "[${BACKUP_DATE}] Starting Mailcow backup process" | tee -a "$LOG_FILE"
# Step 1: Run mailcow's native backup script
echo "[${BACKUP_DATE}] Running mailcow native backup..." | tee -a "$LOG_FILE"
cd "$MAILCOW_DIR"
# Run the backup with multithreading
THREADS=${THREADS} MAILCOW_BACKUP_LOCATION=${BACKUP_DIR} \
./helper-scripts/backup_and_restore.sh backup all --delete-days ${KEEP_DAYS} \
2>&1 | tee -a "$LOG_FILE"
BACKUP_EXIT=${PIPESTATUS[0]}
if [ $BACKUP_EXIT -ne 0 ]; then
echo "[${BACKUP_DATE}] ERROR: Mailcow backup failed with exit code ${BACKUP_EXIT}" | tee -a "$LOG_FILE"
exit 1
fi
echo "[${BACKUP_DATE}] Mailcow native backup completed successfully" | tee -a "$LOG_FILE"
# Step 2: Create Kopia snapshot of backup directory
echo "[${BACKUP_DATE}] Creating Kopia snapshot..." | tee -a "$LOG_FILE"
kopia snapshot create "${BACKUP_DIR}" \
--tags mailcow:tier1-backup \
--description "Mailcow backup ${BACKUP_DATE}" \
2>&1 | tee -a "$LOG_FILE"
KOPIA_EXIT=${PIPESTATUS[0]}
if [ $KOPIA_EXIT -ne 0 ]; then
echo "[${BACKUP_DATE}] WARNING: Kopia snapshot failed with exit code ${KOPIA_EXIT}" | tee -a "$LOG_FILE"
echo "[${BACKUP_DATE}] Local mailcow backup exists but offsite copy may be incomplete" | tee -a "$LOG_FILE"
exit 2
fi
echo "[${BACKUP_DATE}] Kopia snapshot completed successfully" | tee -a "$LOG_FILE"
# Step 3: Also backup the mailcow installation directory (configs, compose files)
echo "[${BACKUP_DATE}] Backing up mailcow installation directory..." | tee -a "$LOG_FILE"
kopia snapshot create "${MAILCOW_DIR}" \
--tags mailcow,config,docker-compose \
--description "Mailcow config ${BACKUP_DATE}" \
2>&1 | tee -a "$LOG_FILE"
echo "[${BACKUP_DATE}] Backup process completed successfully" | tee -a "$LOG_FILE"
echo "[${BACKUP_DATE}] ========================================" | tee -a "$LOG_FILE"
# Optional: Send notification on completion
# Add your notification method here (email, webhook, etc.)
```
Make it executable:
```bash
chmod +x /opt/scripts/backup-mailcow.sh
```
Add to crontab (daily at 2 AM):
```bash
# Edit root's crontab
crontab -e
# Add this line:
0 2 * * * /opt/scripts/backup-mailcow.sh 2>&1 | logger -t mailcow-backup
```
### Offsite Backup to Vaults
After local Kopia snapshots are created, sync to your offsite vaults:
```bash
# Option 1: Kopia repository sync (if using multiple Kopia repos)
kopia repository sync-to filesystem --path /mnt/vault/mailcow-backup
# Option 2: Rsync to vault
rsync -avz --delete /backup/kopia-repo/ /mnt/vault/mailcow-backup/
# Option 3: Rclone to remote vault
rclone sync /backup/kopia-repo/ vault:mailcow-backup/
```
## Recovery Procedures
### Understanding Two Recovery Methods
We have **two restore methods** depending on the scenario:
1. **Mailcow Native Restore** (Preferred): For component-level or same-server recovery
2. **Kopia Full Restore**: For complete disaster recovery to a new server
### Method 1: Mailcow Native Restore (Recommended)
Use this method when:
- Restoring on the same/similar server
- Restoring specific components (just email, just database, etc.)
- Recovering from local mailcow backups
#### Step 1: List Available Backups
```bash
cd /opt/mailcow-dockerized
# Run the restore script
./helper-scripts/backup_and_restore.sh restore
```
The script will prompt:
```
Backup location (absolute path, starting with /): /opt/mailcow-backups
```
#### Step 2: Select Backup
The script displays available backups:
```
Found project name mailcowdockerized
[ 1 ] - /opt/mailcow-backups/mailcow-2026-02-09-02-00-14/
[ 2 ] - /opt/mailcow-backups/mailcow-2026-02-10-02-00-08/
```
Enter the number of the backup to restore.
#### Step 3: Select Components
Choose what to restore:
```
[ 0 ] - all
[ 1 ] - Crypt data
[ 2 ] - Rspamd data
[ 3 ] - Mail directory (/var/vmail)
[ 4 ] - Redis DB
[ 5 ] - Postfix data
[ 6 ] - SQL DB
```
**Important**: The script will:
- Stop mailcow containers automatically
- Restore selected components
- Handle permissions correctly
- Restart containers when done
#### Example: Restore Only Email Data
```bash
cd /opt/mailcow-dockerized
./helper-scripts/backup_and_restore.sh restore
# When prompted:
# - Backup location: /opt/mailcow-backups
# - Select backup: 2 (most recent)
# - Select component: 3 (Mail directory)
```
#### Example: Restore Database Only
```bash
cd /opt/mailcow-dockerized
./helper-scripts/backup_and_restore.sh restore
# When prompted:
# - Backup location: /opt/mailcow-backups
# - Select backup: 2 (most recent)
# - Select component: 6 (SQL DB)
```
**Note**: For database restore, the script will modify `mailcow.conf` with the database credentials from the backup. Review the changes after restore.
### Method 2: Complete Server Rebuild (Kopia Restore)
Use this when recovering to a completely new server or when local backups are unavailable.
#### Step 1: Prepare New Server
```bash
# Update system
apt update && apt upgrade -y
# Install Docker
curl -fsSL https://get.docker.com | sh
systemctl enable docker
systemctl start docker
# Install Docker Compose
apt install docker-compose-plugin -y
# Install Kopia
curl -s https://kopia.io/signing-key | apt-key add -
echo "deb https://packages.kopia.io/apt/ stable main" | tee /etc/apt/sources.list.d/kopia.list
apt update
apt install kopia -y
# Create directory structure
mkdir -p /opt/mailcow-dockerized
mkdir -p /opt/mailcow-backups/database
```
#### Step 2: Restore Kopia Repository
```bash
# Connect to your offsite vault
# If vault is mounted:
kopia repository connect filesystem --path /mnt/vault/mailcow-backup
# If vault is remote:
kopia repository connect s3 --bucket=your-bucket --access-key=xxx --secret-access-key=xxx
# List available snapshots
kopia snapshot list --tags mailcow
```
#### Step 3: Restore Configuration
```bash
# Find and restore the config snapshot
kopia snapshot list --tags config
# Restore to the Mailcow directory
kopia restore <snapshot-id> /opt/mailcow-dockerized/
# Verify critical files
ls -la /opt/mailcow-dockerized/mailcow.conf
ls -la /opt/mailcow-dockerized/docker-compose.yml
```
#### Step 4: Restore Mailcow Backups Directory
```bash
# Restore the entire backup directory from Kopia
kopia snapshot list --tags tier1-backup
# Restore the most recent backup
kopia restore <snapshot-id> /opt/mailcow-backups/
# Verify backups were restored
ls -la /opt/mailcow-backups/
```
#### Step 5: Run Mailcow Native Restore
Now use mailcow's built-in restore script:
```bash
cd /opt/mailcow-dockerized
# Run the restore script
./helper-scripts/backup_and_restore.sh restore
# When prompted:
# - Backup location: /opt/mailcow-backups
# - Select the most recent backup
# - Select [ 0 ] - all (to restore everything)
```
The script will:
1. Stop all mailcow containers
2. Restore all components (vmail, mysql, redis, rspamd, postfix, crypt)
3. Update mailcow.conf with restored database credentials
4. Restart all containers
**Alternative: Manual Restore** (if you prefer more control)
```bash
cd /opt/mailcow-dockerized
# Start containers to create volumes
docker compose up -d --no-start
docker compose down
# Find the most recent backup directory
LATEST_BACKUP=$(ls -td /opt/mailcow-backups/mailcow-* | head -1)
echo "Restoring from: $LATEST_BACKUP"
# Extract each component manually
cd "$LATEST_BACKUP"
# Restore vmail (email data)
docker run --rm \
-v mailcowdockerized_vmail-vol:/backup \
-v "$PWD":/restore \
debian:bookworm-slim \
tar --use-compress-program='zstd -d' -xvf /restore/backup_vmail.tar.zst
# Restore MySQL
docker run --rm \
-v mailcowdockerized_mysql-vol:/backup \
-v "$PWD":/restore \
mariadb:10.11 \
tar --use-compress-program='zstd -d' -xvf /restore/backup_mysql.tar.zst
# Restore Redis
docker run --rm \
-v mailcowdockerized_redis-vol:/backup \
-v "$PWD":/restore \
debian:bookworm-slim \
tar --use-compress-program='zstd -d' -xvf /restore/backup_redis.tar.zst
# Restore other components similarly (rspamd, postfix, crypt)
# ...
# Copy mailcow.conf from backup
cp "$LATEST_BACKUP/mailcow.conf" /opt/mailcow-dockerized/mailcow.conf
```
#### Step 6: Start and Verify Mailcow
```bash
cd /opt/mailcow-dockerized
# Pull latest images (or use versions from backup if preferred)
docker compose pull
# Start all services
docker compose up -d
# Monitor logs
docker compose logs -f
```
#### Step 7: Post-Restore Verification
```bash
# Check container status
docker compose ps
# Test web interface
curl -I https://mail.yourdomain.com
# Check mail log
docker compose logs -f postfix-mailcow
# Verify database
docker compose exec mysql-mailcow mysql -u root -p$(grep DBROOT mailcow.conf | cut -d'=' -f2) -e "SHOW DATABASES;"
# Check email storage
docker compose exec dovecot-mailcow ls -lah /var/vmail/
```
### Scenario 2: Restore Individual Mailbox
To restore a single user's mailbox without affecting others:
#### Option A: Using Mailcow Backups (If Available)
```bash
cd /opt/mailcow-dockerized
# Temporarily mount the backup
BACKUP_DIR="/opt/mailcow-backups/mailcow-YYYY-MM-DD-HH-MM-SS"
# Extract just the vmail archive to a temporary location
mkdir -p /tmp/vmail-restore
cd "$BACKUP_DIR"
tar --use-compress-program='zstd -d' -xvf backup_vmail.tar.zst -C /tmp/vmail-restore
# Find the user's mailbox
# Structure: /tmp/vmail-restore/var/vmail/domain.com/user/
ls -la /tmp/vmail-restore/var/vmail/yourdomain.com/
# Copy specific mailbox
rsync -av /tmp/vmail-restore/var/vmail/yourdomain.com/user@domain.com/ \
/var/lib/docker/volumes/mailcowdockerized_vmail-vol/_data/yourdomain.com/user@domain.com/
# Fix permissions
docker run --rm \
-v mailcowdockerized_vmail-vol:/vmail \
debian:bookworm-slim \
chown -R 5000:5000 /vmail/yourdomain.com/user@domain.com/
# Cleanup
rm -rf /tmp/vmail-restore
# Restart Dovecot to recognize changes
docker compose restart dovecot-mailcow
```
#### Option B: Using Kopia Snapshot (If Local Backups Unavailable)
```bash
# Mount the vmail snapshot temporarily
mkdir -p /mnt/restore
kopia mount <vmail-snapshot-id> /mnt/restore
# Find the user's mailbox
# Structure: /mnt/restore/domain.com/user/
ls -la /mnt/restore/yourdomain.com/
# Copy specific mailbox
rsync -av /mnt/restore/yourdomain.com/user@domain.com/ \
/var/lib/docker/volumes/mailcowdockerized_vmail-vol/_data/yourdomain.com/user@domain.com/
# Fix permissions
chown -R 5000:5000 /var/lib/docker/volumes/mailcowdockerized_vmail-vol/_data/yourdomain.com/user@domain.com/
# Unmount
kopia unmount /mnt/restore
# Restart Dovecot to recognize changes
docker compose restart dovecot-mailcow
```
### Scenario 3: Database Recovery Only
If only the database is corrupted but email data is intact:
#### Option A: Using Mailcow Native Restore (Recommended)
```bash
cd /opt/mailcow-dockerized
# Run the restore script
./helper-scripts/backup_and_restore.sh restore
# When prompted:
# - Backup location: /opt/mailcow-backups
# - Select the most recent backup
# - Select [ 6 ] - SQL DB (database only)
```
The script will:
1. Stop mailcow
2. Restore the MySQL database from the mariabackup archive
3. Update mailcow.conf with the restored database credentials
4. Restart mailcow
#### Option B: Manual Database Restore from Kopia
If local backups are unavailable:
```bash
cd /opt/mailcow-dockerized
# Stop Mailcow
docker compose down
# Start only MySQL
docker compose up -d mysql-mailcow
# Wait for MySQL
sleep 30
# Restore from Kopia database dump
kopia snapshot list --tags database
kopia restore <snapshot-id> /tmp/db-restore/
# Import the dump
LATEST_DUMP=$(ls -t /tmp/db-restore/mailcow_*.sql | head -1)
docker compose exec -T mysql-mailcow mysql -u root -p$(grep DBROOT mailcow.conf | cut -d'=' -f2) < "$LATEST_DUMP"
# Start all services
docker compose down
docker compose up -d
# Verify
docker compose logs -f
```
### Scenario 4: Configuration Recovery Only
If you only need to restore configuration files:
#### Option A: From Mailcow Backup
```bash
# Find the most recent backup
LATEST_BACKUP=$(ls -td /opt/mailcow-backups/mailcow-* | head -1)
# Stop Mailcow
cd /opt/mailcow-dockerized
docker compose down
# Backup current config (just in case)
cp mailcow.conf mailcow.conf.pre-restore
cp docker-compose.yml docker-compose.yml.pre-restore
# Restore mailcow.conf from backup
cp "$LATEST_BACKUP/mailcow.conf" ./mailcow.conf
# If you also need other config files from data/conf/,
# you would need to extract them from the backup archives
# Restart
docker compose up -d
```
#### Option B: From Kopia Snapshot
```bash
# Restore config snapshot to temporary location
kopia restore <config-snapshot-id> /tmp/mailcow-restore/
# Stop Mailcow
cd /opt/mailcow-dockerized
docker compose down
# Backup current config (just in case)
cp mailcow.conf mailcow.conf.pre-restore
cp docker-compose.yml docker-compose.yml.pre-restore
# Restore specific files
cp /tmp/mailcow-restore/mailcow.conf ./
cp /tmp/mailcow-restore/docker-compose.yml ./
cp -r /tmp/mailcow-restore/data/conf/* ./data/conf/
# Restart
docker compose up -d
```
## Verification and Testing
### Regular Backup Verification
Perform monthly restore tests to ensure backups are valid:
```bash
# Test restore to temporary location
mkdir -p /tmp/backup-test
kopia snapshot list --tags mailcow
kopia restore <snapshot-id> /tmp/backup-test/
# Verify files exist and are readable
ls -lah /tmp/backup-test/
cat /tmp/backup-test/mailcow.conf
# Cleanup
rm -rf /tmp/backup-test/
```
### Backup Monitoring Script
Create `/opt/scripts/check-mailcow-backup.sh`:
```bash
#!/bin/bash
# Check last backup age
LAST_BACKUP=$(kopia snapshot list --tags mailcow --json | jq -r '.[0].startTime')
LAST_BACKUP_EPOCH=$(date -d "$LAST_BACKUP" +%s)
NOW=$(date +%s)
AGE_HOURS=$(( ($NOW - $LAST_BACKUP_EPOCH) / 3600 ))
if [ $AGE_HOURS -gt 26 ]; then
echo "WARNING: Last Mailcow backup is $AGE_HOURS hours old"
# Send alert (email, Slack, etc.)
exit 1
else
echo "OK: Last backup $AGE_HOURS hours ago"
fi
```
## Disaster Recovery Checklist
When disaster strikes, follow this checklist:
- [ ] Confirm scope of failure (server, storage, specific component)
- [ ] Gather server information (hostname, IP, DNS records)
- [ ] Access offsite backup vault
- [ ] Provision new server (if needed)
- [ ] Install Docker and dependencies
- [ ] Connect to Kopia repository
- [ ] Restore configurations first
- [ ] Restore database
- [ ] Restore email data
- [ ] Start services and verify
- [ ] Test email sending/receiving
- [ ] Verify webmail access
- [ ] Check DNS records and update if needed
- [ ] Document any issues encountered
- [ ] Update recovery procedures based on experience
## Important Notes
1. **DNS**: Keep DNS records documented separately. Recovery includes updating DNS if server IP changes.
2. **SSL Certificates**: Let's Encrypt certificates are in the backup but may need renewal. Mailcow will handle this automatically.
3. **Permissions**: Docker volumes have specific UID/GID requirements:
- vmail: `5000:5000`
- mysql: `999:999`
4. **Testing**: Always test recovery procedures in a lab environment before trusting them in production.
5. **Documentation**: Keep this guide and server details in a separate location (printed copy, password manager, etc.).
6. **Retention Policy**: Review Kopia retention settings periodically to balance storage costs with recovery needs.
## Backup Architecture Notes
### Why Two Backup Layers?
**Mailcow Native Backups** (Tier 1):
- ✅ Component-aware (knows about mailcow's structure)
- ✅ Uses mariabackup for consistent MySQL hot backups
- ✅ Fast, selective restore (can restore just one component)
- ✅ Architecture-aware (handles x86/ARM differences)
- ❌ No deduplication (full copies each time)
- ❌ Limited to local storage initially
**Kopia Snapshots** (Tier 2):
- ✅ Deduplication and compression
- ✅ Efficient offsite replication to vaults
- ✅ Point-in-time recovery across multiple versions
- ✅ Disaster recovery to completely new infrastructure
- ❌ Less component-aware (treats as files)
- ❌ Slower for granular component restore
### Storage Efficiency
Using this two-tier approach:
- **Local**: Mailcow creates ~7 days of native backups (may be large, but short retention)
- **Offsite**: Kopia deduplicates these backups for long-term vault storage (much smaller)
Example storage calculation (10GB mailbox):
- Local: 7 days × 10GB = ~70GB (before compression)
- Kopia (offsite): First backup ~10GB, subsequent backups only store changes (might be <1GB/day after dedup)
### Compression Formats
Mailcow's script creates `.tar.zst` (Zstandard) or `.tar.gz` (gzip) files:
- **Zstandard** (modern): Better compression ratio, faster (recommended)
- **Gzip** (legacy): Wider compatibility with older systems
Verify your backup compression:
```bash
ls -lh /opt/mailcow-backups/mailcow-*/
# Look for .tar.zst (preferred) or .tar.gz
```
### Cross-Architecture Considerations
**Important for ARM/x86 Migration**:
Mailcow's backup script is architecture-aware. When restoring:
- **Rspamd data** cannot be restored across different architectures (x86 ↔ ARM)
- **All other components** (vmail, mysql, redis, postfix, crypt) are architecture-independent
If migrating between architectures:
```bash
# Restore everything EXCEPT rspamd
# Select components individually: vmail, mysql, redis, postfix, crypt
# Skip rspamd - it will rebuild its learning database over time
```
### Testing Your Backups
**Monthly Test Protocol**:
1. **Verify local backups exist**:
```bash
ls -lh /opt/mailcow-backups/
# Should see recent dated directories
```
2. **Verify Kopia snapshots**:
```bash
kopia snapshot list --tags mailcow
# Should see recent snapshots
```
3. **Test restore in lab** (recommended quarterly):
- Spin up a test VM
- Restore from Kopia
- Run mailcow native restore
- Verify email delivery and webmail access
## Additional Resources
- [Mailcow Official Backup Documentation](https://docs.mailcow.email/backup_restore/b_n_r-backup/)
- [Kopia Documentation](https://kopia.io/docs/)
- [Docker Volume Backup Best Practices](https://docs.docker.com/storage/volumes/#back-up-restore-or-migrate-data-volumes)
## Revision History
| Date | Version | Changes |
|------|---------|---------|
| 2026-02-10 | 1.1 | Integrated mailcow native backup_and_restore.sh script as primary backup method |
| 2026-02-10 | 1.0 | Initial documentation |
---
**Last Updated**: February 10, 2026
**Maintained By**: System Administrator
**Review Schedule**: Quarterly

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,19 @@
---
title: Services Backup
description:
published: true
date: 2026-02-20T04:08:15.923Z
tags:
editor: markdown
dateCreated: 2026-02-05T21:28:23.152Z
---
- [Mailcow](/backup-mailcow)
- [Immich](/immich_backup)
- [Nextcloud](/nextcloud_backup)
- kopia
- forgejo
- bitwarden
- wiki
- journalv

View file

@ -0,0 +1,567 @@
---
title: Wikijs Backup
description: Backup Wikijs
published: true
date: 2026-02-23T04:35:32.870Z
tags:
editor: markdown
dateCreated: 2026-02-23T04:35:24.121Z
---
# Wiki.js Backup & Recovery
**Service:** Wiki.js (Netgrimoire)
**Stack:** Docker Compose — Wiki.js + PostgreSQL
**Backup Targets:** PostgreSQL database dump, Git content repository, Docker Compose config
**Backup Destinations:** Local vault path → Kopia → offsite vaults
---
## Overview
Wiki.js data lives in two separate places that must be backed up independently:
**PostgreSQL database** — stores page metadata, navigation, user accounts, permissions, page history, assets, and all configuration. This is the critical component for a portable restore. Without it, a new instance has no knowledge of your wiki structure.
**Git content repository** — stores the actual page content in markdown files, synced from Forgejo. This is already mirrored on the VAULT SSD at `/vault/repos/wiki/`. It is inherently redundant as long as Forgejo is healthy, but is included in backups for completeness and offline portability.
**Docker Compose config** — the `docker-compose.yml` and `.env` files needed to recreate the stack.
---
## What Gets Backed Up
| Component | Location | Method | Critical? |
|---|---|---|---|
| PostgreSQL database | Docker volume | `pg_dump` → SQL file | Yes — primary restore target |
| Git content repo | `/vault/repos/wiki/` | Already on VAULT SSD | Yes — page content |
| Docker Compose files | `/opt/stacks/wikijs/` | rsync copy | Yes — stack config |
| Wiki.js data volume | Docker volume | Optional rsync | No — DB + Git covers this |
---
## Backup Strategy
### Tier 1 — Daily Dump to Vault Path
A script runs daily via systemd timer. It produces a portable `pg_dump` SQL file written to `/vault/backups/wiki/`. These local dumps are retained for 14 days.
**Key choices:**
- `--format=plain` — plain SQL, portable to any PostgreSQL version and any host
- `--no-owner` — strips role ownership, so the dump restores cleanly on a new instance with a different postgres user (critical for Pocket Grimoire restores)
- `--no-acl` — strips GRANT/REVOKE statements for the same reason
- No application downtime required — PostgreSQL handles consistent dumps natively
### Tier 2 — Kopia Snapshot to Offsite Vaults
After the daily dump completes, Kopia snapshots the entire `/vault/backups/wiki/` directory and replicates to your offsite vaults. Kopia deduplication means only changed blocks are transferred after the first run.
---
## Setup
### Step 0 — Confirm Kopia Repository Exists
If Kopia is not yet initialized on this host, initialize it first. If you already initialized Kopia for Mailcow or another service, skip this step — all services share the same Kopia repository.
```bash
# Check if repository already exists
kopia repository status
# If not initialized, create it against your vault path
kopia repository create filesystem --path=/vault/kopia
# Connect on subsequent logins if disconnected
kopia repository connect filesystem --path=/vault/kopia
```
### Step 1 — Create Backup Directories
```bash
sudo mkdir -p /vault/backups/wiki
sudo chown $(whoami):$(whoami) /vault/backups/wiki
```
### Step 2 — Create the Backup Script
```bash
sudo nano /usr/local/sbin/wikijs-backup.sh
```
```bash
#!/usr/bin/env bash
# wikijs-backup.sh — Daily Wiki.js backup: pg_dump + git repo + config
# Writes to /vault/backups/wiki/, then snapshots with Kopia
set -euo pipefail
# ── Configuration ─────────────────────────────────────────────────────────────
BACKUP_DIR="/vault/backups/wiki"
DATE=$(date +%Y%m%d_%H%M%S)
CONTAINER_DB="wikijs_db" # Adjust to your actual container name
PG_USER="wikijs"
PG_DB="wikijs"
WIKI_STACK_DIR="/opt/stacks/wikijs" # Location of docker-compose.yml and .env
GIT_REPO_DIR="/vault/repos/wiki" # Git content mirror (already on vault SSD)
RETAIN_DAYS=14 # Local dump retention
LOG="/var/log/wikijs-backup.log"
touch "$LOG"
log() { echo "$(date -Is) $*" | tee -a "$LOG"; }
# ── Step 1: PostgreSQL dump ────────────────────────────────────────────────────
log "Starting Wiki.js PostgreSQL dump..."
docker exec "$CONTAINER_DB" pg_dump \
-U "$PG_USER" \
"$PG_DB" \
--format=plain \
--no-owner \
--no-acl \
> "${BACKUP_DIR}/wikijs-db-${DATE}.sql"
gzip "${BACKUP_DIR}/wikijs-db-${DATE}.sql"
log "PostgreSQL dump complete: wikijs-db-${DATE}.sql.gz"
# ── Step 2: Docker Compose config backup ──────────────────────────────────────
log "Backing up Docker Compose config..."
CONFIG_BACKUP="${BACKUP_DIR}/wikijs-config-${DATE}.tar.gz"
tar -czf "$CONFIG_BACKUP" \
-C "$(dirname "$WIKI_STACK_DIR")" \
"$(basename "$WIKI_STACK_DIR")"
log "Config backup complete: wikijs-config-${DATE}.tar.gz"
# ── Step 3: Git repo snapshot (content mirror) ────────────────────────────────
# The git repo lives on the VAULT SSD and is already versioned.
# We record the current HEAD commit for reference.
if [ -d "${GIT_REPO_DIR}/.git" ]; then
GIT_HEAD=$(git -C "$GIT_REPO_DIR" rev-parse HEAD 2>/dev/null || echo "unknown")
echo "Git HEAD at backup time: ${GIT_HEAD}" \
> "${BACKUP_DIR}/wikijs-git-ref-${DATE}.txt"
log "Git content repo HEAD: ${GIT_HEAD}"
else
log "WARNING: Git repo not found at ${GIT_REPO_DIR} — skipping git ref"
fi
# ── Step 4: Cleanup old local dumps ───────────────────────────────────────────
log "Cleaning up dumps older than ${RETAIN_DAYS} days..."
find "$BACKUP_DIR" -name "wikijs-db-*.sql.gz" -mtime +"$RETAIN_DAYS" -delete
find "$BACKUP_DIR" -name "wikijs-config-*.tar.gz" -mtime +"$RETAIN_DAYS" -delete
find "$BACKUP_DIR" -name "wikijs-git-ref-*.txt" -mtime +"$RETAIN_DAYS" -delete
# ── Step 5: Kopia snapshot ────────────────────────────────────────────────────
log "Running Kopia snapshot of /vault/backups/wiki/..."
kopia snapshot create "$BACKUP_DIR" \
--tags "service:wikijs,host:$(hostname -s)"
log "Kopia snapshot complete."
# ── Done ──────────────────────────────────────────────────────────────────────
log "Wiki.js backup finished successfully."
```
```bash
sudo chmod +x /usr/local/sbin/wikijs-backup.sh
```
### Step 3 — Create systemd Service and Timer
```bash
sudo nano /etc/systemd/system/wikijs-backup.service
```
```ini
[Unit]
Description=Wiki.js daily backup (pg_dump + config + Kopia snapshot)
After=docker.service
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/wikijs-backup.sh
```
```bash
sudo nano /etc/systemd/system/wikijs-backup.timer
```
```ini
[Unit]
Description=Run Wiki.js backup daily at 02:00
[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true
[Install]
WantedBy=timers.target
```
```bash
sudo systemctl daemon-reload
sudo systemctl enable wikijs-backup.timer
sudo systemctl start wikijs-backup.timer
# Verify
systemctl list-timers | grep wikijs
```
### Step 4 — Configure Kopia Retention Policy
```bash
# Set retention policy for wiki backups
kopia policy set /vault/backups/wiki \
--keep-daily 14 \
--keep-weekly 8 \
--keep-monthly 12 \
--compression zstd
# Verify policy
kopia policy show /vault/backups/wiki
```
### Step 5 — Test the Backup
```bash
# Run manually first time
sudo /usr/local/sbin/wikijs-backup.sh
# Verify output
ls -lh /vault/backups/wiki/
# Should show: wikijs-db-YYYYMMDD_HHMMSS.sql.gz
# wikijs-config-YYYYMMDD_HHMMSS.tar.gz
# wikijs-git-ref-YYYYMMDD_HHMMSS.txt
# Verify Kopia snapshot was created
kopia snapshot list /vault/backups/wiki
# Check backup log
tail -n 30 /var/log/wikijs-backup.log
```
---
## Verifying Backups
### Check dump is readable
```bash
# Inspect the SQL dump without extracting
zcat /vault/backups/wiki/wikijs-db-YYYYMMDD_HHMMSS.sql.gz | head -50
# Should show PostgreSQL header, version info, and CREATE TABLE statements
```
### Verify Kopia snapshots
```bash
# List recent snapshots
kopia snapshot list /vault/backups/wiki
# Show snapshot details
kopia snapshot list /vault/backups/wiki --all
# Verify snapshot integrity
kopia snapshot verify
```
### Test restore to a temporary database (non-destructive)
```bash
# Start a temporary Postgres container
docker run --rm -d \
--name wikijs-restore-test \
-e POSTGRES_USER=wikijs \
-e POSTGRES_PASSWORD=testpassword \
-e POSTGRES_DB=wikijs_test \
postgres:16-alpine
# Wait for Postgres to be ready
sleep 5
# Restore dump into test container
zcat /vault/backups/wiki/wikijs-db-YYYYMMDD_HHMMSS.sql.gz | \
docker exec -i wikijs-restore-test psql -U wikijs -d wikijs_test
# Verify tables exist
docker exec wikijs-restore-test psql -U wikijs -d wikijs_test -c "\dt"
# Expected output: List of tables (pages, users, pageHistory, assets, etc.)
# Cleanup test container
docker stop wikijs-restore-test
```
---
## Recovery Procedures
### Scenario A — Restore to a New Wiki.js Instance (Any Host)
This covers full disaster recovery to a fresh server, including Pocket Grimoire.
**Requirements on the destination host:**
- Docker and Docker Compose installed
- A `docker-compose.yml` and `.env` ready (from backup or Pocket Grimoire stack)
- Sufficient disk space
**Step 1: Locate the backup**
```bash
# On Netgrimoire, find the dump to restore
ls -lh /vault/backups/wiki/
# Or restore from Kopia
kopia snapshot list /vault/backups/wiki
kopia restore SNAPSHOT_ID /tmp/wiki-restore/
ls /tmp/wiki-restore/
```
**Step 2: Copy dump to the destination host**
```bash
# From Netgrimoire, copy to the destination server
scp /vault/backups/wiki/wikijs-db-YYYYMMDD_HHMMSS.sql.gz \
user@destination-host:/tmp/
# Or to Pocket Grimoire
scp /vault/backups/wiki/wikijs-db-YYYYMMDD_HHMMSS.sql.gz \
user@pocket-grimoire.local:/tmp/
```
**Step 3: Start the database container only**
On the destination host, start just the database — do not start Wiki.js yet:
```bash
cd /srv/pocket-grimoire/stacks/wikijs # Adjust path as needed
# Start only the database container
docker compose up -d db
# Wait for healthy status
docker compose ps
# db should show: healthy
```
**Step 4: Restore the dump**
```bash
# Restore the dump into the running database container
zcat /tmp/wikijs-db-YYYYMMDD_HHMMSS.sql.gz | \
docker exec -i pocketgrimoire_db psql \
-U wikijs \
-d wikijs
# Verify tables restored
docker exec pocketgrimoire_db psql -U wikijs -d wikijs -c "\dt"
```
**Step 5: Start Wiki.js**
```bash
docker compose up -d
# Watch startup logs
docker logs -f pocketgrimoire_wikijs
# Wait for: "HTTP Server started successfully"
```
**Step 6: Verify**
Open `http://pocket-grimoire.local:3000` and confirm:
- Pages load correctly
- Navigation structure is intact
- User accounts are present (if you had multiple users)
**Step 7: Re-sync Git content (if needed)**
The database knows the page structure, but if the Git content repo isn't present on the new host, import it:
```bash
# In Wiki.js admin panel:
# Administration → Storage → Git
# Click "Force Sync" or "Import Content"
# Or copy the repo from VAULT SSD
rsync -avP /vault/repos/wiki/ /srv/pocket-grimoire/repos/wiki/
```
---
### Scenario B — Restore on Existing Netgrimoire Instance
Use this when the Wiki.js database is corrupted but the host is otherwise healthy.
**Step 1: Stop Wiki.js (leave database running)**
```bash
cd /opt/stacks/wikijs
docker compose stop wikijs
```
**Step 2: Drop and recreate the database**
```bash
docker exec -it wikijs_db psql -U postgres -c "DROP DATABASE wikijs;"
docker exec -it wikijs_db psql -U postgres -c "CREATE DATABASE wikijs OWNER wikijs;"
```
**Step 3: Restore**
```bash
zcat /vault/backups/wiki/wikijs-db-YYYYMMDD_HHMMSS.sql.gz | \
docker exec -i wikijs_db psql -U wikijs -d wikijs
```
**Step 4: Restart Wiki.js**
```bash
docker compose start wikijs
docker logs -f wikijs
```
---
### Scenario C — Restore Config Only
If the stack config was lost but the database volume is intact:
```bash
# Extract config from backup
tar -xzf /vault/backups/wiki/wikijs-config-YYYYMMDD_HHMMSS.tar.gz \
-C /opt/stacks/
# Verify
ls /opt/stacks/wikijs/
# Should show: docker-compose.yml .env
# Restart stack
cd /opt/stacks/wikijs
docker compose up -d
```
---
### Restore from Kopia (Offsite)
When local vault files are unavailable, restore the backup directory from Kopia first:
```bash
# List available snapshots
kopia snapshot list /vault/backups/wiki
# Restore snapshot to temp directory
kopia restore SNAPSHOT_ID /tmp/wiki-restore/
# Then proceed with the appropriate scenario above
# using files from /tmp/wiki-restore/ instead of /vault/backups/wiki/
```
---
## Pocket Grimoire Specifics
When restoring to Pocket Grimoire, note the following differences from a full Netgrimoire instance:
**Container names** differ — use `pocketgrimoire_db` instead of `wikijs_db`.
**Stack path** is `/srv/pocket-grimoire/stacks/wikijs/` instead of `/opt/stacks/wikijs/`.
**The database is already initialized** when Pocket Grimoire is first set up. Restoring a Netgrimoire dump overwrites it entirely, which is the intended behavior — Pocket Grimoire becomes a mirror of Netgrimoire's wiki state.
**Git content repo** is located at `/srv/pocket-grimoire/repos/wiki/` and is populated via the sync script (`pocketgrimoire-sync.sh`). A database restore alone is sufficient if the Git repo is already in place.
**Recommended restore workflow for Pocket Grimoire:**
```bash
# 1. Copy dump from VAULT SSD (already available on Pocket Grimoire)
ls /srv/vaultpg/backups/wiki/
# 2. Start db container only
cd /srv/pocket-grimoire/stacks/wikijs && docker compose up -d db
# 3. Restore
zcat /srv/vaultpg/backups/wiki/wikijs-db-LATEST.sql.gz | \
docker exec -i pocketgrimoire_db psql -U wikijs -d wikijs
# 4. Start full stack
docker compose up -d
```
Because the VAULT SSD is always connected to Pocket Grimoire, no file transfer is needed — the dumps are already there.
---
## Monitoring & Alerts
Add the following to your existing ntfy/monitoring setup to alert on backup failures. Wrap the backup script call in an error trap:
```bash
# Add to wikijs-backup.sh after set -euo pipefail:
NTFY_URL="https://ntfy.YOUR_DOMAIN/wikijs-backup"
on_error() {
curl -fsS -X POST "$NTFY_URL" \
-H "Title: Wiki.js backup FAILED ($(hostname -s))" \
-H "Priority: high" \
-H "Tags: rotating_light" \
-d "Backup failed at $(date -Is). Check /var/log/wikijs-backup.log"
}
trap on_error ERR
```
### Check backup age manually
```bash
# Find most recent dump
ls -lt /vault/backups/wiki/wikijs-db-*.sql.gz | head -3
# Check Kopia last snapshot time
kopia snapshot list /vault/backups/wiki | tail -5
```
---
## Quick Reference
```bash
# Run backup manually
sudo /usr/local/sbin/wikijs-backup.sh
# Watch backup log
tail -f /var/log/wikijs-backup.log
# Check timer status
systemctl status wikijs-backup.timer
# List local dumps
ls -lh /vault/backups/wiki/
# List Kopia snapshots
kopia snapshot list /vault/backups/wiki
# Restore dump (generic)
zcat /vault/backups/wiki/wikijs-db-YYYYMMDD_HHMMSS.sql.gz | \
docker exec -i CONTAINER_NAME psql -U wikijs -d wikijs
# Test dump is readable
zcat /vault/backups/wiki/wikijs-db-YYYYMMDD_HHMMSS.sql.gz | head -50
```
---
## Revision History
| Version | Date | Notes |
|---|---|---|
| 1.0 | 2026-02-22 | Initial release — pg_dump + Kopia + Pocket Grimoire restore procedures |