diff --git a/immich_backup.md b/immich_backup.md index b7f6a83..75793d0 100644 --- a/immich_backup.md +++ b/immich_backup.md @@ -2,229 +2,488 @@ title: Immich Backup and Restore description: Immich backup with Kopia published: true -date: 2026-02-14T03:14:32.594Z +date: 2026-02-14T03:16:12.150Z tags: editor: markdown dateCreated: 2026-02-14T03:14:32.594Z --- +# Immich Backup and Recovery Guide + +## Overview + +This document provides comprehensive backup and recovery procedures for Immich photo management server. Following the same proven strategy as our Mailcow backups, we use a **two-tier approach** combining local component-level backups with Kopia for offsite storage in vaults. + +## Quick Reference + +### Common Backup Commands + +```bash +# Run a manual backup (all components) +/opt/scripts/backup-immich.sh + +# List Kopia snapshots +kopia snapshot list --tags immich + +# View backup logs +tail -f /var/log/immich-backup.log +``` + +### Common Restore Commands + +```bash +# Restore from local backup (interactive) +/opt/immich-backups/immich-restore.sh /opt/immich-backups/immich-YYYYMMDD_HHMMSS/ + +# Restore from Kopia to new server +kopia snapshot list --tags tier1-backup +kopia restore /opt/immich-backups/ + +# Check container status after restore +docker compose ps +docker compose logs -f +``` + +## Critical Components to Backup + +### 1. Docker Compose Configuration +- **Location**: `/opt/immich/docker-compose.yml` +- **Purpose**: Defines all containers, networks, and volumes +- **Importance**: Critical for recreating the exact container configuration + +### 2. Environment Configuration +- **Primary Config**: `/opt/immich/.env` +- **Purpose**: Database credentials, upload locations, API keys +- **Importance**: Required for proper service initialization + +### 3. PostgreSQL Database +- **Purpose**: Contains all metadata, user accounts, albums, sharing settings, face recognition data +- **Docker Volume**: `immich_postgres` +- **Backup Method**: `pg_dumpall` (hot backup, no downtime) + +### 4. Photo/Video Library +- **Purpose**: All original photos and videos uploaded by users +- **Docker Volume**: `immich_upload` or `immich_library` +- **Size**: Typically the largest component +- **Critical**: This is your actual data - photos cannot be recreated + +### 5. Additional Important Data +- **Thumbnails**: Can be regenerated but saves processing time +- **Encoded Video**: Transcoded versions, can be regenerated +- **Profile Pictures**: User avatars +- **Machine Learning Models**: Can be re-downloaded + +## Backup Strategy + +### Two-Tier Backup Approach + +We use a **two-tier approach** combining local snapshots with Kopia for offsite storage: + +1. **Tier 1 (Local)**: Component-level backups (database dump + volume archives) +2. **Tier 2 (Offsite)**: Kopia snapshots the local backups and syncs to vaults + +#### Why This Approach? + +- **Best of both worlds**: Local backups ensure quick component-level restore, Kopia provides deduplication and offsite protection +- **Component-level restore**: Can restore individual components (just database, just photos, etc.) +- **Disaster recovery**: Full system restore from Kopia backups on new server +- **Efficient storage**: Kopia's deduplication reduces storage needs for offsite copies +- **Proven strategy**: Same approach used successfully for Mailcow backups + +#### Backup Frequency +- **Daily**: Local Tier 1 backup runs at 2 AM +- **Daily**: Kopia Tier 2 snapshot runs immediately after Tier 1 +- **Retention (Local)**: 7 days of local backups +- **Retention (Kopia/Offsite)**: 30 daily, 12 weekly, 12 monthly + +### What Gets Backed Up + +Each backup creates: +- **database.sql.gz**: Complete PostgreSQL dump +- **immich_*.tar.gz**: Compressed archives of each Docker volume +- **docker-compose.yml**: Container configuration +- **.env**: Environment variables and secrets +- **manifest.txt**: Backup inventory and sizes +- **checksums.sha256**: Integrity verification + +## Setting Up Immich Backups + +### Prerequisites + +Connect to Kopia Repository: + +```bash +sudo kopia repository connect server \ + --url=https://192.168.5.10:51516 \ + --override-username=admin \ + --server-cert-fingerprint=YOUR_FINGERPRINT_HERE +``` + +### Step 1: Configure Backup Location + +```bash +sudo mkdir -p /opt/immich-backups +sudo chown -R root:root /opt/immich-backups +sudo chmod 755 /opt/immich-backups +``` + +### Step 2: Install Backup Scripts + +```bash +sudo mkdir -p /opt/scripts + +sudo cp immich-backup.sh /opt/scripts/backup-immich.sh +sudo cp immich-restore.sh /opt/immich-backups/immich-restore.sh + +sudo chmod +x /opt/scripts/backup-immich.sh +sudo chmod +x /opt/immich-backups/immich-restore.sh +``` + +### Step 3: Configure Backup Script + +Edit `/opt/scripts/backup-immich.sh` and verify: + +```bash +BACKUP_DIR="/opt/immich-backups" +RETENTION_DAYS=7 +IMMICH_DIR="/opt/immich" +POSTGRES_CONTAINER="immich_postgres" +PROJECT_NAME="immich" +ENABLE_KOPIA=true +``` + +Find your PostgreSQL container: +```bash +docker ps | grep postgres +``` + +### Step 4: Test Manual Backup + +```bash +sudo /opt/scripts/backup-immich.sh +``` + +Verify: +```bash +ls -lh /opt/immich-backups/ +kopia snapshot list --tags immich +``` + +### Step 5: Automated Backup with Cron + +```bash +sudo crontab -e +``` + +Add for daily backups at 2 AM: +``` +0 2 * * * /opt/scripts/backup-immich.sh 2>&1 | logger -t immich-backup +``` + +### Step 6: Configure Kopia Retention + +```bash +kopia policy set /opt/immich-backups \ + --keep-latest 30 \ + --keep-daily 30 \ + --keep-weekly 12 \ + --keep-monthly 12 + +kopia policy set /opt/immich \ + --keep-latest 7 \ + --keep-daily 7 +``` + +## Recovery Procedures + +### Method 1: Local Restore (Recommended) + +Full system restore: + +```bash +ls -lh /opt/immich-backups/ + +sudo /opt/immich-backups/immich-restore.sh /opt/immich-backups/immich-YYYYMMDD_HHMMSS/ +``` + +Database-only restore: + +```bash +cd /opt/immich +docker compose down +docker compose up -d postgres +sleep 10 + +BACKUP_PATH="/opt/immich-backups/immich-YYYYMMDD_HHMMSS" +gunzip < "${BACKUP_PATH}/database.sql.gz" | docker exec -i immich_postgres psql -U postgres + +docker compose down +docker compose up -d +``` + +Volume-only restore: + +```bash +cd /opt/immich +docker compose down + +BACKUP_PATH="/opt/immich-backups/immich-YYYYMMDD_HHMMSS" +VOLUME_NAME="immich_upload" + +docker run --rm \ + -v "${VOLUME_NAME}:/data" \ + -v "${BACKUP_PATH}:/backup" \ + alpine sh -c "cd /data && tar -xzf /backup/${VOLUME_NAME}.tar.gz" + +docker compose up -d +``` + +### Method 2: Complete Server Rebuild + +#### Step 1: Prepare New Server + +```bash +# Install Docker +curl -fsSL https://get.docker.com | sh +sudo systemctl enable docker +sudo apt install docker-compose-plugin -y + +# Install Kopia +curl -s https://kopia.io/signing-key | sudo gpg --dearmor -o /usr/share/keyrings/kopia-keyring.gpg +echo "deb [signed-by=/usr/share/keyrings/kopia-keyring.gpg] https://packages.kopia.io/apt/ stable main" | sudo tee /etc/apt/sources.list.d/kopia.list +sudo apt update && sudo apt install kopia -y + +# Create directories +sudo mkdir -p /opt/immich /opt/immich-backups /opt/scripts +``` + +#### Step 2: Connect to Kopia + +```bash +sudo kopia repository connect server \ + --url=https://YOUR_KOPIA_SERVER:51516 \ + --override-username=admin \ + --server-cert-fingerprint=YOUR_FINGERPRINT + +kopia repository status +``` + +#### Step 3: Restore Configuration + +```bash +kopia snapshot list --tags config +kopia restore /opt/immich/ + +ls -la /opt/immich/docker-compose.yml +ls -la /opt/immich/.env +``` + +#### Step 4: Restore Backups + +```bash +kopia snapshot list --tags tier1-backup +kopia restore /opt/immich-backups/ + +ls -la /opt/immich-backups/ +``` + +#### Step 5: Run Local Restore + +```bash +LATEST_BACKUP=$(ls -td /opt/immich-backups/immich-* | head -1) +sudo /opt/immich-backups/immich-restore.sh "$LATEST_BACKUP" +``` + +#### Step 6: Start and Verify + +```bash +cd /opt/immich +docker compose pull +docker compose up -d +docker compose logs -f +``` + +Verify: +- [ ] All containers running: `docker compose ps` +- [ ] Web interface accessible: `curl -I http://localhost:2283` +- [ ] Database accessible: `docker compose exec postgres psql -U postgres -c "\l"` +- [ ] Photos visible: `docker compose exec immich_server ls /usr/src/app/upload/` +- [ ] Login and check albums, timeline, sharing + +## Verification and Testing + +Monthly backup verification: + +```bash +# Verify local backups +ls -lth /opt/immich-backups/ | head -5 + +# Verify checksums +LATEST_BACKUP=$(ls -td /opt/immich-backups/immich-* | head -1) +cd "$LATEST_BACKUP" && sha256sum -c checksums.sha256 + +# Verify Kopia snapshots +kopia snapshot list --tags immich +``` + +### Backup Monitoring Script + +Create `/opt/scripts/check-immich-backup.sh`: + +```bash #!/bin/bash -# Immich Backup Script -# Two-tier approach matching mailcow backup strategy: -# Tier 1 (Local): Component-level backups (database + volumes) -# Tier 2 (Offsite): Kopia snapshots to vaults +LAST_BACKUP=$(ls -td /opt/immich-backups/immich-* | head -1) -set -euo pipefail - -# Configuration -BACKUP_DIR="/opt/immich-backups" -RETENTION_DAYS=7 # Keep local backups for 7 days (like mailcow) -BACKUP_DATE=$(date +%Y%m%d_%H%M%S) -BACKUP_PATH="${BACKUP_DIR}/immich-${BACKUP_DATE}" -LOG_FILE="/var/log/immich-backup.log" - -# Immich configuration (adjust these to your setup) -IMMICH_DIR="/opt/immich" # or wherever your docker-compose.yml is -COMPOSE_FILE="${IMMICH_DIR}/docker-compose.yml" -POSTGRES_CONTAINER="immich_postgres" # adjust if different -PROJECT_NAME="immich" # Docker Compose project name - -# Kopia configuration -ENABLE_KOPIA=true # Set to false to disable offsite backups -KOPIA_TAGS="immich,tier1-backup" - -# Colors for output -RED='\033[0;31m' -GREEN='\033[0;32m' -YELLOW='\033[1;33m' -NC='\033[0m' # No Color - -log() { - echo -e "${GREEN}[$(date +'%Y-%m-%d %H:%M:%S')]${NC} $1" | tee -a "$LOG_FILE" -} - -error() { - echo -e "${RED}[$(date +'%Y-%m-%d %H:%M:%S')] ERROR:${NC} $1" | tee -a "$LOG_FILE" -} - -warn() { - echo -e "${YELLOW}[$(date +'%Y-%m-%d %H:%M:%S')] WARNING:${NC} $1" | tee -a "$LOG_FILE" -} - -# Create backup directory -mkdir -p "${BACKUP_PATH}" - -log "========================================" -log "Starting Immich Tier 1 (Local) backup" -log "Backup path: ${BACKUP_PATH}" -log "========================================" - -# 1. Backup PostgreSQL database -log "Backing up PostgreSQL database..." -docker exec -t "${POSTGRES_CONTAINER}" pg_dumpall -c -U postgres | gzip > "${BACKUP_PATH}/database.sql.gz" - -if [ $? -eq 0 ]; then - DB_SIZE=$(du -h "${BACKUP_PATH}/database.sql.gz" | cut -f1) - log "Database backup completed successfully (${DB_SIZE})" -else - error "Database backup failed" +if [ -z "$LAST_BACKUP" ]; then + echo "ERROR: No backups found!" exit 1 fi -# 2. Backup Immich volumes/data -log "Backing up Immich data volumes..." +BACKUP_DATE=$(basename "$LAST_BACKUP" | sed 's/immich-//') +BACKUP_EPOCH=$(date -d "${BACKUP_DATE:0:8} ${BACKUP_DATE:9:2}:${BACKUP_DATE:11:2}:${BACKUP_DATE:13:2}" +%s 2>/dev/null) +NOW=$(date +%s) +AGE_HOURS=$(( ($NOW - $BACKUP_EPOCH) / 3600 )) -# Get actual Docker volumes for this project -DOCKER_VOLUMES=$(docker volume ls --filter "name=${PROJECT_NAME}" --format "{{.Name}}") - -if [ -z "${DOCKER_VOLUMES}" ]; then - warn "No Docker volumes found with name pattern '${PROJECT_NAME}'. Checking for bind mounts..." - - # Fallback to bind mounts if present - VOLUMES=( - "upload" - "library" - "profile" - "video" - "thumbs" - "encoded-video" - ) - - for volume in "${VOLUMES[@]}"; do - SOURCE_PATH="${IMMICH_DIR}/${volume}" - if [ -d "${SOURCE_PATH}" ]; then - log "Backing up bind mount: ${volume}..." - tar -czf "${BACKUP_PATH}/${volume}.tar.gz" -C "${IMMICH_DIR}" "${volume}" 2>/dev/null || warn "${volume} backup failed" - fi - done +if [ $AGE_HOURS -gt 26 ]; then + echo "WARNING: Last Immich backup is $AGE_HOURS hours old" + exit 1 else - # Backup Docker volumes - for vol in ${DOCKER_VOLUMES}; do - log "Backing up Docker volume: ${vol}" - docker run --rm \ - -v "${vol}:/data:ro" \ - -v "${BACKUP_PATH}:/backup" \ - alpine tar -czf "/backup/${vol}.tar.gz" -C /data . || warn "Failed to backup ${vol}" - - if [ -f "${BACKUP_PATH}/${vol}.tar.gz" ]; then - VOL_SIZE=$(du -h "${BACKUP_PATH}/${vol}.tar.gz" | cut -f1) - log " ✓ ${vol} backed up (${VOL_SIZE})" - fi - done + echo "OK: Last backup $AGE_HOURS hours ago" + BACKUP_SIZE=$(du -sh "$LAST_BACKUP" | cut -f1) + echo "Size: $BACKUP_SIZE" fi +``` -# 3. Backup configuration files -log "Backing up configuration files..." -if [ -f "${COMPOSE_FILE}" ]; then - cp "${COMPOSE_FILE}" "${BACKUP_PATH}/docker-compose.yml" - log " ✓ docker-compose.yml" -fi +Add to cron: +```bash +chmod +x /opt/scripts/check-immich-backup.sh +sudo crontab -e +0 8 * * * /opt/scripts/check-immich-backup.sh | logger -t immich-backup-check +``` -if [ -f "${IMMICH_DIR}/.env" ]; then - cp "${IMMICH_DIR}/.env" "${BACKUP_PATH}/.env" - log " ✓ .env" -fi +## Disaster Recovery Checklist -# Backup any custom config files -if [ -d "${IMMICH_DIR}/config" ]; then - tar -czf "${BACKUP_PATH}/config.tar.gz" -C "${IMMICH_DIR}" config 2>/dev/null && log " ✓ config directory" || true -fi +- [ ] Confirm scope of failure +- [ ] Access offsite Kopia repository +- [ ] Provision new server with Docker and Kopia +- [ ] Connect to Kopia repository +- [ ] Restore configuration from Kopia +- [ ] Restore backup directory from Kopia +- [ ] Run local restore script +- [ ] Start Immich containers +- [ ] Verify web interface and photo access +- [ ] Update DNS if needed +- [ ] Document issues and lessons learned -# 4. Create backup manifest (matching mailcow style) -log "Creating backup manifest..." -cat > "${BACKUP_PATH}/manifest.txt" << EOF -Immich Backup Manifest -Created: ${BACKUP_DATE} -Hostname: $(hostname) -Immich Directory: ${IMMICH_DIR} -Project Name: ${PROJECT_NAME} +## Important Notes -Backup Contents: -$(ls -lh "${BACKUP_PATH}" 2>/dev/null) +1. **Machine Learning Models**: Can be excluded from backup (re-download automatically) +2. **Photo Deduplication**: Remains intact after restore +3. **Facial Recognition**: Stored in database, restored automatically +4. **Permissions**: Handled automatically by Docker +5. **Upgrades**: Always restore with same or newer Immich version +6. **DNS**: Update if restoring to new server IP -Component Sizes: -Database: $(du -h "${BACKUP_PATH}/database.sql.gz" 2>/dev/null | cut -f1 || echo "N/A") -Total Backup Size: $(du -sh "${BACKUP_PATH}" 2>/dev/null | cut -f1) +## Troubleshooting -Docker Volumes Backed Up: -${DOCKER_VOLUMES:-None (using bind mounts)} -EOF +### "Database backup failed" +```bash +docker ps | grep postgres +docker compose logs postgres +docker exec immich_postgres pg_dumpall -U postgres | head +``` -# 5. Calculate checksums -log "Calculating checksums..." -cd "${BACKUP_PATH}" -find . -type f -name "*.tar.gz" -o -name "*.sql.gz" | xargs sha256sum > checksums.sha256 2>/dev/null || warn "Checksum creation failed" -cd - > /dev/null +### "Volume not found" +```bash +docker volume ls | grep immich +docker compose up -d --no-start +docker volume ls +``` -TIER1_SIZE=$(du -sh "${BACKUP_PATH}" | cut -f1) -log "Tier 1 (Local) backup completed! Size: ${TIER1_SIZE}" +### "Kopia snapshot fails" +```bash +kopia repository status +kopia repository connect server --url=... +kopia snapshot create /opt/immich-backups +``` -# 6. Tier 2: Create Kopia snapshot for offsite backup -if [ "$ENABLE_KOPIA" = true ]; then - log "========================================" - log "Starting Tier 2 (Offsite) Kopia backup" - log "========================================" - - # Check if kopia is available - if ! command -v kopia &> /dev/null; then - warn "Kopia not found. Skipping offsite backup." - warn "Install kopia or set ENABLE_KOPIA=false" - else - # Snapshot the backup directory - log "Creating Kopia snapshot of backup directory..." - kopia snapshot create "${BACKUP_DIR}" \ - --tags "${KOPIA_TAGS}" \ - --description "Immich backup ${BACKUP_DATE}" \ - 2>&1 | tee -a "$LOG_FILE" - - KOPIA_EXIT=${PIPESTATUS[0]} - - if [ $KOPIA_EXIT -eq 0 ]; then - log "Kopia snapshot completed successfully" - - # Also snapshot the Immich installation directory (configs) - log "Creating Kopia snapshot of installation directory..." - kopia snapshot create "${IMMICH_DIR}" \ - --tags "immich,config,docker-compose" \ - --description "Immich config ${BACKUP_DATE}" \ - 2>&1 | tee -a "$LOG_FILE" - else - warn "Kopia snapshot failed with exit code ${KOPIA_EXIT}" - warn "Local backup exists but offsite copy may be incomplete" - fi - fi -fi +### "Photos missing after restore" +```bash +docker compose exec immich_server ls -lah /usr/src/app/upload/ +docker volume inspect immich_upload +docker compose logs -f immich_server +``` -# 7. Cleanup old backups (retention policy) -log "========================================" -log "Applying retention policy (${RETENTION_DAYS} days)" -log "========================================" +## Advanced Topics -# Find and remove old local backups -REMOVED_COUNT=0 -while IFS= read -r old_backup; do - log "Removing old backup: $(basename "$old_backup")" - rm -rf "$old_backup" - ((REMOVED_COUNT++)) -done < <(find "${BACKUP_DIR}" -maxdepth 1 -type d -name "immich-*" -mtime +${RETENTION_DAYS} 2>/dev/null) +### Exclude Components to Save Space -if [ $REMOVED_COUNT -gt 0 ]; then - log "Removed ${REMOVED_COUNT} old backup(s)" -else - log "No old backups to remove" -fi +Edit backup script to exclude thumbnails/encoded videos: -# 8. Final summary -log "========================================" -log "Backup Summary" -log "========================================" -log "Tier 1 (Local): ${TIER1_SIZE} in ${BACKUP_PATH}" -log "Tier 2 (Offsite): Kopia snapshot created" -log "Retention: Keeping last ${RETENTION_DAYS} days locally" -log "Log file: ${LOG_FILE}" -log "========================================" -log "Backup completed successfully!" -log "========================================" +```bash +docker volume ls --filter "name=${PROJECT_NAME}" --format "{{.Name}}" | grep -v "thumbs" +``` -exit 0 \ No newline at end of file +### Incremental Backups + +``` +# Daily: Database + config only +0 2 * * 1-6 /opt/scripts/backup-immich-db-only.sh + +# Weekly: Full backup +0 2 * * 0 /opt/scripts/backup-immich.sh +``` + +### Backup Notifications + +Add to end of backup script: + +```bash +ADMIN_EMAIL="admin@example.com" +echo "Immich backup completed: ${TIER1_SIZE}" | \ + mail -s "✓ Immich Backup Success" "$ADMIN_EMAIL" + +curl -fsS --retry 3 https://hc-ping.com/your-uuid-here +``` + +## Backup Architecture Notes + +### Storage Efficiency Example + +For a 500GB photo library: + +**Without Kopia**: +- 7 days × 500GB = 3.5TB local storage + +**With Two-Tier**: +- Local: 3.5TB (7 days) +- Kopia vault: ~500GB + (30 × 10GB) = ~800GB +- **Savings**: 70-80% in vault storage + +### Component Restore Priority + +| Component | When to Restore | Can Regenerate? | Priority | +|-----------|----------------|-----------------|----------| +| Database | Always | No | Critical | +| Upload/Library | Always | No | Critical | +| Thumbnails | Optional | Yes | Medium | +| Encoded Videos | Optional | Yes | Low | +| Model Cache | Never | Yes | Low | + +## Additional Resources + +- [Immich Official Documentation](https://immich.app/docs) +- [Kopia Documentation](https://kopia.io/docs/) +- [Docker Volume Backup Best Practices](https://docs.docker.com/storage/volumes/#back-up-restore-or-migrate-data-volumes) + +## Revision History + +| Date | Version | Changes | +|------|---------|---------| +| 2026-02-13 | 1.0 | Initial documentation - two-tier backup strategy | + +--- + +**Last Updated**: February 13, 2026 +**Review Schedule**: Quarterly