diff --git a/nextcloud_backup.md b/nextcloud_backup.md index 464bcf2..45fc927 100644 --- a/nextcloud_backup.md +++ b/nextcloud_backup.md @@ -2,29 +2,36 @@ title: Nextcloud Backup description: Native + Kopia published: true -date: 2026-02-15T00:11:07.746Z +date: 2026-02-18T04:40:14.455Z tags: editor: markdown dateCreated: 2026-02-14T23:52:25.405Z --- +--- +title: Nextcloud AIO Backup and Recovery Guide +description: Comprehensive backup and recovery procedures for Nextcloud All-in-One using native BorgBackup and Kopia offsite storage +published: true +date: 2026-02-17 +tags: nextcloud, backup, borgbackup, kopia, docker, aio +editor: markdown +--- + # Nextcloud AIO Backup and Recovery Guide ## Overview -This document provides comprehensive backup and recovery procedures for Nextcloud All-in-One (AIO). Nextcloud AIO includes a **built-in BorgBackup solution** that handles database, files, and configuration. We enhance this with Kopia for offsite storage in vaults, following the same two-tier approach used for Mailcow and Immich. +This document provides comprehensive backup and recovery procedures for Nextcloud All-in-One (AIO). Nextcloud AIO includes a **built-in BorgBackup solution** that runs daily automated backups. We enhance this with Kopia for offsite storage in vaults, following the same two-tier approach used for Mailcow and Immich. ## Quick Reference -### Important: Borg Backup password: 0c038ada7a620e59802f43422b6fea409b46bab8821be6d3 - ### Common Backup Commands ```bash -# Run automated backup script +# Run Kopia snapshot script (snapshots AIO's backups) /opt/scripts/backup-nextcloud.sh -# Trigger AIO backup manually (via web interface) +# Manually trigger AIO backup via web interface # Navigate to: https://your-server:8080 → Backup section → Create Backup # List Kopia snapshots @@ -34,8 +41,7 @@ kopia snapshot list --tags nextcloud tail -f /var/log/nextcloud-backup.log # Check AIO backup status -docker exec -it nextcloud-aio-mastercontainer bash -ls -lh /mnt/docker-aio-config/data/borg/ +docker exec -it nextcloud-aio-mastercontainer ls -lh /mnt/docker-aio-config/data/borg/ ``` ### Common Restore Commands @@ -58,14 +64,15 @@ docker logs -f nextcloud-aio-mastercontainer ### 1. Nextcloud AIO BorgBackup Repository - **Location**: Docker volume `nextcloud_aio_mastercontainer` → `/mnt/docker-aio-config/data/borg/` - **Contains**: - - Complete database (PostgreSQL) + - Complete PostgreSQL database - All user files and data - Nextcloud configuration - Apps and their data - **Importance**: This is Nextcloud AIO's primary backup - contains everything needed for restore +- **Encryption**: Encrypted with BorgBackup password ### 2. Nextcloud Data Directory -- **Location**: `/srv/NextCloud-AIO` (per your NEXTCLOUD_DATADIR setting) +- **Location**: `/srv/NextCloud-AIO` (per NEXTCLOUD_DATADIR setting) - **Purpose**: User files, photos, documents - **Note**: Already included in AIO BorgBackup, but can be backed up separately with Kopia @@ -79,10 +86,10 @@ docker logs -f nextcloud-aio-mastercontainer - **Purpose**: Container definitions, network settings, environment variables - **Importance**: Needed to recreate the AIO setup -### 5. Additional Components -- **Apache Port**: 11000 (your custom port) -- **AIO Admin Interface**: Port 8080 -- **Networks**: `netgrimoire` external network +### 5. Backup Credentials +- **BorgBackup Password**: `0c038ada7a620e59802f43422b6fea409b46bab8821be6d3` +- **Storage**: Store securely in password manager +- **Critical**: Required for restoring from Borg backups ## Backup Strategy @@ -90,8 +97,8 @@ docker logs -f nextcloud-aio-mastercontainer We use a **two-tier approach** combining Nextcloud AIO's native BorgBackup with Kopia for offsite storage: -1. **Tier 1 (Nextcloud AIO Native)**: BorgBackup creates deduplicated, compressed backups of everything -2. **Tier 2 (Offsite)**: Kopia snapshots the Borg repository and configs to vaults +1. **Tier 1 (Nextcloud AIO Native)**: BorgBackup creates deduplicated, encrypted backups daily at 02:00 +2. **Tier 2 (Offsite)**: Kopia snapshots the Borg repository at 03:00 and syncs to vaults #### Why This Approach? @@ -101,10 +108,10 @@ We use a **two-tier approach** combining Nextcloud AIO's native BorgBackup with - **Deduplication at two levels**: Borg deduplicates within Nextcloud data, Kopia deduplicates across backups - **Proven strategy**: Same approach used for Mailcow and Immich -#### Backup Frequency -- **AIO BorgBackup**: Configurable in AIO interface (recommend daily at 4 AM) -- **Kopia Tier 2**: Daily at 2 AM (snapshots the Borg repository) -- **Retention (AIO Borg)**: `--keep-within=7d --keep-weekly=4 --keep-monthly=6` (customizable) +#### Backup Schedule +- **02:00** - AIO's daily BorgBackup runs automatically +- **03:00** - Kopia script snapshots the completed Borg repository +- **Retention (AIO Borg)**: `--keep-within=7d --keep-weekly=4 --keep-monthly=6` - **Retention (Kopia/Offsite)**: 30 daily, 12 weekly, 12 monthly ### Nextcloud AIO Native Backup (BorgBackup) @@ -114,13 +121,13 @@ Nextcloud AIO includes an integrated BorgBackup solution that: - Backs up all user files and data - Backs up Nextcloud configuration and apps - Provides deduplication and compression -- Can be triggered manually or on schedule +- Runs on daily schedule (configured in AIO interface) - Includes built-in restore interface **Key Features:** - **Deduplication**: Only stores changed blocks (very efficient) - **Compression**: Reduces backup size significantly -- **Encryption**: Backups are encrypted by default +- **Encryption**: Backups are encrypted with password - **Incremental**: Only backs up changes after first backup - **Web UI**: Manage backups through AIO's admin interface @@ -130,8 +137,9 @@ Nextcloud AIO includes an integrated BorgBackup solution that: ## Setting Up Nextcloud AIO Backups -### Prereq: -Make sure you are connected to the repository, +### Prerequisites + +#### Connect to Kopia Repository ```bash sudo kopia repository connect server \ @@ -140,9 +148,9 @@ sudo kopia repository connect server \ --server-cert-fingerprint=696a4999f594b5273a174fd7cab677d8dd1628f9b9d27e557daa87103ee064b2 ``` -**Note on Multiple Kopia Repositories**: The backup script handles multiple Kopia repositories by explicitly connecting to the Nextcloud-specific repository at the start of each backup. To use a different repository, edit the variables in `/opt/scripts/backup-nextcloud.sh`. +> **Note on Multiple Kopia Repositories**: The backup script handles multiple Kopia repositories by explicitly connecting to the Nextcloud-specific repository at the start of each backup. To use a different repository, edit the variables in `/opt/scripts/backup-nextcloud.sh`. -#### Step 1: Configure AIO Built-in Backup +### Step 1: Configure AIO's Daily Backup First, configure Nextcloud AIO's built-in backup system: @@ -153,37 +161,31 @@ First, configure Nextcloud AIO's built-in backup system: 2. **Navigate to Backup Section** in the AIO interface -3. **Configure Backup Settings**: - - **Backup time**: Set to 04:00 (4 AM) to avoid conflicts with Kopia script - - **Retention policy**: Default is good (`--keep-within=7d --keep-weekly=4 --keep-monthly=6`) - - Can be customized with `BORG_RETENTION_POLICY` environment variable in docker-compose +3. **Configure Daily Backup**: + - Set **Backup time**: `02:00` (2 AM) + - Enable **Daily backup** + - Optionally enable **Automatic updates after backup** + - Optionally enable **Success notification** 4. **Verify Backup Location**: - ```bash - # Check that Borg repository exists - ls -lh /var/lib/docker/volumes/nextcloud_aio_mastercontainer/_data/data/borg/ + The backup location should be set to (default): + ``` + /mnt/docker-aio-config/data/borg ``` -5. **Trigger First Backup** via AIO interface to verify it works +5. **Trigger First Backup** manually via AIO interface to verify it works: + - Click "Create Backup" button + - Wait for completion (may take a while on first run) + - Verify success in the interface -**Optional: Customize Borg Retention** +6. **Verify Backup Exists**: + ```bash + docker exec nextcloud-aio-mastercontainer ls -lh /mnt/docker-aio-config/data/borg/ + ``` -Edit your docker-compose.yml to add: -```yaml -environment: - BORG_RETENTION_POLICY: --keep-within=7d --keep-weekly=4 --keep-monthly=6 -``` +### Step 2: Configure Local Backup Directory -Then restart: -```bash -cd /opt/nextcloud -docker compose down -docker compose up -d -``` - -#### Step 2: Configure Backup Location - -Create the backup directory for metadata and Kopia: +Create the backup directory for Kopia metadata: ```bash sudo mkdir -p /opt/nextcloud-backups @@ -191,7 +193,7 @@ sudo chown -R root:root /opt/nextcloud-backups sudo chmod 755 /opt/nextcloud-backups ``` -#### Step 3: Install Backup Script +### Step 3: Install Kopia Backup Script ```bash # Create scripts directory if not exists @@ -202,7 +204,7 @@ sudo cp backup-nextcloud.sh /opt/scripts/ sudo chmod +x /opt/scripts/backup-nextcloud.sh ``` -#### Step 4: Configure Backup Script +### Step 4: Configure Backup Script Edit `/opt/scripts/backup-nextcloud.sh` and verify these settings: @@ -216,9 +218,9 @@ KOPIA_USERNAME="admin" KOPIA_FINGERPRINT="696a4999..." ``` -#### Step 5: Test Manual Backup +### Step 5: Test Manual Backup -Run your first backup manually: +Run your first Kopia snapshot manually: ```bash sudo /opt/scripts/backup-nextcloud.sh @@ -227,17 +229,22 @@ sudo /opt/scripts/backup-nextcloud.sh Expected output: ``` ======================================== -Starting Nextcloud AIO backup process +Starting Nextcloud Kopia snapshot process +Note: AIO daily backup runs at 02:00, this script snapshots it ======================================== [timestamp] Connecting to Kopia repository... [timestamp] Kopia repository connected successfully -[timestamp] Triggering Nextcloud AIO BorgBackup... -[timestamp] Found 3 BorgBackup archives in AIO -[timestamp] Latest BorgBackup is recent (within 24 hours) -[timestamp] Exporting Nextcloud configuration... +[timestamp] Checking Nextcloud AIO status... +[timestamp] Mastercontainer is running +[timestamp] Verifying AIO Borg backups... +[timestamp] Found Borg backup repository at: /mnt/docker-aio-config/data/borg [timestamp] Backing up AIO mastercontainer volume... -[timestamp] Creating Kopia snapshot of backup directory... +[timestamp] Mastercontainer volume backup completed (5.2G) +[timestamp] Creating Kopia snapshots for offsite storage ... +✓ AIO Borg backups verified +✓ Mastercontainer volume backed up (5.2G) +✓ Kopia snapshots created for offsite storage ======================================== ``` @@ -250,15 +257,15 @@ cat /opt/nextcloud-backups/nextcloud-*/manifest.txt # Check Kopia snapshots kopia snapshot list --tags nextcloud -# Check AIO Borg backups (via web interface or command line) -docker exec -it nextcloud-aio-mastercontainer ls -lh /mnt/docker-aio-config/data/borg/ +# Check AIO Borg backups +docker exec nextcloud-aio-mastercontainer ls -lh /mnt/docker-aio-config/data/borg/ ``` -#### Step 6: Automated Backup with Cron +### Step 6: Automated Backup with Cron **Important Timing**: Schedule this AFTER AIO's built-in backup completes. -- AIO backup: 4:00 AM (configured in AIO interface) -- Kopia script: 5:00 AM (snapshots the completed Borg backup) +- AIO backup: **02:00** (configured in AIO interface) +- Kopia script: **03:00** (snapshots the completed Borg backup) Add to root's crontab: @@ -266,24 +273,24 @@ Add to root's crontab: sudo crontab -e ``` -Add this line for daily backups at 5 AM: +Add this line for daily backups at 3 AM: ``` -0 5 * * * /opt/scripts/backup-nextcloud.sh 2>&1 | logger -t nextcloud-backup +0 3 * * * /opt/scripts/backup-nextcloud.sh 2>&1 | logger -t nextcloud-backup ``` Alternative schedules: -**Twice daily (5 AM and 5 PM):** +**Twice daily (3 AM and 3 PM):** ``` -0 5,17 * * * /opt/scripts/backup-nextcloud.sh 2>&1 | logger -t nextcloud-backup +0 3,15 * * * /opt/scripts/backup-nextcloud.sh 2>&1 | logger -t nextcloud-backup ``` -**After AIO backup completes (4 AM AIO + 1 hour buffer):** +**After AIO backup completes (2 AM AIO + 1 hour buffer):** ``` -0 5 * * * /opt/scripts/backup-nextcloud.sh 2>&1 | logger -t nextcloud-backup +0 3 * * * /opt/scripts/backup-nextcloud.sh 2>&1 | logger -t nextcloud-backup ``` -#### Step 7: Configure Kopia Retention Policy +### Step 7: Configure Kopia Retention Policy Set retention for Nextcloud snapshots: @@ -298,12 +305,156 @@ kopia policy set /opt/nextcloud \ --keep-latest 7 \ --keep-daily 7 -# If backing up datadir directly +# If backing up datadir directly (and it's small enough) kopia policy set /srv/NextCloud-AIO \ --keep-latest 7 \ --keep-daily 7 ``` +## Backup Validation + +### Regular Verification + +Perform these checks monthly to ensure backups are working: + +#### Test 1: Verify AIO Borg Backups Exist + +```bash +# List Borg archives inside container +docker exec nextcloud-aio-mastercontainer \ + borg list /mnt/docker-aio-config/data/borg/borgbackup + +# Expected output: List of backup archives with dates +``` + +#### Test 2: Check Borg Backup Integrity + +```bash +# Run Borg check (non-destructive) +docker exec nextcloud-aio-mastercontainer \ + borg check /mnt/docker-aio-config/data/borg/borgbackup + +# Expected output: Repository and archive checks passing +``` + +#### Test 3: Verify Local Kopia Backups + +```bash +# List recent local backups +ls -lth /opt/nextcloud-backups/ | head -5 + +# Check most recent backup +LATEST=$(ls -td /opt/nextcloud-backups/nextcloud-* | head -1) +cat "${LATEST}/manifest.txt" +``` + +#### Test 4: Verify Kopia Snapshots + +```bash +# List Kopia snapshots +kopia snapshot list --tags nextcloud + +# Verify snapshot content (without restoring) +kopia snapshot list --show-identical + +# Check Kopia repository status +kopia repository status +``` + +#### Test 5: Verify Backup Encryption + +```bash +# The BorgBackup password should be required for restore +# Test by attempting to list archives (should require password) + +docker exec -it nextcloud-aio-mastercontainer bash +export BORG_PASSPHRASE="0c038ada7a620e59802f43422b6fea409b46bab8821be6d3" +borg list /mnt/docker-aio-config/data/borg/borgbackup +exit +``` + +### Backup Monitoring Script + +Create `/opt/scripts/check-nextcloud-backup.sh`: + +```bash +#!/bin/bash + +LOG_PREFIX="[Nextcloud Backup Check]" + +# Check 1: AIO Borg backups +echo "$LOG_PREFIX Checking AIO Borg backups..." +BORG_COUNT=$(docker exec nextcloud-aio-mastercontainer \ + find /mnt/docker-aio-config/data/borg -type d -maxdepth 2 2>/dev/null | wc -l) + +if [ "$BORG_COUNT" -gt 0 ]; then + echo "$LOG_PREFIX ✓ AIO has $BORG_COUNT Borg backup items" +else + echo "$LOG_PREFIX ✗ WARNING: No Borg backups found!" + exit 1 +fi + +# Check 2: Local backups +echo "$LOG_PREFIX Checking local backups..." +LAST_BACKUP=$(ls -td /opt/nextcloud-backups/nextcloud-* 2>/dev/null | head -1) + +if [ -z "$LAST_BACKUP" ]; then + echo "$LOG_PREFIX ✗ WARNING: No local backups found!" + exit 1 +fi + +BACKUP_DATE=$(basename "$LAST_BACKUP" | sed 's/nextcloud-//') +BACKUP_EPOCH=$(date -d "${BACKUP_DATE:0:8} ${BACKUP_DATE:9:2}:${BACKUP_DATE:11:2}:${BACKUP_DATE:13:2}" +%s 2>/dev/null) +NOW=$(date +%s) +AGE_HOURS=$(( ($NOW - $BACKUP_EPOCH) / 3600 )) + +if [ $AGE_HOURS -gt 26 ]; then + echo "$LOG_PREFIX ✗ WARNING: Last backup is $AGE_HOURS hours old" + exit 1 +else + echo "$LOG_PREFIX ✓ Last backup: $AGE_HOURS hours ago" + BACKUP_SIZE=$(du -sh "$LAST_BACKUP" 2>/dev/null | cut -f1) + echo "$LOG_PREFIX Size: $BACKUP_SIZE" +fi + +# Check 3: Kopia snapshots +echo "$LOG_PREFIX Checking Kopia snapshots..." +KOPIA_COUNT=$(kopia snapshot list --tags nextcloud --json 2>/dev/null | jq '. | length' 2>/dev/null) + +if [ -n "$KOPIA_COUNT" ] && [ "$KOPIA_COUNT" -gt 0 ]; then + echo "$LOG_PREFIX ✓ Kopia has $KOPIA_COUNT snapshots" + KOPIA_LAST=$(kopia snapshot list --tags nextcloud --json 2>/dev/null | jq -r '.[0].startTime' 2>/dev/null) + echo "$LOG_PREFIX Latest: $KOPIA_LAST" +else + echo "$LOG_PREFIX ✗ WARNING: Cannot verify Kopia snapshots" + exit 1 +fi + +echo "$LOG_PREFIX =========================================" +echo "$LOG_PREFIX All backup checks passed!" +echo "$LOG_PREFIX =========================================" +``` + +Make executable and add to cron: +```bash +chmod +x /opt/scripts/check-nextcloud-backup.sh + +# Add to crontab (check daily at 8 AM) +sudo crontab -e +0 8 * * * /opt/scripts/check-nextcloud-backup.sh | logger -t nextcloud-backup-check +``` + +### Test Restore (Non-Production) + +Perform a test restore in a non-production environment quarterly: + +1. **Provision test server** with same OS +2. **Install Docker** and Kopia +3. **Restore from Kopia** following disaster recovery steps below +4. **Verify Nextcloud** works correctly +5. **Document any issues** encountered +6. **Destroy test environment** + ## Recovery Procedures ### Understanding Two Recovery Methods @@ -333,9 +484,14 @@ This is the **easiest and most reliable** method for Nextcloud AIO. 3. **Select Restore** -4. **Choose Backup Date** from available Borg backups +4. **Enter BorgBackup Password**: + ``` + 0c038ada7a620e59802f43422b6fea409b46bab8821be6d3 + ``` -5. **Confirm Restore** +5. **Choose Backup Date** from available Borg backups + +6. **Confirm Restore** - AIO will: - Stop Nextcloud containers - Restore database @@ -343,27 +499,34 @@ This is the **easiest and most reliable** method for Nextcloud AIO. - Restore configuration - Restart containers -6. **Verify** Nextcloud is working after restore +7. **Verify** Nextcloud is working after restore: + - Access web interface + - Check files are accessible + - Verify user logins work #### Via Command Line (Advanced) +If you need to restore from Borg backup directly: + ```bash -# Stop Nextcloud (if needed) -docker exec -it nextcloud-aio-mastercontainer \ - /usr/bin/docker-compose -f /path/to/compose.yml down +# Enter the mastercontainer +docker exec -it nextcloud-aio-mastercontainer bash -# List available Borg backups -docker exec -it nextcloud-aio-mastercontainer \ - borg list /mnt/docker-aio-config/data/borg/borgbackup +# Set Borg password +export BORG_PASSPHRASE="0c038ada7a620e59802f43422b6fea409b46bab8821be6d3" -# Restore from specific backup -docker exec -it nextcloud-aio-mastercontainer \ - /restore-script.sh +# List available backups +borg list /mnt/docker-aio-config/data/borg/borgbackup -# Note: Exact restore commands depend on AIO version -# Recommend using web interface for most reliable restore +# Extract specific backup (example - adjust archive name) +borg extract /mnt/docker-aio-config/data/borg/borgbackup::20260217-020001 + +# Exit container +exit ``` +> **Note**: Using AIO's web interface is much safer and handles all the container orchestration automatically. + ### Method 2: Complete Server Rebuild (Kopia Restore) Use this when recovering to a completely new server or when AIO is completely lost. @@ -372,33 +535,33 @@ Use this when recovering to a completely new server or when AIO is completely lo ```bash # Update system -apt update && apt upgrade -y +sudo apt update && sudo apt upgrade -y # Install Docker curl -fsSL https://get.docker.com | sh -systemctl enable docker -systemctl start docker +sudo systemctl enable docker +sudo systemctl start docker # Install Docker Compose -apt install docker-compose-plugin -y +sudo apt install docker-compose-plugin -y # Install Kopia curl -s https://kopia.io/signing-key | sudo gpg --dearmor -o /usr/share/keyrings/kopia-keyring.gpg echo "deb [signed-by=/usr/share/keyrings/kopia-keyring.gpg] https://packages.kopia.io/apt/ stable main" | sudo tee /etc/apt/sources.list.d/kopia.list -apt update -apt install kopia -y +sudo apt update +sudo apt install kopia -y # Create directory structure -mkdir -p /opt/nextcloud -mkdir -p /opt/nextcloud-backups -mkdir -p /srv/NextCloud-AIO +sudo mkdir -p /opt/nextcloud +sudo mkdir -p /opt/nextcloud-backups +sudo mkdir -p /srv/NextCloud-AIO ``` #### Step 2: Connect to Kopia Repository ```bash # Connect to your offsite vault -kopia repository connect server \ +sudo kopia repository connect server \ --url=https://192.168.5.10:51516 \ --override-username=admin \ --server-cert-fingerprint=696a4999f594b5273a174fd7cab677d8dd1628f9b9d27e557daa87103ee064b2 @@ -427,7 +590,7 @@ ls -la /opt/nextcloud/docker-compose.yml ```bash # Find the backup with mastercontainer -kopia snapshot list --tags tier1-backup +kopia snapshot list --tags mastercontainer # Restore the most recent backup kopia restore /opt/nextcloud-backups/ @@ -436,9 +599,10 @@ kopia restore /opt/nextcloud-backups/ LATEST_BACKUP=$(ls -td /opt/nextcloud-backups/nextcloud-* | head -1) echo "Restoring from: $LATEST_BACKUP" -# Extract mastercontainer volume backup +# Create the Docker volume docker volume create nextcloud_aio_mastercontainer +# Extract mastercontainer volume backup docker run --rm \ -v nextcloud_aio_mastercontainer:/target \ -v "${LATEST_BACKUP}":/backup:ro \ @@ -447,17 +611,7 @@ docker run --rm \ echo "Mastercontainer volume restored" ``` -#### Step 5: Restore Borg Repository (Optional but Recommended) - -```bash -# Restore the Borg repository directly from Kopia -kopia snapshot list --tags borg - -# Restore Borg backup to volume location -kopia restore /var/lib/docker/volumes/nextcloud_aio_mastercontainer/_data/data/borg/ -``` - -#### Step 6: Start Nextcloud AIO +#### Step 5: Start Nextcloud AIO ```bash cd /opt/nextcloud @@ -472,25 +626,29 @@ docker compose up -d docker compose logs -f ``` -#### Step 7: Access AIO and Restore +#### Step 6: Access AIO and Restore from Borg -1. **Wait for mastercontainer to initialize** (may take 1-2 minutes) +1. **Wait for mastercontainer to initialize** (1-2 minutes) 2. **Access AIO Interface**: ``` https://new-server-ip:8080 ``` -3. **If Borg backups were restored** (Step 5): - - Navigate to Backup section - - Select restore from available backup - - AIO will restore everything automatically +3. **Navigate to Backup Section** -4. **If starting fresh**: - - Follow AIO initial setup - - Restore data from datadir backup if available +4. **Select Restore** -#### Step 8: Post-Restore Verification +5. **Enter BorgBackup Password**: + ``` + 0c038ada7a620e59802f43422b6fea409b46bab8821be6d3 + ``` + +6. **Choose backup date** and restore + +7. **Wait for restore to complete** (may take significant time depending on data size) + +#### Step 7: Post-Restore Verification ```bash # Check all containers are running @@ -502,9 +660,11 @@ docker ps | grep nextcloud # - nextcloud-aio-nextcloud # - nextcloud-aio-database # - nextcloud-aio-redis -# (and others depending on your setup) +# - nextcloud-aio-imaginary +# - nextcloud-aio-fulltextsearch (if enabled) +# - nextcloud-aio-talk (if enabled) -# Check Nextcloud is accessible +# Check Nextcloud is accessible via Apache curl -I http://localhost:11000 # Access web interface @@ -515,50 +675,49 @@ curl -I http://localhost:11000 # - Files are accessible # - Apps are functioning # - Sharing works +# - Uploads work ``` ### Scenario 2: Restore Specific User's Files To restore a single user's files without affecting others: -**Via AIO Web Interface:** +#### Option A: Via AIO Web Interface (Safest) -1. Navigate to Files → Deleted Files (if available) -2. Use Nextcloud's built-in file versioning +1. Use AIO's restore to restore to a specific point in time +2. Users can access previous versions via Nextcloud's version history -**Via Borg Backup:** +#### Option B: Extract from Borg Backup ```bash -# Mount a Borg backup +# Enter mastercontainer docker exec -it nextcloud-aio-mastercontainer bash -# Inside container: -cd /mnt/docker-aio-config/data/borg/ +# Set Borg password +export BORG_PASSPHRASE="0c038ada7a620e59802f43422b6fea409b46bab8821be6d3" # List available backups -borg list borgbackup +borg list /mnt/docker-aio-config/data/borg/borgbackup # Mount a specific backup mkdir -p /tmp/borg-mount -borg mount borgbackup:: /tmp/borg-mount +borg mount /mnt/docker-aio-config/data/borg/borgbackup:: /tmp/borg-mount # Navigate to user's files -cd /tmp/borg-mount/nextcloud/data//files/ +cd /tmp/borg-mount/nextcloud_aio_nextcloud_data/ -# Copy needed files to restore location -cp -r /tmp/borg-mount/nextcloud/data//files/* \ - /actual/nextcloud/data//files/ +# Find the user's directory +ls -la + +# Copy specific files out (example) +cp -r /tmp/borg-mount/nextcloud_aio_nextcloud_data// /recovery/ # Unmount borg umount /tmp/borg-mount exit - -# Run Nextcloud file scan -docker exec -it nextcloud-aio-nextcloud bash -php occ files:scan ``` -**Via Kopia (if datadir was backed up):** +#### Option C: Using Kopia Mount ```bash # Mount Kopia snapshot @@ -577,168 +736,110 @@ rsync -av /mnt/kopia-nextcloud/${USER}/files/ \ # Unmount kopia unmount /mnt/kopia-nextcloud -# Rescan files -docker exec -u www-data -it nextcloud-aio-nextcloud \ +# Rescan files in Nextcloud +docker exec -u www-data nextcloud-aio-nextcloud \ php occ files:scan ${USER} ``` -### Scenario 3: Database Recovery Only - -If only the database is corrupted: - -**Via AIO Interface (Recommended):** -1. Use AIO's restore feature -2. Select database-only restore (if available) - -**Via Borg Backup:** - -```bash -docker exec -it nextcloud-aio-mastercontainer bash - -# Extract database from Borg -borg extract borgbackup:: nextcloud/database/ - -# Restore to PostgreSQL -# (Exact commands depend on AIO's database structure) -``` - -**Note**: Database-only restore is tricky with AIO. Recommend using full AIO restore. - -### Scenario 4: Rollback After Failed Update +### Scenario 3: Rollback After Failed Update If a Nextcloud update breaks your installation: -1. **Stop Nextcloud**: - ```bash - docker exec -it nextcloud-aio-mastercontainer \ - /path/to/stop-script.sh - ``` +1. **Access AIO Interface**: `https://your-server:8080` -2. **Use AIO Restore**: - - Access AIO interface (port 8080) - - Navigate to Backup section - - Restore from backup before the update +2. **Navigate to Backup Section** -3. **Verify** everything works +3. **Select Restore** -4. **Disable auto-updates** until update issue is resolved +4. **Enter password**: `0c038ada7a620e59802f43422b6fea409b46bab8821be6d3` -## Verification and Testing +5. **Choose backup** from before the update -### Regular Backup Verification +6. **Restore and verify** -Perform monthly verification: +7. **Disable auto-updates** in AIO until issue is resolved -```bash -# Test 1: Verify AIO Borg backups exist -docker exec -it nextcloud-aio-mastercontainer \ - borg list /mnt/docker-aio-config/data/borg/borgbackup +### Scenario 4: Database-Only Recovery -# Test 2: Verify local backups -ls -lth /opt/nextcloud-backups/ | head -5 +If only the database is corrupted but files are intact: -# Test 3: Verify Kopia snapshots -kopia snapshot list --tags nextcloud +**Via AIO Restore (Recommended):** +1. Use AIO's restore feature +2. It will restore the complete state including database -# Test 4: Check Borg backup integrity (in test environment) -docker exec -it nextcloud-aio-mastercontainer \ - borg check /mnt/docker-aio-config/data/borg/borgbackup -``` - -### Backup Monitoring Script - -Create `/opt/scripts/check-nextcloud-backup.sh`: - -```bash -#!/bin/bash - -# Check last backup age -LAST_BACKUP=$(ls -td /opt/nextcloud-backups/nextcloud-* 2>/dev/null | head -1) - -if [ -z "$LAST_BACKUP" ]; then - echo "WARNING: No Nextcloud backups found" - exit 1 -fi - -BACKUP_DATE=$(basename "$LAST_BACKUP" | sed 's/nextcloud-//') -BACKUP_EPOCH=$(date -d "${BACKUP_DATE:0:8} ${BACKUP_DATE:9:2}:${BACKUP_DATE:11:2}:${BACKUP_DATE:13:2}" +%s 2>/dev/null) -NOW=$(date +%s) -AGE_HOURS=$(( ($NOW - $BACKUP_EPOCH) / 3600 )) - -if [ $AGE_HOURS -gt 26 ]; then - echo "WARNING: Last Nextcloud backup is $AGE_HOURS hours old" - exit 1 -else - echo "OK: Last backup $AGE_HOURS hours ago" -fi - -# Check AIO Borg backups -BORG_PATH="/var/lib/docker/volumes/nextcloud_aio_mastercontainer/_data/data/borg" -if [ -d "$BORG_PATH" ]; then - BORG_COUNT=$(find "$BORG_PATH" -type d -name "borgbackup_*" | wc -l) - echo "AIO Borg backups available: $BORG_COUNT" -else - echo "WARNING: Cannot access AIO Borg backups" -fi - -# Check Kopia snapshots -KOPIA_LAST=$(kopia snapshot list --tags nextcloud --json 2>/dev/null | jq -r '.[0].startTime' 2>/dev/null) - -if [ -n "$KOPIA_LAST" ]; then - echo "Last Kopia snapshot: $KOPIA_LAST" -else - echo "WARNING: Cannot verify Kopia snapshots" -fi -``` - -Make executable and add to cron: -```bash -chmod +x /opt/scripts/check-nextcloud-backup.sh - -sudo crontab -e -0 8 * * * /opt/scripts/check-nextcloud-backup.sh | logger -t nextcloud-backup-check -``` +**Note**: AIO's BorgBackup stores everything together, so you cannot easily restore just the database. A full restore is safer and ensures consistency. ## Disaster Recovery Checklist When disaster strikes, follow this checklist: -- [ ] Confirm scope of failure (server down, storage failure, database corruption) +- [ ] Confirm scope of failure (server down, storage failure, data corruption) - [ ] Gather server information (hostname, IP, domain configuration) +- [ ] Retrieve BorgBackup password from secure storage - [ ] Access offsite Kopia repository - [ ] Provision new server (if needed) with Docker and Kopia - [ ] Connect to Kopia repository - [ ] Restore docker-compose configuration - [ ] Restore AIO mastercontainer volume -- [ ] Restore Borg repository (if available) - [ ] Start AIO mastercontainer -- [ ] Use AIO's restore interface to restore from Borg backup +- [ ] Use AIO's restore interface with BorgBackup password - [ ] Verify web interface accessible - [ ] Test file access and sharing - [ ] Verify user logins work - [ ] Check all apps are functioning - [ ] Update DNS if server IP changed - [ ] Document issues and lessons learned +- [ ] Update this guide based on experience ## Important Notes -1. **AIO Backup Schedule**: Configure AIO's built-in backup to run BEFORE the Kopia script (e.g., AIO at 4 AM, Kopia at 5 AM) +### 1. Backup Schedule Coordination -2. **Storage Requirements**: Borg backups are deduplicated but can still be large. Ensure adequate space: - - Borg repository: ~1.5x your data size initially - - Kopia adds minimal overhead due to deduplication +- **AIO backup**: 02:00 (configured in AIO interface) +- **Kopia script**: 03:00 (must run AFTER AIO backup completes) +- Ensure sufficient time gap for large backups -3. **Network Configuration**: Your setup uses external network `netgrimoire` and custom Apache port 11000. Ensure these are documented for restore. +### 2. BorgBackup Password Security -4. **SKIP_DOMAIN_VALIDATION**: Your setup has this enabled. Ensure domain is properly configured for restore. +- **Password**: `0c038ada7a620e59802f43422b6fea409b46bab8821be6d3` +- **Storage**: Keep in password manager (LastPass, 1Password, etc.) +- **Backup**: Print and store in safe location +- **Critical**: Without this password, Borg backups cannot be restored -5. **Testing**: Always test recovery procedures in a lab environment before trusting them in production. +### 3. Storage Requirements -6. **Documentation**: Keep this guide and server details in a separate location (printed copy, password manager, etc.). +BorgBackup is deduplicated but can still be large: +- Borg repository: ~1.5x your data size initially +- Grows with retention policy (7 days + 4 weeks + 6 months) +- Kopia adds minimal overhead due to deduplication +- Monitor disk space regularly -7. **Retention Policy**: Review both Borg and Kopia retention settings periodically to balance storage costs with recovery needs. +### 4. Network Configuration -8. **Updates**: When updating Nextcloud via AIO, ensure a backup is taken first (AIO should do this automatically). +Your setup uses: +- External network: `netgrimoire` +- Custom Apache port: `11000` +- AIO admin port: `8080` +- Ensure these are documented for restore + +### 5. Testing + +- Test recovery procedures quarterly in lab environment +- Verify backups monthly with validation scripts +- Keep printed copy of this guide offsite + +### 6. Updates + +- AIO can auto-update containers after successful backup +- This is configured in the daily backup settings +- Ensure backups run BEFORE updates + +### 7. Performance Considerations + +First backup will be slow (full copy): +- Subsequent backups are incremental (much faster) +- Borg deduplication improves over time +- Large files (videos) take longer ## Backup Architecture Notes @@ -748,8 +849,8 @@ When disaster strikes, follow this checklist: - ✅ Nextcloud-aware (knows how to backup everything correctly) - ✅ Built-in restore interface (easiest to use) - ✅ Deduplication and compression +- ✅ Encrypted with password - ✅ Incremental backups (efficient) -- ✅ Encryption included - ❌ Single location (inside Docker volume) - ❌ No offsite replication built-in @@ -766,23 +867,23 @@ When disaster strikes, follow this checklist: For a 200GB Nextcloud instance: -**Borg (Tier 1):** +**BorgBackup (Tier 1):** - First backup: ~200GB - Daily incremental: ~2-5GB (only changes) -- With retention policy: ~250GB total (7 days daily + 4 weekly + 6 monthly) +- With retention policy: ~250GB total -**Kopia (Tier 2) backing up Borg:** +**Kopia (Tier 2) backing up mastercontainer:** - First snapshot: ~250GB -- Daily changes: ~5-10GB (only changed Borg chunks) -- Much smaller than backing up raw data +- Daily changes: ~5-10GB (only changed data in Borg repo) +- Much smaller than backing up raw data repeatedly -**Combined savings**: Borg handles Nextcloud deduplication, Kopia handles backup deduplication +**Combined savings**: Borg handles Nextcloud-level deduplication, Kopia handles backup-to-backup deduplication ### Deduplication Layers -1. **Nextcloud level**: Versioning and deleted files +1. **Nextcloud level**: File versioning and deleted file retention 2. **Borg level**: Block-level deduplication within Nextcloud data -3. **Kopia level**: File-level deduplication of Borg repository +3. **Kopia level**: File-level deduplication of mastercontainer volume ## Troubleshooting @@ -816,13 +917,17 @@ sudo ufw allow 8080/tcp # Check disk space df -h -# Check Borg repository -docker exec -it nextcloud-aio-mastercontainer \ - borg check /mnt/docker-aio-config/data/borg/borgbackup +# Check Borg repository integrity +docker exec nextcloud-aio-mastercontainer bash +export BORG_PASSPHRASE="0c038ada7a620e59802f43422b6fea409b46bab8821be6d3" +borg check /mnt/docker-aio-config/data/borg/borgbackup +exit # Repair if needed (last resort) -docker exec -it nextcloud-aio-mastercontainer \ - borg check --repair /mnt/docker-aio-config/data/borg/borgbackup +docker exec nextcloud-aio-mastercontainer bash +export BORG_PASSPHRASE="0c038ada7a620e59802f43422b6fea409b46bab8821be6d3" +borg check --repair /mnt/docker-aio-config/data/borg/borgbackup +exit ``` ### "Restore hangs or fails" @@ -837,31 +942,26 @@ df -h /srv/NextCloud-AIO # Check Docker resources docker stats -# Increase Docker memory/CPU limits if needed +# Check if daily_backup_running marker is stuck +docker exec nextcloud-aio-mastercontainer \ + rm /mnt/docker-aio-config/data/daily_backup_running -# Try restoring to different location first (test) +# Restart and try again +docker restart nextcloud-aio-mastercontainer ``` -### "Files missing after restore" +### "Wrong Borg password" -**Symptoms**: Nextcloud restored but files don't appear +**Symptoms**: Cannot restore or list backups -**Solutions**: -```bash -# Rescan files -docker exec -u www-data -it nextcloud-aio-nextcloud \ - php occ files:scan --all - -# Check file permissions -ls -la /srv/NextCloud-AIO/ - -# Verify datadir mount -docker inspect nextcloud-aio-nextcloud | grep -A 10 Mounts -``` +**Solution**: +- Verify password is exactly: `0c038ada7a620e59802f43422b6fea409b46bab8821be6d3` +- No spaces, no quotes when entering in AIO interface +- If password is truly lost, backups cannot be recovered ### "Kopia snapshot fails" -**Symptoms**: Tier 2 backup fails but local backup succeeds +**Symptoms**: Tier 2 backup fails but AIO backup succeeds **Solutions**: ```bash @@ -872,23 +972,45 @@ kopia repository status kopia repository disconnect kopia repository connect server --url=... -# Check available space in Kopia repository +# Check Kopia repository space kopia repository status | grep Space -# Manually create snapshot to test +# Manually test snapshot kopia snapshot create /opt/nextcloud-backups ``` +### "Files missing after restore" + +**Symptoms**: Nextcloud restored but files don't appear + +**Solutions**: +```bash +# Rescan all files +docker exec -u www-data nextcloud-aio-nextcloud \ + php occ files:scan --all + +# Check file permissions +docker exec nextcloud-aio-nextcloud \ + ls -la /mnt/ncdata/ + +# Verify datadir mount +docker inspect nextcloud-aio-nextcloud | grep -A 10 Mounts + +# Check Nextcloud logs +docker exec nextcloud-aio-nextcloud \ + tail -100 /var/www/html/data/nextcloud.log +``` + ## Advanced Topics ### Customizing Borg Retention Policy -Edit docker-compose.yml: +Edit your docker-compose.yml: ```yaml environment: - # Keep 7 days, 4 weeks, 12 months - BORG_RETENTION_POLICY: --keep-within=7d --keep-weekly=4 --keep-monthly=12 + # Default: Keep 7 days, 4 weeks, 6 months + BORG_RETENTION_POLICY: --keep-within=7d --keep-weekly=4 --keep-monthly=6 # More aggressive (less space, shorter history) # BORG_RETENTION_POLICY: --keep-within=3d --keep-weekly=2 --keep-monthly=3 @@ -904,27 +1026,22 @@ docker compose down docker compose up -d ``` -### Excluding Large Directories from Backup - -If certain directories don't need backup (e.g., temporary files, caches): - -This must be configured in AIO's backup interface or Borg configuration. Consult AIO documentation for current version's options. - -### Backup to Multiple Destinations +### Backup to Multiple Kopia Repositories For critical data, backup to multiple vaults: ```bash -# In backup script, add second Kopia repository -# After first snapshot completes: +# In backup script, after first Kopia snapshot: # Connect to second repository kopia repository disconnect -kopia repository connect server --url=https://backup2.example.com:51517 ... +kopia repository connect server \ + --url=https://backup2.example.com:51517 \ + --override-username=admin \ + --server-cert-fingerprint=... # Create snapshots in second repository kopia snapshot create /opt/nextcloud-backups -kopia snapshot create /var/lib/docker/volumes/nextcloud_aio_mastercontainer/_data/data/borg/ ``` ### Backup Notifications @@ -932,13 +1049,13 @@ kopia snapshot create /var/lib/docker/volumes/nextcloud_aio_mastercontainer/_dat Add to the end of `/opt/scripts/backup-nextcloud.sh`: ```bash +# Email notification ADMIN_EMAIL="admin@example.com" - if [ $? -eq 0 ]; then echo "Nextcloud backup completed successfully" | \ mail -s "✓ Nextcloud Backup Success" "$ADMIN_EMAIL" else - echo "Nextcloud backup FAILED! Check logs at /var/log/nextcloud-backup.log" | \ + echo "Nextcloud backup FAILED! Check /var/log/nextcloud-backup.log" | \ mail -s "✗ Nextcloud Backup FAILED" "$ADMIN_EMAIL" fi @@ -951,6 +1068,39 @@ curl -X POST https://monitor.example.com/backup-status \ -d '{"service":"nextcloud","status":"success"}' ``` +### Excluding Directories from Backup + +If certain directories don't need backup (temporary files, caches): + +This must be configured through AIO's interface or environment variables. Check AIO documentation for current version's options for excluding paths. + +### Manual Borg Operations + +Advanced users can interact with Borg directly: + +```bash +# Enter mastercontainer +docker exec -it nextcloud-aio-mastercontainer bash + +# Set password +export BORG_PASSPHRASE="0c038ada7a620e59802f43422b6fea409b46bab8821be6d3" + +# List all archives +borg list /mnt/docker-aio-config/data/borg/borgbackup + +# Get info about specific archive +borg info /mnt/docker-aio-config/data/borg/borgbackup:: + +# Check repository +borg check /mnt/docker-aio-config/data/borg/borgbackup + +# Compact repository (reclaim space) +borg compact /mnt/docker-aio-config/data/borg/borgbackup + +# Exit +exit +``` + ## Additional Resources - [Nextcloud AIO Official Documentation](https://github.com/nextcloud/all-in-one) @@ -959,14 +1109,43 @@ curl -X POST https://monitor.example.com/backup-status \ - [Kopia Documentation](https://kopia.io/docs/) - [Docker Volume Backup Best Practices](https://docs.docker.com/storage/volumes/#back-up-restore-or-migrate-data-volumes) +## Security Considerations + +### BorgBackup Password + +- **Never commit** the password to git repositories +- **Store securely** in password manager +- **Print and store** physical copy in safe +- **Share with team** via secure channel only +- **Document location** of password storage + +### Kopia Repository Access + +- Kopia repository credentials stored on server +- Limit access to backup server +- Use firewall rules to restrict Kopia server access +- Consider separate Kopia repositories for different services + +### Network Security + +- AIO admin interface (8080) - restrict access via firewall +- Apache interface (11000) - behind reverse proxy +- Kopia server (51516) - internal network only + ## Revision History | Date | Version | Changes | |------|---------|---------| -| 2026-02-13 | 1.0 | Initial documentation - two-tier backup strategy using AIO's BorgBackup + Kopia | +| 2026-02-17 | 1.0 | Initial documentation - two-tier backup strategy using AIO's BorgBackup + Kopia | --- -**Last Updated**: February 13, 2026 +**Last Updated**: February 17, 2026 **Maintained By**: System Administrator -**Review Schedule**: Quarterly +**Review Schedule**: Quarterly +**Next Review**: May 17, 2026 + +**Critical Information**: +- BorgBackup Password: Store securely, required for all restores +- Backup Schedule: AIO 02:00, Kopia 03:00 +- Kopia Server: 192.168.5.10:51516