diff --git a/Netgrimoire/Pocket/Deployment_Guide.md b/Netgrimoire/Pocket/Deployment_Guide.md index 4ccbe06..a595973 100644 --- a/Netgrimoire/Pocket/Deployment_Guide.md +++ b/Netgrimoire/Pocket/Deployment_Guide.md @@ -2,7 +2,7 @@ title: Pocket Grimoire description: published: true -date: 2026-02-21T04:25:58.904Z +date: 2026-02-21T06:10:17.060Z tags: editor: markdown dateCreated: 2026-02-20T04:41:35.122Z @@ -44,14 +44,29 @@ Pocket Grimoire is a portable companion to the Netgrimoire homelab, providing of ### Storage (3 SSDs, 2 Active at a Time) - **SSD #1 – VAULT** (1-2TB, encrypted, always connected) - - Wiki.js data, PostgreSQL, git repos, photos, documents, backups -- **SSD #2 – MEDIA-PERSONAL** (2TB+, encrypted) - - Your curated H.264/AAC movies and TV shows - - Connected for personal trips -- **SSD #3 – MEDIA-FAMILY** (2TB+, unencrypted) + - Git repository mirrors (from Forgejo) + - Wiki.js content backups + - Kopia repository (backup data) + - Photos and documents + - System backups and configs + - SSH keys + - VeraCrypt containers (optional) + - **Does NOT contain media or Stash data** + +- **SSD #2 – GREEN** (2TB+, encrypted, for personal trips) + - Personal media library (H.264/AAC movies and TV) + - Stash-Pocket data (database, previews, blobs) + - Personal content organized under `/Green/` structure + - Connected during personal/solo trips + - Syncs from `/export/vault/Green/` on Netgrimoire + +- **SSD #3 – MEDIA-FAMILY** (2TB+, unencrypted, for family trips) - Family-friendly movies and TV shows (H.264/AAC) - - Connected for family visits - - Shareable/portable to other devices without Pocket Grimoire + - Simple `/library/movies/` and `/library/tv/` structure + - Connected during family visits/trips + - Unencrypted for easy sharing with relatives + - Can be used on other devices without Pocket Grimoire + - **USB drive – ISO/Rebuild** (64GB+, labeled, write-protected) - **USB drive – Data Transfer** (128GB+, labeled) @@ -237,7 +252,7 @@ Headroom: 78W ## Directory Structure ``` -/srv/pocket-grimoire/ # Main application root (on Vault SSD) +/srv/pocket-grimoire/ # Main application root (on VAULT SSD) ├── stacks/ # Docker Compose files │ ├── wikijs/ │ │ ├── docker-compose.yml @@ -245,6 +260,9 @@ Headroom: 78W │ ├── jellyfin/ │ │ ├── docker-compose.yml │ │ └── .env +│ ├── stash/ +│ │ ├── docker-compose.yml +│ │ └── .env │ └── filebrowser/ # Optional │ └── docker-compose.yml ├── data/ # Persistent container data @@ -258,23 +276,38 @@ Headroom: 78W ├── forgejo_wiki_ro # Read-only wiki deploy key └── zfs_pull_ro # ZFS replication key -/srv/vaultpg/ # Vault SSD ZFS mount -├── veracrypt-containers/ # VeraCrypt container files (optional) -│ └── vault.vc # Encrypted container -└── (other mirrors from Netgrimoire) +/srv/vaultpg/ # VAULT SSD (always connected) +├── kopia/ # Kopia backup repository +├── backups/ # System backups +│ ├── wiki/ # Wiki.js backups +│ ├── photos/ # Photo backups +│ └── documents/ # Document backups +├── repos/ # Git repository mirrors +└── veracrypt-containers/ # VeraCrypt containers (optional) + └── vault.vc + +/srv/greenpg/ # GREEN SSD (personal, rotated) +├── media/library/ # Personal media files +│ ├── movies/ +│ └── tv/ +└── stash/ # Stash-Pocket data + ├── config/ # Stash database + ├── generated/ # Previews + └── blobs/ # Scene markers + +/srv/mediapg/ # MEDIA-FAMILY SSD (family, rotated) +└── library/ # Family media files + ├── movies/ + └── tv/ /mnt/veracrypt/ # VeraCrypt mount points (optional) ├── vault1/ # Mounted container 1 └── vault2/ # Mounted container 2 (if needed) -/srv/mediapg/ # Media SSD ZFS mount (rotated) -└── library/ # H.264 encoded media - ├── movies/ - └── tv/ - /usr/local/sbin/ # System scripts ├── pocketgrimoire-sync.sh # Main sync script ├── pocketgrimoire-zfs-pull.sh # ZFS replication script +├── unlock-pocket-grimoire.sh # Headless unlock script └── mount-veracrypt-vault.sh # VeraCrypt mount script (optional) /etc/ # Config files @@ -512,11 +545,28 @@ This approach allows you to: - Verify data integrity before moving drives - Test encryption/unlock on powerful hardware first +#### Drive Configuration Overview + +**Drive #1: VAULT** (1-2TB, encrypted, always connected) +- Purpose: Backups and system data ONLY +- Contains: Git repos, Wiki backups, Kopia repository, photos, documents +- Does NOT contain media or Stash data + +**Drive #2: GREEN** (2TB+, encrypted, rotated for personal trips) +- Purpose: Personal media and Stash-Pocket data +- Contains: Personal media library, Stash database/previews/blobs +- Syncs from `/export/vault/Green/` on Netgrimoire + +**Drive #3: MEDIA-FAMILY** (2TB+, unencrypted, rotated for family trips) +- Purpose: Family-friendly shareable content +- Contains: Simple library structure with movies and TV +- Unencrypted for easy sharing with relatives + #### On Netgrimoire: Create and Populate Drives **Connect drives to Netgrimoire:** - VAULT SSD (1-2TB) via USB 3.0 or SATA -- MEDIA-PERSONAL SSD (2TB+) via USB 3.0 or SATA +- GREEN SSD (2TB+) via USB 3.0 or SATA - MEDIA-FAMILY SSD (2TB+) via USB 3.0 or SATA (optional, can be created later) **Identify drives:** @@ -526,7 +576,7 @@ lsblk # Note device names: /dev/sdX, /dev/sdY, /dev/sdZ ``` -**Create VAULT pool (encrypted):** +**Create VAULT pool (encrypted - backups only):** ```bash # On Netgrimoire sudo zpool create -o ashift=12 \ @@ -542,21 +592,21 @@ sudo zpool create -o ashift=12 \ # Enter STRONG passphrase when prompted # Write down this passphrase - you'll need it on Pocket Grimoire -# Create datasets matching Pocket structure -sudo zfs create -o recordsize=16K pocket-vault/wiki-pg -sudo zfs create pocket-vault/repos -sudo zfs create pocket-vault/pocket-grimoire -sudo zfs create pocket-vault/Green -sudo zfs create pocket-vault/Green/Pocket -sudo zfs create pocket-vault/Green/Pocket/stash -sudo zfs create pocket-vault/Green/Pocket/media -sudo zfs create pocket-vault/veracrypt-containers +# Create datasets for backups and system data +sudo zfs create -o recordsize=16K pocket-vault/wiki-pg # PostgreSQL backups +sudo zfs create pocket-vault/repos # Git repository mirrors +sudo zfs create pocket-vault/kopia # Kopia backup repository +sudo zfs create pocket-vault/backups # General backups +sudo zfs create pocket-vault/backups/wiki # Wiki.js backups +sudo zfs create pocket-vault/backups/photos # Photo backups +sudo zfs create pocket-vault/backups/documents # Document backups +sudo zfs create pocket-vault/veracrypt-containers # VeraCrypt (optional) # Set ownership sudo chown -R 1000:1000 /mnt/pocket-vault ``` -**Create MEDIA-PERSONAL pool (encrypted):** +**Create GREEN pool (encrypted - personal media + Stash):** ```bash # On Netgrimoire sudo zpool create -o ashift=12 \ @@ -566,78 +616,120 @@ sudo zpool create -o ashift=12 \ -O compression=lz4 \ -O atime=off \ -O recordsize=1M \ - -m /mnt/pocket-media-personal \ - pocket-media-personal /dev/sdY + -m /mnt/pocket-green \ + pocket-green /dev/sdY # Enter STRONG passphrase (can be different from VAULT) # Write down this passphrase -# Create datasets -sudo zfs create pocket-media-personal/library -sudo zfs create pocket-media-personal/library/movies -sudo zfs create pocket-media-personal/library/tv +# Create datasets matching Netgrimoire structure +sudo zfs create pocket-green/media # Media library root +sudo zfs create pocket-green/media/library # Library directory +sudo zfs create pocket-green/media/library/movies # Movies +sudo zfs create pocket-green/media/library/tv # TV shows +sudo zfs create pocket-green/stash # Stash-Pocket data +sudo zfs create pocket-green/stash/config # Stash database +sudo zfs create pocket-green/stash/generated # Previews +sudo zfs create pocket-green/stash/blobs # Scene markers +sudo zfs create pocket-green/stash/cache # Cache (don't sync) # Set ownership -sudo chown -R 1000:1000 /mnt/pocket-media-personal +sudo chown -R 1000:1000 /mnt/pocket-green ``` -**Create MEDIA-FAMILY pool (unencrypted, optional now or later):** +**Create MEDIA-FAMILY pool (unencrypted - family content):** ```bash # On Netgrimoire sudo zpool create -o ashift=12 \ -O compression=lz4 \ -O atime=off \ -O recordsize=1M \ - -m /mnt/pocket-media-family \ - pocket-media-family /dev/sdZ + -m /mnt/pocket-media \ + pocket-media /dev/sdZ # No encryption - family can use this drive on any system -# Create datasets -sudo zfs create pocket-media-family/library -sudo zfs create pocket-media-family/library/movies -sudo zfs create pocket-media-family/library/tv +# Create simple library structure +sudo zfs create pocket-media/library +sudo zfs create pocket-media/library/movies +sudo zfs create pocket-media/library/tv # Set ownership -sudo chown -R 1000:1000 /mnt/pocket-media-family +sudo chown -R 1000:1000 /mnt/pocket-media ``` **Perform initial sync to VAULT:** ```bash # On Netgrimoire -# Sync data from Netgrimoire to Pocket VAULT drive +# Sync backups and system data to VAULT drive -# Sync Pocket directory structure -sudo rsync -avP --exclude='cache' \ - /export/vault/Green/Pocket/ \ - /mnt/pocket-vault/Green/Pocket/ +# Sync Wiki backups +sudo rsync -avP \ + /export/vault/wiki-backups/ \ + /mnt/pocket-vault/backups/wiki/ -# Or use ZFS send/receive for atomic snapshot -sudo zfs snapshot vault/Green/Pocket@initial -sudo zfs send vault/Green/Pocket@initial | \ - sudo zfs receive pocket-vault/Green/Pocket +# Sync Git repositories +sudo rsync -avP \ + /export/vault/repos/ \ + /mnt/pocket-vault/repos/ + +# Sync Kopia repository (if exists) +sudo rsync -avP \ + /export/vault/kopia/ \ + /mnt/pocket-vault/kopia/ + +# Sync photos and documents +sudo rsync -avP \ + /export/vault/photos/ \ + /mnt/pocket-vault/backups/photos/ + +sudo rsync -avP \ + /export/vault/documents/ \ + /mnt/pocket-vault/backups/documents/ # Verify data -ls -lh /mnt/pocket-vault/Green/Pocket/ -du -sh /mnt/pocket-vault/Green/Pocket/ +ls -lh /mnt/pocket-vault/ +du -sh /mnt/pocket-vault/ ``` -**Populate MEDIA-PERSONAL (optional - curate content):** +**Perform initial sync to GREEN:** ```bash # On Netgrimoire -# Copy curated personal media to Pocket media drive +# Sync personal media and Stash data from /export/vault/Green/ -# Example: Copy favorite movies -sudo cp /export/vault/media/library/movies/favorites/*.mp4 \ - /mnt/pocket-media-personal/library/movies/ +# Sync entire Green directory structure +sudo rsync -avP --exclude='cache' \ + /export/vault/Green/ \ + /mnt/pocket-green/ + +# Or use ZFS send/receive for atomic snapshot (if Green is a ZFS dataset) +sudo zfs snapshot vault/Green@initial +sudo zfs send vault/Green@initial | \ + sudo zfs receive pocket-green/Green + +# Verify data +ls -lh /mnt/pocket-green/ +ls -lh /mnt/pocket-green/media/library/ +ls -lh /mnt/pocket-green/stash/ +du -sh /mnt/pocket-green/ +``` + +**Populate MEDIA-FAMILY (optional - curate family content):** +```bash +# On Netgrimoire +# Copy family-friendly media to MEDIA-FAMILY drive + +# Example: Copy family movies +sudo cp /export/vault/media/family-movies/*.mp4 \ + /mnt/pocket-media/library/movies/ # Or use rsync for large transfers sudo rsync -avP \ - /export/vault/media/library/tv/FavoriteShow/ \ - /mnt/pocket-media-personal/library/tv/FavoriteShow/ + /export/vault/media/family-shows/ \ + /mnt/pocket-media/library/tv/ # Verify -du -sh /mnt/pocket-media-personal/library/ +du -sh /mnt/pocket-media/library/ ``` **Export pools before disconnecting:** @@ -646,8 +738,8 @@ du -sh /mnt/pocket-media-personal/library/ # CRITICAL: Export pools before physically disconnecting drives sudo zpool export pocket-vault -sudo zpool export pocket-media-personal -sudo zpool export pocket-media-family # if created +sudo zpool export pocket-green +sudo zpool export pocket-media # if created # Verify exported zpool list @@ -659,8 +751,10 @@ zpool list ### 5. Configure ZFS Pools on Pocket Grimoire **Now connect drives to Pocket Grimoire:** -- VAULT → Anker USB-A port #2 -- MEDIA-PERSONAL or MEDIA-FAMILY → Raspberry Pi USB 3.0 port +- VAULT → Anker USB-A port #2 (always connected) +- GREEN (for personal trips) → Raspberry Pi USB 3.0 port + OR +- MEDIA-FAMILY (for family trips) → Raspberry Pi USB 3.0 port **Import and rename pools:** ```bash @@ -670,25 +764,29 @@ ssh user@pocket-grimoire.local # Import VAULT pool with new name sudo zpool import pocket-vault vaultpg -# Import MEDIA pool with new name -sudo zpool import pocket-media-personal mediapg -# Or: -# sudo zpool import pocket-media-family mediapg +# Import GREEN pool with new name (for personal trips) +sudo zpool import pocket-green greenpg + +# OR import MEDIA-FAMILY pool (for family trips) +# sudo zpool import pocket-media mediapg # Verify pools imported zpool list -# Should show: vaultpg, mediapg +# Should show: vaultpg, greenpg (or mediapg for family) ``` **Set mount points for Pocket Grimoire:** ```bash # Set proper mount points sudo zfs set mountpoint=/srv/vaultpg vaultpg -sudo zfs set mountpoint=/srv/mediapg mediapg +sudo zfs set mountpoint=/srv/greenpg greenpg +# Or for family drive: +# sudo zfs set mountpoint=/srv/mediapg mediapg # Create mount points sudo mkdir -p /srv/vaultpg -sudo mkdir -p /srv/mediapg +sudo mkdir -p /srv/greenpg +sudo mkdir -p /srv/mediapg # Create both, use as needed # Unmount and remount with new paths sudo zfs unmount -a @@ -698,11 +796,14 @@ sudo zfs mount -a df -h | grep srv # Should show: # vaultpg mounted on /srv/vaultpg -# mediapg mounted on /srv/mediapg +# greenpg mounted on /srv/greenpg (or mediapg on /srv/mediapg) # Verify data -ls /srv/vaultpg/Green/Pocket/ -ls /srv/mediapg/library/ +ls /srv/vaultpg/ +ls /srv/greenpg/media/library/ +ls /srv/greenpg/stash/ +# Or for family: +# ls /srv/mediapg/library/ ``` **Configure for headless unlock:** @@ -711,7 +812,8 @@ ls /srv/mediapg/library/ # This prevents boot hanging waiting for passphrase sudo zfs set canmount=noauto vaultpg -sudo zfs set canmount=noauto mediapg +sudo zfs set canmount=noauto greenpg +sudo zfs set canmount=noauto mediapg # For when you swap to family drive # Pools will need manual unlock via SSH after boot ``` @@ -748,9 +850,9 @@ echo " Pocket Grimoire ZFS Unlock (Headless)" echo "==========================================" echo -# Check if pools are already unlocked +# Check if VAULT pool is already unlocked if zfs list vaultpg &>/dev/null && mount | grep -q /srv/vaultpg; then - echo "✓ vaultpg already unlocked and mounted" + echo "✓ vaultpg (VAULT) already unlocked and mounted" else # Import pool if needed if ! zpool list vaultpg &>/dev/null; then @@ -758,8 +860,8 @@ else sudo zpool import vaultpg fi - # Unlock Vault pool - echo "Unlocking vaultpg (VAULT SSD)..." + # Unlock VAULT pool + echo "Unlocking vaultpg (VAULT - backups and system data)..." sudo zfs load-key vaultpg # Mount all vaultpg datasets @@ -776,14 +878,37 @@ fi echo -# Check if media pool is present and unlock +# Check for GREEN pool (personal media + Stash) +if zpool list greenpg &>/dev/null; then + if zfs list greenpg &>/dev/null && mount | grep -q /srv/greenpg; then + echo "✓ greenpg (GREEN - personal media + Stash) already unlocked" + else + echo "Unlocking greenpg (GREEN - personal media + Stash)..." + sudo zfs load-key greenpg + sudo zfs mount greenpg + sudo zfs mount -a + + if mount | grep -q /srv/greenpg; then + echo "✓ greenpg unlocked and mounted at /srv/greenpg" + else + echo "✗ Failed to mount greenpg" + exit 1 + fi + fi +else + echo "ℹ greenpg pool not found (GREEN drive not connected)" +fi + +echo + +# Check for MEDIA-FAMILY pool (family content) if zpool list mediapg &>/dev/null; then if zfs list mediapg &>/dev/null && mount | grep -q /srv/mediapg; then - echo "✓ mediapg already unlocked and mounted" + echo "✓ mediapg (MEDIA-FAMILY) already unlocked" else - echo "Unlocking mediapg (MEDIA-PERSONAL or MEDIA-FAMILY)..." + echo "Unlocking mediapg (MEDIA-FAMILY - family content)..." - # Check if encrypted (MEDIA-PERSONAL) or not (MEDIA-FAMILY) + # Check if encrypted (shouldn't be, but check anyway) if zfs get encryption mediapg | grep -q "encryption.*on"; then sudo zfs load-key mediapg fi @@ -799,7 +924,7 @@ if zpool list mediapg &>/dev/null; then fi fi else - echo "ℹ mediapg pool not found (media drive may not be connected)" + echo "ℹ mediapg pool not found (MEDIA-FAMILY drive not connected)" fi echo @@ -849,6 +974,17 @@ echo "==========================================" echo " Pocket Grimoire Ready!" echo "==========================================" echo +echo "Drives mounted:" +if mount | grep -q /srv/vaultpg; then + echo " ✓ VAULT (vaultpg) at /srv/vaultpg" +fi +if mount | grep -q /srv/greenpg; then + echo " ✓ GREEN (greenpg) at /srv/greenpg - Personal media + Stash" +fi +if mount | grep -q /srv/mediapg; then + echo " ✓ MEDIA-FAMILY (mediapg) at /srv/mediapg - Family content" +fi +echo echo "Services available at:" echo " Wiki.js: http://pocket-grimoire.local:3000" echo " Jellyfin: http://pocket-grimoire.local:8096" @@ -945,17 +1081,21 @@ ssh user@pocket-grimoire.local # Import pools if needed sudo zpool import vaultpg -sudo zpool import mediapg +sudo zpool import greenpg # For GREEN (personal) +# Or: +# sudo zpool import mediapg # For MEDIA-FAMILY # Load encryption keys -sudo zfs load-key vaultpg -sudo zfs load-key mediapg # Only if MEDIA-PERSONAL (encrypted) +sudo zfs load-key vaultpg # VAULT (always encrypted) +sudo zfs load-key greenpg # GREEN (encrypted) +# mediapg is unencrypted (MEDIA-FAMILY) - no key needed # Mount all datasets sudo zfs mount -a # Verify mounted df -h | grep srv +# Should show vaultpg and either greenpg or mediapg # Start Docker sudo systemctl start docker @@ -1717,8 +1857,8 @@ sudo umount /mnt/pocket-media ### 8. Document Passphrases - [ ] ZFS encryption passphrases (written down, secured) - VAULT (vaultpg): [write passphrase on paper] - - MEDIA-PERSONAL (mediapg): [write passphrase on paper] - - MEDIA-FAMILY: N/A (unencrypted) + - GREEN (greenpg): [write passphrase on paper] + - MEDIA-FAMILY (mediapg): N/A (unencrypted) - [ ] VeraCrypt container passwords (if using, written down, secured) - [ ] WiFi credentials for travel router (portapotty network) - [ ] Jellyfin admin password @@ -1742,8 +1882,9 @@ ssh user@pocket-grimoire.local /usr/local/sbin/unlock-pocket-grimoire.sh # 5. Enter passphrases when prompted -# - VAULT passphrase -# - MEDIA-PERSONAL passphrase (if encrypted) +# - VAULT passphrase (always) +# - GREEN passphrase (if GREEN drive connected for personal trip) +# - MEDIA-FAMILY has no passphrase (unencrypted) # - VeraCrypt password (if applicable) # 6. Wait for Docker containers to start (~30 seconds) @@ -1765,20 +1906,26 @@ docker ps ### 10. Verify Data Synced from Netgrimoire ```bash -# Check Vault data present -ls /srv/vaultpg/Green/Pocket/stash/ -ls /srv/vaultpg/Green/Pocket/media/ -du -sh /srv/vaultpg/Green/Pocket/ +# Check VAULT data present (backups only) +ls /srv/vaultpg/kopia/ +ls /srv/vaultpg/backups/ +ls /srv/vaultpg/repos/ +du -sh /srv/vaultpg/ -# Check media drive populated (if applicable) +# Check GREEN data (personal media + Stash) +ls /srv/greenpg/media/library/ +ls /srv/greenpg/stash/ +du -sh /srv/greenpg/ + +# Or check MEDIA-FAMILY data (if that drive is connected) ls /srv/mediapg/library/ du -sh /srv/mediapg/ -# Verify Stash database and previews -ls -lh /srv/vaultpg/Green/Pocket/stash/config/ +# Verify Stash database and previews (only on GREEN) +ls -lh /srv/greenpg/stash/config/ # Should show: stash-go.sqlite -ls /srv/vaultpg/Green/Pocket/stash/generated/ | wc -l +ls /srv/greenpg/stash/generated/ | wc -l # Should show: hundreds of preview files ``` @@ -2178,12 +2325,23 @@ veracrypt --text --list # Unmount and export ZFS pools sudo zfs unmount -a + +# Export VAULT (always present) sudo zpool export vaultpg -sudo zpool export mediapg + +# Export GREEN (if connected for personal trip) +if zpool list greenpg &>/dev/null; then + sudo zpool export greenpg +fi + +# Export MEDIA-FAMILY (if connected for family trip) +if zpool list mediapg &>/dev/null; then + sudo zpool export mediapg +fi # Verify pools exported zpool list -# Should NOT show vaultpg or mediapg +# Should NOT show vaultpg, greenpg, or mediapg # Shutdown Pi sudo shutdown -h now