From 47402071b2703b1b41a6230ca844332bfe71ba14 Mon Sep 17 00:00:00 2001 From: Administrator Date: Fri, 20 Feb 2026 04:48:24 +0000 Subject: [PATCH] docs: create Netgrimoire/Pocket/Stash_Integration --- Netgrimoire/Pocket/Stash_Integration.md | 741 ++++++++++++++++++++++++ 1 file changed, 741 insertions(+) create mode 100644 Netgrimoire/Pocket/Stash_Integration.md diff --git a/Netgrimoire/Pocket/Stash_Integration.md b/Netgrimoire/Pocket/Stash_Integration.md new file mode 100644 index 0000000..3c8567f --- /dev/null +++ b/Netgrimoire/Pocket/Stash_Integration.md @@ -0,0 +1,741 @@ +--- +title: Pocket Clips +description: Integrating Stash +published: true +date: 2026-02-20T04:48:11.191Z +tags: +editor: markdown +dateCreated: 2026-02-20T04:48:11.191Z +--- + +# Pocket Grimoire - Stash Integration Guide + +**Adding Stash media library manager to Pocket Grimoire using ZFS replication** + +--- + +## Overview + +This guide extends the Pocket Grimoire deployment to include Stash, a powerful media library manager. The architecture uses ZFS replication to mirror Stash data from Netgrimoire (home) to Pocket Grimoire (travel), allowing: + +- Full Stash browsing offline +- All previews, thumbnails, and metadata available +- Zero CPU load on Pi (no scanning/generation) +- Automatic synchronization via existing sync jobs +- Read-only operation on travel + +**Key Principle:** Netgrimoire does all heavy lifting (scanning, preview generation), Pocket Grimoire just serves pre-generated content. + +--- + +## Architecture + +``` +┌─────────────────────────────────────────┐ +│ NETGRIMOIRE (Home - Heavy Lifting) │ +├─────────────────────────────────────────┤ +│ Stash Container: │ +│ - Full scanning │ +│ - Preview generation │ +│ - Scene detection │ +│ - Database writes │ +│ - All CPU/GPU intensive work │ +│ │ +│ ZFS Dataset: vault/stash │ +│ - Stash database │ +│ - Generated previews │ +│ - Thumbnails & sprites │ +│ - Scene markers │ +└─────────────────────────────────────────┘ + ↓ + ZFS Send/Receive + (via syncoid every 6 hours) + ↓ +┌─────────────────────────────────────────┐ +│ POCKET GRIMOIRE (Travel - Read Only) │ +├─────────────────────────────────────────┤ +│ Stash Container: │ +│ - Read-only mode │ +│ - No scanning │ +│ - No generation │ +│ - Just browse existing data │ +│ │ +│ ZFS Dataset: vaultpg/stash │ +│ - Mirrored from Netgrimoire │ +│ - Complete database replica │ +│ - All previews pre-generated │ +└─────────────────────────────────────────┘ +``` + +--- + +## Storage Requirements + +### On Netgrimoire +``` +Database: 500MB - 2GB (depends on library size) +Generated previews: 5GB - 50GB (depends on preview settings) +Blobs/markers: 1GB - 10GB +─────────────────────────────────────────────────────── +Total: 6.5GB - 62GB +``` + +### On Pocket Grimoire +``` +Same as Netgrimoire (full replica via ZFS) +Stored on Vault SSD (vaultpg pool) +``` + +### Sync Bandwidth +- **Initial sync:** 10-30GB (1-2 hours on gigabit LAN) +- **Incremental sync:** 50MB - 500MB per 6 hours (2-10 minutes) +- **After adding 10 videos:** ~210MB (database + previews) + +--- + +## Resource Impact + +### Updated Resource Profile (With Stash) + +**Idle:** +``` +Wiki.js + PostgreSQL: ~250MB RAM +Jellyfin (idle): ~150MB RAM +Stash (read-only): ~200MB RAM # Much lighter than active Stash +ZFS ARC: ~512MB RAM +System: ~200MB RAM +───────────────────────────────────────── +Total: ~1.3GB / 8GB RAM ✓ +CPU: <10% +Temperature: Cool +``` + +**Browsing Stash:** +``` +Stash (active): ~300MB RAM +Other services: ~1.1GB RAM +───────────────────────────────────────── +Total: ~1.4GB / 8GB RAM ✓ +CPU: <15% +Temperature: Cool to Warm +``` + +**Media Playback + Stash Browsing:** +``` +Jellyfin (serving): ~200MB RAM +Stash (active): ~300MB RAM +Wiki.js + PostgreSQL: ~250MB RAM +ZFS ARC: ~512MB RAM +System: ~200MB RAM +───────────────────────────────────────── +Total: ~1.5GB / 8GB RAM ✓ +Still plenty of headroom +``` + +--- + +## Installation: Netgrimoire (Home) + +### 1. Create ZFS Datasets + +```bash +# On Netgrimoire +sudo zfs create -o recordsize=16K vault/stash +sudo zfs create -o recordsize=16K vault/stash/config # Database +sudo zfs create -o recordsize=1M vault/stash/generated # Previews +sudo zfs create -o recordsize=128K vault/stash/blobs # Scene markers +sudo zfs create vault/stash/cache # Temporary (won't replicate) + +# Set appropriate permissions +sudo chown -R 1000:1000 /vault/stash +``` + +**Why different recordsizes:** +- **16K** for config (SQLite database performs best with small records) +- **1M** for generated (large video preview files) +- **128K** for blobs (medium-sized screenshots and markers) + +### 2. Create Stash Docker Compose + +```bash +mkdir -p /srv/netgrimoire/stacks/stash +nano /srv/netgrimoire/stacks/stash/docker-compose.yml +``` + +```yaml +services: + stash: + image: stashapp/stash:latest + container_name: netgrimoire_stash + environment: + - STASH_STASH=/data/ + - STASH_GENERATED=/generated/ + - STASH_CACHE=/cache/ + - STASH_BLOBS=/blobs/ + - TZ=America/Chicago + volumes: + - /vault/stash/config:/root/.stash + - /vault/stash/generated:/generated + - /vault/stash/cache:/cache + - /vault/stash/blobs:/blobs + - /vault/media:/data:ro # Your media library (read-only) + ports: + - "9999:9999" + restart: unless-stopped +``` + +**Start Stash:** +```bash +cd /srv/netgrimoire/stacks/stash +docker compose up -d +``` + +### 3. Configure Stash + +1. **Access Stash:** + - Open browser: `http://netgrimoire.local:9999` + - Complete initial setup wizard + +2. **Add Library:** + - Settings → Library + - Add folder: `/data/library/movies` (or your media path) + - Save + +3. **Configure Previews:** + - Settings → Tasks → Generate + - Preview Generation: Enable + - Preview Settings: + - Video encoding: VP9 or H.264 + - Resolution: 720p (good quality, reasonable size) + - Segment duration: 10 seconds + - Generate image previews: Enable + - Generate sprites: Enable + +4. **Run Initial Scan:** + - Tasks → Scan + - Wait for completion (can take hours depending on library size) + +5. **Generate Previews:** + - Tasks → Generate → Generate Previews + - This is CPU intensive - let run at home + - Can take many hours depending on library size + +6. **Optional: Scene Detection:** + - Tasks → Generate → Auto Tag + - Scene detection, performer matching, etc. + - Very CPU intensive, run at home only + +### 4. Take ZFS Snapshots (Optional but Recommended) + +```bash +# After initial scan and preview generation +sudo zfs snapshot vault/stash/config@initial +sudo zfs snapshot vault/stash/generated@initial +sudo zfs snapshot vault/stash/blobs@initial +``` + +--- + +## Installation: Pocket Grimoire (Travel) + +### 1. Create ZFS Datasets + +```bash +# On Pocket Grimoire +sudo zfs create -o recordsize=16K vaultpg/stash +sudo zfs create -o recordsize=16K vaultpg/stash/config +sudo zfs create -o recordsize=1M vaultpg/stash/generated +sudo zfs create -o recordsize=128K vaultpg/stash/blobs +sudo zfs create vaultpg/stash/cache # Local cache only, not synced + +# Set permissions +sudo chown -R 1000:1000 /srv/vaultpg/stash +``` + +### 2. Create Stash Docker Compose (Read-Only) + +```bash +mkdir -p /srv/pocket-grimoire/stacks/stash +nano /srv/pocket-grimoire/stacks/stash/docker-compose.yml +``` + +```yaml +services: + stash: + image: stashapp/stash:latest + container_name: pocketgrimoire_stash + environment: + - STASH_STASH=/data/ + - STASH_GENERATED=/generated/ + - STASH_CACHE=/cache/ + - STASH_BLOBS=/blobs/ + - TZ=America/Chicago + volumes: + - /srv/vaultpg/stash/config:/root/.stash:ro # READ-ONLY + - /srv/vaultpg/stash/generated:/generated:ro # READ-ONLY + - /srv/vaultpg/stash/blobs:/blobs:ro # READ-ONLY + - /srv/pocket-grimoire/data/stash/cache:/cache # Local cache (writable) + - /srv/mediapg:/data:ro # Media (already present, read-only) + ports: + - "9999:9999" + restart: unless-stopped +``` + +**Note the `:ro` flags** - Filesystem is mounted read-only, preventing any writes. + +**Start Stash:** +```bash +cd /srv/pocket-grimoire/stacks/stash +docker compose up -d +``` + +### 3. Configure Stash for Read-Only Operation + +**After first start:** + +1. **Access Stash:** + - Open browser: `http://pocket-grimoire.local:9999` + - Should show library from Netgrimoire (after first sync) + +2. **Disable Background Tasks:** + - Settings → Tasks + - Disable all automatic tasks: + - ❌ Auto-scan + - ❌ Auto-tag + - ❌ Auto-preview generation + - ❌ Auto-cleanup + +3. **Verify Read-Only:** + - Try to edit a scene or performer + - Should fail with permission error + - This confirms read-only mode working + +--- + +## ZFS Replication Configuration + +### Update Sync Script + +Edit your existing sync script to include Stash datasets: + +```bash +sudo nano /usr/local/sbin/pocketgrimoire-zfs-pull.sh +``` + +Add these lines: + +```bash +#!/usr/bin/env bash +set -euo pipefail + +SRC_HOST="netgrimoire.local" +SSH_KEY="/srv/pocket-grimoire/keys/zfs_pull_ro" + +# Existing vault data syncs... +syncoid --no-sync-snap --recursive \ + --sshkey "${SSH_KEY}" \ + "root@${SRC_HOST}:vault/docs" \ + "vaultpg/mirror/docs" + +syncoid --no-sync-snap --recursive \ + --sshkey "${SSH_KEY}" \ + "root@${SRC_HOST}:vault/photos" \ + "vaultpg/mirror/photos" + +# NEW: Stash database and generated content +syncoid --no-sync-snap \ + --sshkey "${SSH_KEY}" \ + "root@${SRC_HOST}:vault/stash/config" \ + "vaultpg/stash/config" + +syncoid --no-sync-snap \ + --sshkey "${SSH_KEY}" \ + "root@${SRC_HOST}:vault/stash/generated" \ + "vaultpg/stash/generated" + +syncoid --no-sync-snap \ + --sshkey "${SSH_KEY}" \ + "root@${SRC_HOST}:vault/stash/blobs" \ + "vaultpg/stash/blobs" + +# Note: We skip cache dataset (temporary data, not needed on Pocket) +``` + +**Sync runs automatically every 6 hours** via existing systemd timer. + +### Manual Sync (For Testing) + +```bash +# Trigger sync immediately +sudo systemctl start pocketgrimoire-sync.service + +# Watch progress +tail -f /var/log/pocketgrimoire-sync.log + +# Verify Stash data synced +ls -lh /srv/vaultpg/stash/config +ls -lh /srv/vaultpg/stash/generated +``` + +--- + +## Media Path Consistency + +**Critical:** Media paths must be consistent between Netgrimoire and Pocket Grimoire. + +### Option 1: Matching Paths + +**If your media is in same location:** +``` +Netgrimoire: /vault/media/library/ +Pocket: /srv/mediapg/library/ +``` + +**Stash stores absolute paths** - ensure they match or use symlinks. + +### Option 2: Symlinks + +```bash +# On Pocket Grimoire +sudo ln -s /srv/mediapg /vault/media +``` + +This makes `/vault/media` point to `/srv/mediapg`, matching Netgrimoire's paths. + +### Option 3: Stash Path Mapping (If Stash supports it) + +Some versions of Stash support path mapping in config. Check Stash documentation. + +--- + +## Workflow + +### At Home (Before Trips) + +**On Netgrimoire:** +1. Add new media to library +2. Stash auto-scans (or trigger manually) +3. Previews generate automatically (or trigger manually) +4. Tag/organize content as desired +5. Everything happens in background + +**Pocket Grimoire (automatic):** +1. Every 6 hours: syncoid pulls changes from Netgrimoire +2. Stash database updated +3. New previews synced +4. Scene markers updated +5. Ready for next trip - no manual intervention + +### While Traveling + +**On Pocket Grimoire:** +1. Access Stash: `http://pocket-grimoire.local:9999` +2. Browse library, tags, performers, studios +3. View previews and scene markers +4. Click to play via Jellyfin or external player +5. **Cannot:** Scan, edit tags, generate previews (read-only) + +**If you need to edit:** +1. VPN back to Netgrimoire (via WireGuard) +2. Access Netgrimoire Stash: `http://netgrimoire.local:9999` +3. Make changes there +4. Changes sync on next 6-hour cycle + +### After Returning Home + +1. Sync happens automatically +2. Any changes made on Netgrimoire while away are pulled +3. Pocket Grimoire stays up-to-date +4. No manual intervention required + +--- + +## Accessing Stash + +### From Onn Streaming Boxes + +**Install StashApp for Android TV:** +- See "Onn 4K Streaming Box Setup Guide" for detailed instructions +- Download APK from: https://github.com/damontecres/StashAppAndroidTV/releases +- Sideload onto Onn boxes +- Configure server: `http://pocket-grimoire.local:9999` + +### From Laptop + +**Via Web Browser:** +- Open: `http://pocket-grimoire.local:9999` +- Full Stash web interface available +- Same as Netgrimoire, but read-only + +### From Phone/Tablet + +**Via Web Browser:** +- Connect to portapotty WiFi +- Open: `http://pocket-grimoire.local:9999` +- Mobile-responsive interface + +--- + +## Updated Service Access Summary + +**When connected to portapotty network:** + +``` +Wiki.js: http://pocket-grimoire.local:3000 +Jellyfin: http://pocket-grimoire.local:8096 +Stash: http://pocket-grimoire.local:9999 ← NEW +File Browser: http://pocket-grimoire.local:8080 +Dozzle: http://pocket-grimoire.local:8888 +SSH: ssh user@pocket-grimoire.local +NFS Media: nfs://pocket-grimoire.local/srv/mediapg +Router Admin: http://192.168.8.1 +``` + +--- + +## Troubleshooting + +### Stash Won't Start + +**Check Docker container status:** +```bash +docker ps | grep stash +docker logs pocketgrimoire_stash +``` + +**Common issues:** +- ZFS datasets not mounted: `zfs mount -a` +- Permission denied: `sudo chown -R 1000:1000 /srv/vaultpg/stash` +- Port conflict: Another service using port 9999 + +### Stash Shows "Database is locked" + +**This means read-only mode is working correctly.** + +If you need to make changes: +1. VPN to Netgrimoire +2. Edit on Netgrimoire Stash +3. Changes sync to Pocket on next cycle + +### Previews Not Showing + +**Verify previews synced:** +```bash +ls /srv/vaultpg/stash/generated/ +# Should show many .webp or .mp4 files +``` + +**If empty:** +- Previews not generated on Netgrimoire yet +- Sync hasn't completed (check sync logs) +- Check: `tail -n 200 /var/log/pocketgrimoire-sync.log` + +### Media Files Not Found + +**Check paths match:** +```bash +# On Netgrimoire, Stash sees files at: +# /vault/media/library/movies/ + +# On Pocket, files are at: +# /srv/mediapg/library/movies/ + +# Create symlink to match: +sudo ln -s /srv/mediapg /vault/media +``` + +**Or verify Docker volume mount:** +```bash +docker inspect pocketgrimoire_stash | grep -A 10 Mounts +# Should show /srv/mediapg mounted as /data +``` + +### Sync Takes Too Long + +**Check what's being synced:** +```bash +# Watch sync in progress +sudo zfs list -t snapshot | grep stash + +# Check dataset sizes +sudo zfs list | grep stash +``` + +**Optimization:** +- Reduce preview quality on Netgrimoire (smaller files) +- Sync less frequently (change timer from 6h to 12h) +- Only sync when on fast network (manual trigger) + +### Stash API Key Issues + +**If StashApp asks for API key:** + +1. **Find API key on Netgrimoire:** + ```bash + # On Netgrimoire + cat /vault/stash/config/config.yml | grep api_key + ``` + +2. **Or via Stash web UI:** + - Settings → Security → API Key + - Generate key if not present + - Copy key + +3. **Configure StashApp:** + - Settings → Server → API Key + - Paste key from Netgrimoire + +**Note:** API key is in config, which syncs to Pocket, so should be the same. + +--- + +## Optimization Tips + +### Reduce Storage Usage + +**On Netgrimoire (affects Pocket via sync):** + +1. **Lower preview quality:** + - Settings → Tasks → Generate + - Video resolution: 480p instead of 720p + - Lower bitrate + - Smaller file sizes = faster sync + +2. **Disable sprite generation:** + - Settings → Tasks → Generate + - Disable sprite generation (just use video previews) + - Saves significant space + +3. **Limit preview duration:** + - Settings → Tasks → Generate + - Preview duration: 20 seconds instead of 60 seconds + - Smaller files + +### Speed Up Sync + +**Initial sync optimization:** +```bash +# First sync: Compress during transfer +syncoid --compress=lz4 \ + --sshkey "${SSH_KEY}" \ + "root@${SRC_HOST}:vault/stash/generated" \ + "vaultpg/stash/generated" +``` + +**Bandwidth limiting (if needed):** +```bash +# Limit to 10MB/s +syncoid --bwlimit=10M \ + --sshkey "${SSH_KEY}" \ + "root@${SRC_HOST}:vault/stash/config" \ + "vaultpg/stash/config" +``` + +### Reduce Preview Generation Load + +**On Netgrimoire:** +- Generate previews during off-hours (nightly) +- Use Task Scheduler in Stash +- Limit concurrent preview generation +- Lower thread count for generation + +--- + +## Maintenance + +### Weekly + +**On Netgrimoire:** +- Check Stash is running: `docker ps | grep stash` +- Verify previews generating (if auto-enabled) +- Check disk usage: `zfs list | grep stash` + +**On Pocket Grimoire:** +- Verify sync is working: `tail /var/log/pocketgrimoire-sync.log` +- Check Stash is accessible: `http://pocket-grimoire.local:9999` + +### Monthly + +**On Netgrimoire:** +- Run manual scan if auto-scan disabled +- Clean up orphaned files: Tasks → Clean +- Check database integrity: Tasks → Optimize Database +- ZFS scrub: `sudo zfs scrub vault/stash` + +**On Pocket Grimoire:** +- Verify dataset health: `sudo zpool status vaultpg` +- Check sync logs for errors +- Test Stash browsing and preview playback + +### Before Trips + +- [ ] Run scan on Netgrimoire (capture new content) +- [ ] Generate previews for new content +- [ ] Verify sync completed successfully +- [ ] Test Stash on Pocket Grimoire +- [ ] Verify StashApp on Onn boxes connects + +### After Trips + +- Check sync logs for any errors during trip +- No action needed - sync continues automatically + +--- + +## Alternative: Manual Database Export (Not Recommended) + +**If ZFS replication is not available:** + +You can manually export/import Stash database: + +```bash +# On Netgrimoire - Export +sqlite3 /vault/stash/config/stash-go.sqlite ".backup /tmp/stash-backup.db" +scp /tmp/stash-backup.db pocket-grimoire:/tmp/ + +# On Pocket Grimoire - Import +docker compose down +cp /tmp/stash-backup.db /srv/vaultpg/stash/config/stash-go.sqlite +docker compose up -d +``` + +**Why this is not recommended:** +- Manual process (error-prone) +- Doesn't sync previews/blobs automatically +- Must copy those separately (tens of GB) +- ZFS replication is much cleaner + +--- + +## Summary + +**What you get with Stash on Pocket Grimoire:** +- ✅ Full Stash browsing offline +- ✅ All previews and metadata available +- ✅ Zero CPU load on Pi (read-only) +- ✅ Automatic synchronization +- ✅ Professional media library management +- ✅ Scene markers and performer tagging +- ✅ Works with StashApp on Onn boxes + +**What you give up:** +- ❌ Cannot edit/tag while traveling (must VPN to Netgrimoire) +- ❌ Cannot generate previews on travel (shouldn't anyway) +- ❌ 10-30GB additional storage on Vault SSD +- ❌ Slightly more complex setup (but worth it) + +**Recommended for:** +- Large media libraries that need organization +- Users who want professional media management +- Those who already use or want to use Stash +- Anyone who values rich metadata and previews + +**Skip if:** +- Just want simple media playback (Jellyfin is enough) +- Very limited storage on Vault SSD +- Don't want complexity of ZFS replication +- Don't need tagging/scene detection features + +--- + +*This guide supplements the main Pocket Grimoire deployment guide. Ensure main guide is completed before adding Stash.*