[Docs Draft] znas filesystem -- pending review (20260715-201238-docs-znas-13a5) #15

New issue

Closed

opened 2026-07-15 15:14:26 -05:00 by traveler · 1 comment

traveler commented 2026-07-15 15:14:26 -05:00

Owner

Copy link

AUTO-GENERATED DRAFT -- NOT YET PUBLISHED. This page was written automatically by the @document pilot (chat -> n8n -> Mistral via gremlin-proxy) from a mix of prior memory notes and one live SSH inspection pass. Read it carefully before manually committing it to the wiki -- do not trust it blindly, especially anything under "Discrepancies noted".

# ZNAS Filesystem Topology

## Overview
ZNAS is the primary storage system for NetGrimoire, running Ubuntu with OpenZFS. It serves as the source of truth for all storage, including datasets that replicate to the Pocket Grimoire portable system. All NFS exports are ZFS datasets mounted directly under `/export/`.

---

## ZFS Pools and Datasets

### Pools
ZNAS has two ZFS pools:

1. **`vault`** (primary pool)
   - ~129TB raw capacity (70% used)
   - Two RAIDZ1 VDEVs:
     - 3×24TB drives
     - 5×16TB drives
   - Compression: `zstd` (default), `lz4` (select datasets)
   - Deduplication: off

2. **`pocket-media`** (Pocket Grimoire SSD)
   - 4.55TB single-disk pool
   - Mounted at `/mnt/pm/mnt/pocket-media`
   - Compression: `lz4`
   - Deduplication: off

### Datasets (Live State)
#### `vault` Pool
| Dataset                     | Mountpoint                     | Purpose                          | Compression | Notes                          |
|-----------------------------|--------------------------------|----------------------------------|-------------|--------------------------------|
| `vault`                     | `/export`                      | NFS root                         | zstd        |                                |
| `vault/Common`              | `/export/Common`               | Shared documents, configs        | zstd        |                                |
| `vault/Data`                | `/export/Data`                 | Media root                       | lz4         |                                |
| `vault/Data/media_books`    | `/export/Data/media/books`     | Books library                    | zstd        | `nohide` in NFS exports        |
| `vault/Data/media_comics`   | `/export/Data/media/comics`    | Comics library                   | zstd        | `nohide` in NFS exports        |
| `vault/Docker`              | `/export/Docker`               | Swarm volumes                    | zstd        |                                |
| `vault/Green`               | `/export/Green`                | Green Grimoire media             | off         |                                |
| `vault/Green/Pocket`        | `/export/Green/Pocket`         | Pocket sync source               | off         |                                |
| `vault/Photos`              | `/export/Photos`               | Photo library                    | zstd        |                                |
| `vault/Kopia`               | `/srv/vault/kopia_repository`  | Backup repository                | zstd        | Local-only (not NFS-exported)  |
| `vault/NextCloud-AIO`       | `/srv/NextCloud-AIO`           | NextCloud data                   | lz4         | Local-only                     |
| `vault/backup`              | `/srv/vault/backup`            | Local backups                    | zstd        | Local-only                     |
| `vault/immich`              | `/srv/immich`                  | Immich photo library             | lz4         | Local-only                     |
| `vault/pocket`              | `none`                         | Pocket sync parent               | zstd        |                                |
| `vault/pocket/green`        | `/pocket/green`                | Green Grimoire sync target       | zstd        |                                |
| `vault/pocket/green/media`  | `/pocket/green/media`          | Media sync target                | zstd        |                                |
| `vault/pocket/media`        | `/pocket/media`                | Media sync target                | zstd        |                                |
| `vault/pocket/services`     | `/pocket/services`             | Service data sync target         | zstd        |                                |

#### `pocket-media` Pool
| Dataset                              | Mountpoint                                      | Purpose                          |
|--------------------------------------|-------------------------------------------------|----------------------------------|
| `pocket-media`                       | `/mnt/pm/mnt/pocket-media`                      | Root                              |
| `pocket-media/media`                 | `/mnt/pm/mnt/pocket-media/media`                | Media root                        |
| `pocket-media/media/movies`          | `/mnt/pm/mnt/pocket-media/media/movies`         | Movies library                    |
| `pocket-media/backups`               | `/mnt/pm/mnt/pocket-media/backups`              | Backup root                       |
| `pocket-media/backups/documents`     | `/mnt/pm/mnt/pocket-media/backups/documents`    | Documents backup                  |
| `pocket-media/backups/photos`        | `/mnt/pm/mnt/pocket-media/backups/photos`       | Photos backup                     |
| `pocket-media/backups/wiki`          | `/mnt/pm/mnt/pocket-media/backups/wiki`         | Wiki backup                       |
| `pocket-media/wiki-pg`               | `/mnt/pm/mnt/pocket-media/wiki-pg`              | Wiki database                     |

---

## NFS Exports

### Server Configuration
- **NFSv4 only**: All exports use NFSv4 with a pseudo-root at `/export`.
- **Exports file** (`/etc/exports`):
  ```plaintext
  /export *(rw,fsid=0,no_root_squash,no_subtree_check,crossmnt)
  /export/Common *(fsid=4,rw,no_root_squash,no_subtree_check,insecure)
  /export/Data *(fsid=5,rw,no_root_squash,no_subtree_check,insecure,crossmnt)
  /export/Data/media/books *(fsid=51,rw,no_root_squash,no_subtree_check,insecure,nohide)
  /export/Data/media/comics *(fsid=52,rw,no_root_squash,no_subtree_check,insecure,nohide)
  /export/Docker *(fsid=29,rw,no_root_squash,sync,no_subtree_check,insecure)
  /export/Green *(fsid=30,rw,no_root_squash,no_subtree_check,insecure,crossmnt)
  /export/photos *(fsid=31,rw,no_root_squash,no_subtree_check,insecure)
  /export/Pocket *(fsid=32,rw,no_root_squash,no_subtree_check,insecure)
  /export/Grimoire *(fsid=33,rw,no_root_squash,no_subtree_check,insecure)
  /export/green/media/pocket *(fsid=35,rw,no_root_squash,no_subtree_check,insecure)

• Key options:
  • fsid=0: NFSv4 pseudo-root (required for client enumeration).
  • crossmnt: Allows traversal into sub-datasets (e.g., vault/Data → vault/Data/media_books).
  • nohide: Required for vault/Data/media_books and vault/Data/media_comics to be visible to clients.
  • no_root_squash: Preserves root permissions (required for Docker Swarm).

Loopback Mount (ZNAS Only)

ZNAS mounts its own NFS exports back to itself at /data/nfs/znas to provide a consistent path for Docker Swarm containers, regardless of which node they land on. This is configured via ZFS mountpoints (not /etc/fstab):

vault/Common on /data/nfs/znas/Common type zfs (rw,noatime,xattr,posixacl,casesensitive)
vault/Data on /data/nfs/znas/Data type zfs (rw,noatime,xattr,posixacl,casesensitive)
vault/Docker on /data/nfs/znas/Docker type zfs (rw,noatime,xattr,posixacl,casesensitive)
vault/Green on /data/nfs/znas/Green type zfs (rw,noatime,xattr,posixacl,casesensitive)
vault/Photos on /data/nfs/znas/Photos type zfs (rw,noatime,xattr,posixacl,casesensitive)
vault/Pocket on /data/nfs/znas/Pocket type zfs (rw,noatime,xattr,posixacl,casesensitive)
vault/Data/media_books on /data/nfs/znas/Data/media/books type zfs (rw,noatime,xattr,posixacl,casesensitive)
vault/Data/media_comics on /data/nfs/znas/Data/media/comics type zfs (rw,noatime,xattr,posixacl,casesensitive)
vault/pocket/green/media on /data/nfs/znas/Green/Pocket type zfs (rw,noatime,xattr,posixacl,casesensitive)

---

Client Autofs Mounts

Configuration

All Swarm nodes (including ZNAS) use autofs to mount NFS exports on-demand at /data/nfs/znas. This ensures consistent paths across all nodes.

/etc/auto.master

/data/nfs    /etc/auto.nfs

/etc/auto.nfs

znas    -fstype=nfs4    192.168.5.10:/

Client Behavior

• On-demand mounting: Autofs mounts the NFSv4 pseudo-root (/) from ZNAS at /data/nfs/znas, then traverses into subdirectories as needed.
• Consistency: All nodes (including ZNAS) access storage via /data/nfs/znas/, ensuring Docker Swarm containers use the same paths regardless of scheduling.
• Example mounts (from docker5):

  192.168.5.10:/ on /data/nfs/znas type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,fatal_neterrors=none,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.5.18,local_lock=none,addr=192.168.5.10)
  192.168.5.10:/Data on /data/nfs/znas/Data type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,fatal_neterrors=none,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.5.18,local_lock=none,addr=192.168.5.10)
  192.168.5.10:/Data/media/books on /data/nfs/znas/Data/media/books type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,fatal_neterrors=none,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.5.18,local_lock=none,addr=192.168.5.10)
  

---

Loop-Mounted Images

None detected. Live state confirms no loop devices are in use on ZNAS:

losetup -a  # No output
mount | grep loop  # No output

---

Pocket ZFS Sync Mechanism

Overview

The Pocket Grimoire portable system (a 5TB SSD) is synced from ZNAS using syncoid (part of the zfs-auto-snapshot package). The sync is unidirectional: vault/pocket/* → pocket-media/*.

Sync Targets

Source Dataset	Destination Dataset	Purpose
vault/pocket/green/media	pocket-media/media/green	Green Grimoire media
vault/pocket/media/reference	pocket-media/media/reference	Reference media
vault/pocket/services/DockerVol	pocket-media/services/DockerVol	Docker volumes
vault/pocket/services/stacks	pocket-media/services/stacks	Swarm stack files

Sync Command

The sync is triggered by the "Gremlin Pocket Dismount" n8n workflow via SSH:

syncoid -r vault/pocket/services/DockerVol pocket-green/services/DockerVol

• -r: Recursive sync.
• No compression or encryption is used (local network).

Current State

• The Pocket SSD (pocket-media pool) is physically docked to ZNAS as of 2026-07-08 (mid-migration).
• The mycroft host (Pocket Grimoire) is unreachable and presumed offline.

---

Discrepancies Noted

1. Dataset Layout:

   • Memory: vault/Green/Pocket was listed as /export/Green/Pocket.
   • Live State: vault/Green/Pocket is mounted at /export/Green/Pocket, but vault/pocket/green/media is mounted at /data/nfs/znas/Green/Pocket (loopback). This suggests the Pocket sync source is now vault/pocket/green/media.

2. NFS Exports:

   • Memory: /export/Pocket was not listed in the exports file.
   • Live State: /export/Pocket is explicitly exported (fsid=32).

3. Pocket Sync:

   • Memory: No mention of vault/pocket/services/DockerVol or vault/pocket/services/stacks.
   • Live State: These datasets exist and are synced to Pocket.

4. Loopback Mounts:

   • Memory: Suggested loopback mounts were in /etc/fstab.
   • Live State: Loopback mounts are handled by ZFS mountpoints (not /etc/fstab).

> **AUTO-GENERATED DRAFT -- NOT YET PUBLISHED.** This page was written automatically by the `@document` pilot (chat -> n8n -> Mistral via gremlin-proxy) from a mix of prior memory notes and one live SSH inspection pass. Read it carefully before manually committing it to the wiki -- do not trust it blindly, especially anything under "Discrepancies noted". ```markdown # ZNAS Filesystem Topology ## Overview ZNAS is the primary storage system for NetGrimoire, running Ubuntu with OpenZFS. It serves as the source of truth for all storage, including datasets that replicate to the Pocket Grimoire portable system. All NFS exports are ZFS datasets mounted directly under `/export/`. --- ## ZFS Pools and Datasets ### Pools ZNAS has two ZFS pools: 1. **`vault`** (primary pool) - ~129TB raw capacity (70% used) - Two RAIDZ1 VDEVs: - 3×24TB drives - 5×16TB drives - Compression: `zstd` (default), `lz4` (select datasets) - Deduplication: off 2. **`pocket-media`** (Pocket Grimoire SSD) - 4.55TB single-disk pool - Mounted at `/mnt/pm/mnt/pocket-media` - Compression: `lz4` - Deduplication: off ### Datasets (Live State) #### `vault` Pool | Dataset | Mountpoint | Purpose | Compression | Notes | |-----------------------------|--------------------------------|----------------------------------|-------------|--------------------------------| | `vault` | `/export` | NFS root | zstd | | | `vault/Common` | `/export/Common` | Shared documents, configs | zstd | | | `vault/Data` | `/export/Data` | Media root | lz4 | | | `vault/Data/media_books` | `/export/Data/media/books` | Books library | zstd | `nohide` in NFS exports | | `vault/Data/media_comics` | `/export/Data/media/comics` | Comics library | zstd | `nohide` in NFS exports | | `vault/Docker` | `/export/Docker` | Swarm volumes | zstd | | | `vault/Green` | `/export/Green` | Green Grimoire media | off | | | `vault/Green/Pocket` | `/export/Green/Pocket` | Pocket sync source | off | | | `vault/Photos` | `/export/Photos` | Photo library | zstd | | | `vault/Kopia` | `/srv/vault/kopia_repository` | Backup repository | zstd | Local-only (not NFS-exported) | | `vault/NextCloud-AIO` | `/srv/NextCloud-AIO` | NextCloud data | lz4 | Local-only | | `vault/backup` | `/srv/vault/backup` | Local backups | zstd | Local-only | | `vault/immich` | `/srv/immich` | Immich photo library | lz4 | Local-only | | `vault/pocket` | `none` | Pocket sync parent | zstd | | | `vault/pocket/green` | `/pocket/green` | Green Grimoire sync target | zstd | | | `vault/pocket/green/media` | `/pocket/green/media` | Media sync target | zstd | | | `vault/pocket/media` | `/pocket/media` | Media sync target | zstd | | | `vault/pocket/services` | `/pocket/services` | Service data sync target | zstd | | #### `pocket-media` Pool | Dataset | Mountpoint | Purpose | |--------------------------------------|-------------------------------------------------|----------------------------------| | `pocket-media` | `/mnt/pm/mnt/pocket-media` | Root | | `pocket-media/media` | `/mnt/pm/mnt/pocket-media/media` | Media root | | `pocket-media/media/movies` | `/mnt/pm/mnt/pocket-media/media/movies` | Movies library | | `pocket-media/backups` | `/mnt/pm/mnt/pocket-media/backups` | Backup root | | `pocket-media/backups/documents` | `/mnt/pm/mnt/pocket-media/backups/documents` | Documents backup | | `pocket-media/backups/photos` | `/mnt/pm/mnt/pocket-media/backups/photos` | Photos backup | | `pocket-media/backups/wiki` | `/mnt/pm/mnt/pocket-media/backups/wiki` | Wiki backup | | `pocket-media/wiki-pg` | `/mnt/pm/mnt/pocket-media/wiki-pg` | Wiki database | --- ## NFS Exports ### Server Configuration - **NFSv4 only**: All exports use NFSv4 with a pseudo-root at `/export`. - **Exports file** (`/etc/exports`): ```plaintext /export *(rw,fsid=0,no_root_squash,no_subtree_check,crossmnt) /export/Common *(fsid=4,rw,no_root_squash,no_subtree_check,insecure) /export/Data *(fsid=5,rw,no_root_squash,no_subtree_check,insecure,crossmnt) /export/Data/media/books *(fsid=51,rw,no_root_squash,no_subtree_check,insecure,nohide) /export/Data/media/comics *(fsid=52,rw,no_root_squash,no_subtree_check,insecure,nohide) /export/Docker *(fsid=29,rw,no_root_squash,sync,no_subtree_check,insecure) /export/Green *(fsid=30,rw,no_root_squash,no_subtree_check,insecure,crossmnt) /export/photos *(fsid=31,rw,no_root_squash,no_subtree_check,insecure) /export/Pocket *(fsid=32,rw,no_root_squash,no_subtree_check,insecure) /export/Grimoire *(fsid=33,rw,no_root_squash,no_subtree_check,insecure) /export/green/media/pocket *(fsid=35,rw,no_root_squash,no_subtree_check,insecure) ``` - **Key options**: - `fsid=0`: NFSv4 pseudo-root (required for client enumeration). - `crossmnt`: Allows traversal into sub-datasets (e.g., `vault/Data` → `vault/Data/media_books`). - `nohide`: Required for `vault/Data/media_books` and `vault/Data/media_comics` to be visible to clients. - `no_root_squash`: Preserves root permissions (required for Docker Swarm). ### Loopback Mount (ZNAS Only) ZNAS mounts its own NFS exports back to itself at `/data/nfs/znas` to provide a consistent path for Docker Swarm containers, regardless of which node they land on. This is configured via ZFS mountpoints (not `/etc/fstab`): ```plaintext vault/Common on /data/nfs/znas/Common type zfs (rw,noatime,xattr,posixacl,casesensitive) vault/Data on /data/nfs/znas/Data type zfs (rw,noatime,xattr,posixacl,casesensitive) vault/Docker on /data/nfs/znas/Docker type zfs (rw,noatime,xattr,posixacl,casesensitive) vault/Green on /data/nfs/znas/Green type zfs (rw,noatime,xattr,posixacl,casesensitive) vault/Photos on /data/nfs/znas/Photos type zfs (rw,noatime,xattr,posixacl,casesensitive) vault/Pocket on /data/nfs/znas/Pocket type zfs (rw,noatime,xattr,posixacl,casesensitive) vault/Data/media_books on /data/nfs/znas/Data/media/books type zfs (rw,noatime,xattr,posixacl,casesensitive) vault/Data/media_comics on /data/nfs/znas/Data/media/comics type zfs (rw,noatime,xattr,posixacl,casesensitive) vault/pocket/green/media on /data/nfs/znas/Green/Pocket type zfs (rw,noatime,xattr,posixacl,casesensitive) ``` --- ## Client Autofs Mounts ### Configuration All Swarm nodes (including ZNAS) use **autofs** to mount NFS exports on-demand at `/data/nfs/znas`. This ensures consistent paths across all nodes. #### `/etc/auto.master` ```plaintext /data/nfs /etc/auto.nfs ``` #### `/etc/auto.nfs` ```plaintext znas -fstype=nfs4 192.168.5.10:/ ``` ### Client Behavior - **On-demand mounting**: Autofs mounts the NFSv4 pseudo-root (`/`) from ZNAS at `/data/nfs/znas`, then traverses into subdirectories as needed. - **Consistency**: All nodes (including ZNAS) access storage via `/data/nfs/znas/`, ensuring Docker Swarm containers use the same paths regardless of scheduling. - **Example mounts** (from `docker5`): ```plaintext 192.168.5.10:/ on /data/nfs/znas type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,fatal_neterrors=none,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.5.18,local_lock=none,addr=192.168.5.10) 192.168.5.10:/Data on /data/nfs/znas/Data type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,fatal_neterrors=none,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.5.18,local_lock=none,addr=192.168.5.10) 192.168.5.10:/Data/media/books on /data/nfs/znas/Data/media/books type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,fatal_neterrors=none,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.5.18,local_lock=none,addr=192.168.5.10) ``` --- ## Loop-Mounted Images **None detected**. Live state confirms no loop devices are in use on ZNAS: ```bash losetup -a # No output mount | grep loop # No output ``` --- ## Pocket ZFS Sync Mechanism ### Overview The Pocket Grimoire portable system (a 5TB SSD) is synced from ZNAS using **syncoid** (part of the `zfs-auto-snapshot` package). The sync is unidirectional: `vault/pocket/*` → `pocket-media/*`. ### Sync Targets | Source Dataset | Destination Dataset | Purpose | |------------------------------------|-----------------------------------------|----------------------------------| | `vault/pocket/green/media` | `pocket-media/media/green` | Green Grimoire media | | `vault/pocket/media/reference` | `pocket-media/media/reference` | Reference media | | `vault/pocket/services/DockerVol` | `pocket-media/services/DockerVol` | Docker volumes | | `vault/pocket/services/stacks` | `pocket-media/services/stacks` | Swarm stack files | ### Sync Command The sync is triggered by the **"Gremlin Pocket Dismount"** n8n workflow via SSH: ```bash syncoid -r vault/pocket/services/DockerVol pocket-green/services/DockerVol ``` - `-r`: Recursive sync. - No compression or encryption is used (local network). ### Current State - The Pocket SSD (`pocket-media` pool) is **physically docked to ZNAS** as of 2026-07-08 (mid-migration). - The `mycroft` host (Pocket Grimoire) is **unreachable** and presumed offline. --- ## Discrepancies Noted 1. **Dataset Layout**: - **Memory**: `vault/Green/Pocket` was listed as `/export/Green/Pocket`. - **Live State**: `vault/Green/Pocket` is mounted at `/export/Green/Pocket`, but `vault/pocket/green/media` is mounted at `/data/nfs/znas/Green/Pocket` (loopback). This suggests the Pocket sync source is now `vault/pocket/green/media`. 2. **NFS Exports**: - **Memory**: `/export/Pocket` was not listed in the exports file. - **Live State**: `/export/Pocket` is explicitly exported (fsid=32). 3. **Pocket Sync**: - **Memory**: No mention of `vault/pocket/services/DockerVol` or `vault/pocket/services/stacks`. - **Live State**: These datasets exist and are synced to Pocket. 4. **Loopback Mounts**: - **Memory**: Suggested loopback mounts were in `/etc/fstab`. - **Live State**: Loopback mounts are handled by ZFS mountpoints (not `/etc/fstab`). ```

traveler commented 2026-07-15 18:18:58 -05:00

Author

Owner

Copy link

Superseded by #16: same run fixed the pool-capacity/pocket-pool content issues found in this draft (see the ntfy/staging debug session), plus the notification-routing bug that meant this draft's ntfy ping never reached a channel you were watching. Closing in favor of #16.

Superseded by #16: same run fixed the pool-capacity/pocket-pool content issues found in this draft (see the ntfy/staging debug session), plus the notification-routing bug that meant this draft's ntfy ping never reached a channel you were watching. Closing in favor of #16.

traveler closed this issue 2026-07-15 18:18:58 -05:00

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

gremlin/pocket-multiarch-relocate-docs

gremlin/pocket-hardware-build-doc

gremlin/pocket-hw-and-ops-corrections

No results found.

Labels

Clear labels   

doc-staleness

No labels

doc-staleness

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

traveler/Netgrimoire#15

No description provided.