traveler/Netgrimoire

Fork 0

[Docs Draft] znas filesystem stream-test -- pending review (20260715-195522-docs-znas-7414) #14

New issue

Open

opened 2026-07-15 14:56:20 -05:00 by traveler · 0 comments

traveler commented

2026-07-15 14:56:20 -05:00

Owner

AUTO-GENERATED DRAFT -- NOT YET PUBLISHED. This page was written automatically by the @document pilot (chat -> n8n -> Mistral via gremlin-proxy) from a mix of prior memory notes and one live SSH inspection pass. Read it carefully before manually committing it to the wiki -- do not trust it blindly, especially anything under "Discrepancies noted".

# ZNAS Filesystem Topology

## Overview
ZNAS is the primary NAS for NetGrimoire, running Ubuntu with OpenZFS. It serves as the source of truth for all storage, including datasets that replicate to the Pocket Grimoire portable system.

All NFS exports are ZFS datasets mounted directly under `/export/`. ZNAS mounts its own NFS exports back to itself at `/data/nfs/znas` to provide consistent NFS-backed paths for Docker Swarm containers regardless of which node they land on.

---

## ZFS Pools and Datasets

### Pools
| Pool           | Size   | Alloc  | Free   | Health  | Notes                                  |
|----------------|--------|--------|--------|---------|----------------------------------------|
| `vault`        | 129T   | 91.5T  | 37.7T  | ONLINE  | Primary storage, two RAIDZ1 VDEVs      |
| `pocket-media` | 4.55T  | 0      | 4.55T  | ONLINE  | Pocket Grimoire media SSD (currently docked to ZNAS) |

### Datasets (Live State)
#### `vault` Pool
| Dataset                     | Mountpoint                     | Compression | Notes                                  |
|-----------------------------|--------------------------------|-------------|----------------------------------------|
| `vault`                     | `/export`                      | zstd        | NFSv4 pseudo-root                      |
| `vault/Common`              | `/export/Common`               | zstd        |                                        |
| `vault/Data`                | `/export/Data`                 | lz4         |                                        |
| `vault/Data/media_books`    | `/export/Data/media/books`     | zstd        | `nohide` in NFS exports                |
| `vault/Data/media_comics`   | `/export/Data/media/comics`    | zstd        | `nohide` in NFS exports                |
| `vault/Docker`              | `/export/Docker`               | zstd        | Swarm persistent volumes               |
| `vault/Green`               | `/export/Green`                | off         |                                        |
| `vault/Green/Pocket`        | `/export/Green/Pocket`         | off         |                                        |
| `vault/Kopia`               | `/srv/vault/kopia_repository`  | zstd        | Local Kopia repo                       |
| `vault/NextCloud-AIO`       | `/srv/NextCloud-AIO`           | lz4         |                                        |
| `vault/Photos`              | `/export/Photos`               | zstd        |                                        |
| `vault/backup`              | `/srv/vault/backup`            | zstd        |                                        |
| `vault/immich`              | `/srv/immich`                  | lz4         |                                        |
| `vault/pocket`              | `none`                         | zstd        | Parent dataset for Pocket sync         |
| `vault/pocket/green`        | `/pocket/green`                | zstd        |                                        |
| `vault/pocket/green/media`  | `/pocket/green/media`          | zstd        |                                        |
| `vault/pocket/media`        | `/pocket/media`                | zstd        |                                        |
| `vault/pocket/services`     | `/pocket/services`             | zstd        | Docker volumes for Pocket Grimoire     |

#### `pocket-media` Pool
| Dataset                              | Mountpoint                                      | Compression | Notes                                  |
|--------------------------------------|-------------------------------------------------|-------------|----------------------------------------|
| `pocket-media`                       | `/mnt/pm/mnt/pocket-media`                      | lz4         |                                        |
| `pocket-media/backups`               | `/mnt/pm/mnt/pocket-media/backups`              | lz4         |                                        |
| `pocket-media/media`                 | `/mnt/pm/mnt/pocket-media/media`                | lz4         |                                        |
| `pocket-media/media/green`           | `/mnt/pm/mnt/pocket-media/media/green`          | lz4         |                                        |
| `pocket-media/media/movies`          | `/mnt/pm/mnt/pocket-media/media/movies`         | lz4         |                                        |
| `pocket-media/media/tv`              | `/mnt/pm/mnt/pocket-media/media/tv`             | lz4         |                                        |
| `pocket-media/wiki-pg`               | `/mnt/pm/mnt/pocket-media/wiki-pg`              | lz4         |                                        |

---

## NFS Exports

### Server Configuration (`/etc/exports`)
```plaintext
/export *(rw,fsid=0,no_root_squash,no_subtree_check,crossmnt)
/export/Common *(fsid=4,rw,no_root_squash,no_subtree_check,insecure)
/export/Data *(fsid=5,rw,no_root_squash,no_subtree_check,insecure,crossmnt)
/export/Data/media/books *(fsid=51,rw,no_root_squash,no_subtree_check,insecure,nohide)
/export/Data/media/comics *(fsid=52,rw,no_root_squash,no_subtree_check,insecure,nohide)
/export/Docker *(fsid=29,rw,no_root_squash,sync,no_subtree_check,insecure)
/export/Green *(fsid=30,rw,no_root_squash,no_subtree_check,insecure,crossmnt)
/export/photos *(fsid=31,rw,no_root_squash,no_subtree_check,insecure)
/export/Pocket *(fsid=32,rw,no_root_squash,no_subtree_check,insecure)
/export/Grimoire *(fsid=33,rw,no_root_squash,no_subtree_check,insecure)
/export/green/media/pocket *(fsid=35,rw,no_root_squash,no_subtree_check,insecure)

Key NFS Configuration Notes

NFSv4 pseudo-root (fsid=0) is required for clients to enumerate the export tree.
nohide is used for vault/Data/media_books and vault/Data/media_comics to ensure NFS crosses ZFS dataset boundaries.
crossmnt is used where child datasets are mounted beneath the parent export path.
No bind mounts are used for ZFS datasets. All datasets mount directly to /export/*.

Loopback Mount (ZNAS)

ZNAS mounts its own NFS exports back to itself at /data/nfs/znas to provide consistent NFS-backed paths for Docker Swarm containers.

Mounts (Live State)

vault/Common on /data/nfs/znas/Common type zfs (rw,noatime,xattr,posixacl,casesensitive)
vault/Data on /data/nfs/znas/Data type zfs (rw,noatime,xattr,posixacl,casesensitive)
vault/docker on /data/nfs/znas/Docker type zfs (rw,noatime,xattr,posixacl,casesensitive)
vault/Green on /data/nfs/znas/Green type zfs (rw,noatime,xattr,posixacl,casesensitive)
vault on /data/nfs/znas/Grimoire type zfs (rw,noatime,xattr,posixacl,casesensitive)
vault/Photos on /data/nfs/znas/Photos type zfs (rw,noatime,xattr,posixacl,casesensitive)
vault on /data/nfs/znas/Pocket type zfs (rw,noatime,xattr,posixacl,casesensitive)
vault/Data/media_books on /data/nfs/znas/Data/media/books type zfs (rw,noatime,xattr,posixacl,casesensitive)
vault/Data/media_comics on /data/nfs/znas/Data/media/comics type zfs (rw,noatime,xattr,posixacl,casesensitive)
vault/pocket/green/media on /data/nfs/znas/Green/Pocket type zfs (rw,noatime,xattr,posixacl,casesensitive)

Autofs Mounts (Swarm Nodes)

Client Configuration

All Swarm nodes (including ZNAS) use autofs to mount NFS exports on demand.

`/etc/auto.master`

/data/nfs /etc/auto.nfs

`/etc/auto.nfs`

znas -fstype=nfs4 192.168.5.10:/

Mounts (Live State)

`docker3` (192.168.5.15)

192.168.5.10:/ on /data/nfs/znas type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.5.15,local_lock=none,addr=192.168.5.10)

`docker4` (192.168.5.16)

192.168.5.10:/Docker on /data/nfs/znas/Docker type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.5.16,local_lock=none,addr=192.168.5.10)
192.168.5.10:/Grimoire on /data/nfs/znas/Grimoire type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.5.16,local_lock=none,addr=192.168.5.10)
192.168.5.10:/Data on /data/nfs/znas/Data type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.5.16,local_lock=none,addr=192.168.5.10)

`docker5` (192.168.5.18)

192.168.5.10:/ on /data/nfs/znas type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,fatal_neterrors=none,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.5.18,local_lock=none,addr=192.168.5.10)
192.168.5.10:/Data on /data/nfs/znas/Data type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,fatal_neterrors=none,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.5.18,local_lock=none,addr=192.168.5.10)
192.168.5.10:/Data/media/books on /data/nfs/znas/Data/media/books type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,fatal_neterrors=none,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.5.18,local_lock=none,addr=192.168.5.10)

Loop-Mounted Images

None detected. Live state shows no active loop devices (losetup -a empty).

Pocket ZFS Sync Mechanism

Overview

The Pocket Grimoire portable system receives ZFS dataset replications from ZNAS via syncoid. The vault/pocket dataset tree is replicated to the pocket-green pool on the Pocket SSD.

Sync Command (Live State)

syncoid -r vault/pocket/services/DockerVol pocket-green/services/DockerVol

Sync Workflow

Trigger: The sync is invoked by the "Gremlin Pocket Dismount" n8n workflow's "SSH: Sync" node.
Source: vault/pocket dataset tree (ZNAS).
Destination: pocket-green pool (Pocket SSD, currently docked to ZNAS).
Scope: Currently syncs vault/pocket/services/DockerVol only. Other datasets (e.g., vault/pocket/green/media) are not synced via syncoid in the live workflow.

Discrepancies Noted

MEMORY: Lists /export/Pocket as an export, but LIVE STATE shows /export/Pocket is exported but not mounted in the loopback (mount | grep nfs on ZNAS). The vault/pocket dataset is mounted at /pocket/* instead.
MEMORY: Lists /export/Grimoire as a ZFS dataset, but LIVE STATE shows it is a ZFS mountpoint (vault on /data/nfs/znas/Grimoire) with no dedicated dataset.
MEMORY: Lists /export/green/media/pocket as an export, but LIVE STATE shows it is commented out in /etc/exports and not exported.
MEMORY: Lists vault/Green/Pocket as a dataset, but LIVE STATE shows vault/Green/Pocket is mounted at /export/Green/Pocket and also loopback-mounted at /data/nfs/znas/Green/Pocket via vault/pocket/green/media.
MEMORY: No mention of vault/pocket dataset tree or pocket-media pool, which are present in LIVE STATE.

> **AUTO-GENERATED DRAFT -- NOT YET PUBLISHED.** This page was written automatically by the `@document` pilot (chat -> n8n -> Mistral via gremlin-proxy) from a mix of prior memory notes and one live SSH inspection pass. Read it carefully before manually committing it to the wiki -- do not trust it blindly, especially anything under "Discrepancies noted". ```markdown # ZNAS Filesystem Topology ## Overview ZNAS is the primary NAS for NetGrimoire, running Ubuntu with OpenZFS. It serves as the source of truth for all storage, including datasets that replicate to the Pocket Grimoire portable system. All NFS exports are ZFS datasets mounted directly under `/export/`. ZNAS mounts its own NFS exports back to itself at `/data/nfs/znas` to provide consistent NFS-backed paths for Docker Swarm containers regardless of which node they land on. --- ## ZFS Pools and Datasets ### Pools | Pool | Size | Alloc | Free | Health | Notes | |----------------|--------|--------|--------|---------|----------------------------------------| | `vault` | 129T | 91.5T | 37.7T | ONLINE | Primary storage, two RAIDZ1 VDEVs | | `pocket-media` | 4.55T | 0 | 4.55T | ONLINE | Pocket Grimoire media SSD (currently docked to ZNAS) | ### Datasets (Live State) #### `vault` Pool | Dataset | Mountpoint | Compression | Notes | |-----------------------------|--------------------------------|-------------|----------------------------------------| | `vault` | `/export` | zstd | NFSv4 pseudo-root | | `vault/Common` | `/export/Common` | zstd | | | `vault/Data` | `/export/Data` | lz4 | | | `vault/Data/media_books` | `/export/Data/media/books` | zstd | `nohide` in NFS exports | | `vault/Data/media_comics` | `/export/Data/media/comics` | zstd | `nohide` in NFS exports | | `vault/Docker` | `/export/Docker` | zstd | Swarm persistent volumes | | `vault/Green` | `/export/Green` | off | | | `vault/Green/Pocket` | `/export/Green/Pocket` | off | | | `vault/Kopia` | `/srv/vault/kopia_repository` | zstd | Local Kopia repo | | `vault/NextCloud-AIO` | `/srv/NextCloud-AIO` | lz4 | | | `vault/Photos` | `/export/Photos` | zstd | | | `vault/backup` | `/srv/vault/backup` | zstd | | | `vault/immich` | `/srv/immich` | lz4 | | | `vault/pocket` | `none` | zstd | Parent dataset for Pocket sync | | `vault/pocket/green` | `/pocket/green` | zstd | | | `vault/pocket/green/media` | `/pocket/green/media` | zstd | | | `vault/pocket/media` | `/pocket/media` | zstd | | | `vault/pocket/services` | `/pocket/services` | zstd | Docker volumes for Pocket Grimoire | #### `pocket-media` Pool | Dataset | Mountpoint | Compression | Notes | |--------------------------------------|-------------------------------------------------|-------------|----------------------------------------| | `pocket-media` | `/mnt/pm/mnt/pocket-media` | lz4 | | | `pocket-media/backups` | `/mnt/pm/mnt/pocket-media/backups` | lz4 | | | `pocket-media/media` | `/mnt/pm/mnt/pocket-media/media` | lz4 | | | `pocket-media/media/green` | `/mnt/pm/mnt/pocket-media/media/green` | lz4 | | | `pocket-media/media/movies` | `/mnt/pm/mnt/pocket-media/media/movies` | lz4 | | | `pocket-media/media/tv` | `/mnt/pm/mnt/pocket-media/media/tv` | lz4 | | | `pocket-media/wiki-pg` | `/mnt/pm/mnt/pocket-media/wiki-pg` | lz4 | | --- ## NFS Exports ### Server Configuration (`/etc/exports`) ```plaintext /export *(rw,fsid=0,no_root_squash,no_subtree_check,crossmnt) /export/Common *(fsid=4,rw,no_root_squash,no_subtree_check,insecure) /export/Data *(fsid=5,rw,no_root_squash,no_subtree_check,insecure,crossmnt) /export/Data/media/books *(fsid=51,rw,no_root_squash,no_subtree_check,insecure,nohide) /export/Data/media/comics *(fsid=52,rw,no_root_squash,no_subtree_check,insecure,nohide) /export/Docker *(fsid=29,rw,no_root_squash,sync,no_subtree_check,insecure) /export/Green *(fsid=30,rw,no_root_squash,no_subtree_check,insecure,crossmnt) /export/photos *(fsid=31,rw,no_root_squash,no_subtree_check,insecure) /export/Pocket *(fsid=32,rw,no_root_squash,no_subtree_check,insecure) /export/Grimoire *(fsid=33,rw,no_root_squash,no_subtree_check,insecure) /export/green/media/pocket *(fsid=35,rw,no_root_squash,no_subtree_check,insecure) ``` ### Key NFS Configuration Notes - **NFSv4 pseudo-root (`fsid=0`)** is required for clients to enumerate the export tree. - **`nohide`** is used for `vault/Data/media_books` and `vault/Data/media_comics` to ensure NFS crosses ZFS dataset boundaries. - **`crossmnt`** is used where child datasets are mounted beneath the parent export path. - **No bind mounts** are used for ZFS datasets. All datasets mount directly to `/export/*`. --- ## Loopback Mount (ZNAS) ZNAS mounts its own NFS exports back to itself at `/data/nfs/znas` to provide consistent NFS-backed paths for Docker Swarm containers. ### Mounts (Live State) ```plaintext vault/Common on /data/nfs/znas/Common type zfs (rw,noatime,xattr,posixacl,casesensitive) vault/Data on /data/nfs/znas/Data type zfs (rw,noatime,xattr,posixacl,casesensitive) vault/docker on /data/nfs/znas/Docker type zfs (rw,noatime,xattr,posixacl,casesensitive) vault/Green on /data/nfs/znas/Green type zfs (rw,noatime,xattr,posixacl,casesensitive) vault on /data/nfs/znas/Grimoire type zfs (rw,noatime,xattr,posixacl,casesensitive) vault/Photos on /data/nfs/znas/Photos type zfs (rw,noatime,xattr,posixacl,casesensitive) vault on /data/nfs/znas/Pocket type zfs (rw,noatime,xattr,posixacl,casesensitive) vault/Data/media_books on /data/nfs/znas/Data/media/books type zfs (rw,noatime,xattr,posixacl,casesensitive) vault/Data/media_comics on /data/nfs/znas/Data/media/comics type zfs (rw,noatime,xattr,posixacl,casesensitive) vault/pocket/green/media on /data/nfs/znas/Green/Pocket type zfs (rw,noatime,xattr,posixacl,casesensitive) ``` --- ## Autofs Mounts (Swarm Nodes) ### Client Configuration All Swarm nodes (including ZNAS) use autofs to mount NFS exports on demand. #### `/etc/auto.master` ```plaintext /data/nfs /etc/auto.nfs ``` #### `/etc/auto.nfs` ```plaintext znas -fstype=nfs4 192.168.5.10:/ ``` ### Mounts (Live State) #### `docker3` (192.168.5.15) ```plaintext 192.168.5.10:/ on /data/nfs/znas type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.5.15,local_lock=none,addr=192.168.5.10) ``` #### `docker4` (192.168.5.16) ```plaintext 192.168.5.10:/Docker on /data/nfs/znas/Docker type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.5.16,local_lock=none,addr=192.168.5.10) 192.168.5.10:/Grimoire on /data/nfs/znas/Grimoire type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.5.16,local_lock=none,addr=192.168.5.10) 192.168.5.10:/Data on /data/nfs/znas/Data type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.5.16,local_lock=none,addr=192.168.5.10) ``` #### `docker5` (192.168.5.18) ```plaintext 192.168.5.10:/ on /data/nfs/znas type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,fatal_neterrors=none,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.5.18,local_lock=none,addr=192.168.5.10) 192.168.5.10:/Data on /data/nfs/znas/Data type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,fatal_neterrors=none,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.5.18,local_lock=none,addr=192.168.5.10) 192.168.5.10:/Data/media/books on /data/nfs/znas/Data/media/books type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,fatal_neterrors=none,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.5.18,local_lock=none,addr=192.168.5.10) ``` --- ## Loop-Mounted Images **None detected.** Live state shows no active loop devices (`losetup -a` empty). --- ## Pocket ZFS Sync Mechanism ### Overview The Pocket Grimoire portable system receives ZFS dataset replications from ZNAS via `syncoid`. The `vault/pocket` dataset tree is replicated to the `pocket-green` pool on the Pocket SSD. ### Sync Command (Live State) ```bash syncoid -r vault/pocket/services/DockerVol pocket-green/services/DockerVol ``` ### Sync Workflow 1. **Trigger**: The sync is invoked by the "Gremlin Pocket Dismount" n8n workflow's "SSH: Sync" node. 2. **Source**: `vault/pocket` dataset tree (ZNAS). 3. **Destination**: `pocket-green` pool (Pocket SSD, currently docked to ZNAS). 4. **Scope**: Currently syncs `vault/pocket/services/DockerVol` only. Other datasets (e.g., `vault/pocket/green/media`) are not synced via `syncoid` in the live workflow. --- ## Discrepancies Noted 1. **MEMORY**: Lists `/export/Pocket` as an export, but **LIVE STATE** shows `/export/Pocket` is exported but not mounted in the loopback (`mount | grep nfs` on ZNAS). The `vault/pocket` dataset is mounted at `/pocket/*` instead. 2. **MEMORY**: Lists `/export/Grimoire` as a ZFS dataset, but **LIVE STATE** shows it is a ZFS mountpoint (`vault on /data/nfs/znas/Grimoire`) with no dedicated dataset. 3. **MEMORY**: Lists `/export/green/media/pocket` as an export, but **LIVE STATE** shows it is commented out in `/etc/exports` and not exported. 4. **MEMORY**: Lists `vault/Green/Pocket` as a dataset, but **LIVE STATE** shows `vault/Green/Pocket` is mounted at `/export/Green/Pocket` and also loopback-mounted at `/data/nfs/znas/Green/Pocket` via `vault/pocket/green/media`. 5. **MEMORY**: No mention of `vault/pocket` dataset tree or `pocket-media` pool, which are present in **LIVE STATE**. ```

No labels

doc-staleness

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

traveler/Netgrimoire#14

No description provided.

Rows
Columns

[Docs Draft] znas filesystem stream-test -- pending review (20260715-195522-docs-znas-7414) #14

Key NFS Configuration Notes

Loopback Mount (ZNAS)

Mounts (Live State)

Autofs Mounts (Swarm Nodes)

Client Configuration

/etc/auto.master

/etc/auto.nfs

Mounts (Live State)

docker3 (192.168.5.15)

docker4 (192.168.5.16)

docker5 (192.168.5.18)

Loop-Mounted Images

Pocket ZFS Sync Mechanism

Overview

Sync Command (Live State)

Sync Workflow

Discrepancies Noted

`/etc/auto.master`

`/etc/auto.nfs`

`docker3` (192.168.5.15)

`docker4` (192.168.5.16)

`docker5` (192.168.5.18)