diff --git a/Netgrimoire/Storage/ZNAS_NFS_Exports.md b/Netgrimoire/Storage/ZNAS_NFS_Exports.md index 51f94a7..9dccbf0 100644 --- a/Netgrimoire/Storage/ZNAS_NFS_Exports.md +++ b/Netgrimoire/Storage/ZNAS_NFS_Exports.md @@ -2,170 +2,287 @@ title: ZFS-NFS-Exports description: Exporting NFS shares from ZFS datasets published: true -date: 2026-02-01T20:45:40.210Z +date: 2026-02-23T21:58:11.949Z tags: editor: markdown dateCreated: 2026-02-01T20:45:40.210Z --- -# NFS + ZFS Configuration Fix +# NFS Configuration -## Problems Identified +## Overview -1. **Boot order issue**: NFS was starting before ZFS mounted the datasets -2. **Autofs recursive loop**: The server was NFS-mounting its own exports back to itself via autofs, creating conflicts -3. **Mountpoint mismatch**: ZFS datasets were mounting to `/srv/vault/*` but NFS was trying to export from `/export/*` using bind mounts that didn't work properly -4. **Child dataset mountpoints**: When the parent dataset mountpoint changed, child datasets still had old mountpoints and weren't visible +ZNAS exports storage via NFSv4. All exports are ZFS datasets mounted directly to `/export/*` — no bind mounts. NFS is configured to wait for ZFS at boot via a systemd override. -## Complete Solution +ZNAS also mounts its own NFS exports back to itself at `/data/nfs/znas`. This is intentional: Docker Swarm containers scheduled to ZNAS need to access NAS storage at the same paths as containers running on other swarm members. The loopback mount provides a consistent NFS-backed path regardless of which node a container lands on. -### Step 1: Stop and Disable Autofs on NFS Server -```bash -sudo systemctl stop autofs -sudo systemctl disable autofs +All other clients are Linux systems using autofs. + +--- + +## Server Configuration + +### ZFS Mountpoints + +ZFS datasets mount directly to `/export/*`. No bind mounts are used. + +``` +vault → /export +vault/Common → /export/Common +vault/Data → /export/Data +vault/Data/media_books → /export/Data/media/books +vault/Data/media_comics → /export/Data/media/comics +vault/Docker → /export/Docker +vault/Green → /export/Green +vault/Green/Pocket → /export/Green/Pocket +vault/Photos → /export/Photos ``` -**Note**: Autofs should only run on NFS clients, not on the server itself. +Verify at any time: -### Step 2: Set ZFS Datasets to Mount Directly to `/export/*` - -Instead of using bind mounts, configure ZFS to mount directly to the export paths: ```bash -sudo zfs set mountpoint=/export vault -sudo zfs set mountpoint=/export/Data vault/Data -sudo zfs set mountpoint=/export/Green vault/Green -sudo zfs set mountpoint=/export/Docker vault/docker -sudo zfs set mountpoint=/export/Common vault/Common +mount | grep export ``` -### Step 3: Configure Child Dataset Mountpoints +### /etc/exports -For any child datasets (subdirectories under parent datasets), set their mountpoints explicitly: -```bash -sudo zfs set mountpoint=/export/Data/media/books vault/Data/media_books -sudo zfs set mountpoint=/export/Data/media/comics vault/Data/media_comics ``` - -### Step 4: Unmount and Remount Datasets - -Unmount children first, then parents, then remount all: -```bash -# Unmount child datasets first -sudo zfs unmount vault/Data/media_books -sudo zfs unmount vault/Data/media_comics - -# Unmount parent datasets -sudo zfs unmount vault/Data -sudo zfs unmount vault/Green -sudo zfs unmount vault/docker -sudo zfs unmount vault/Common - -# Remount all -sudo zfs mount -a -``` - -### Step 5: Configure NFS Exports - -Edit `/etc/exports`: -```bash -sudo nano /etc/exports -``` - -Content: -```bash -# /etc/exports: the access control list for filesystems which may be exported -# to NFS clients. See exports(5). - # NFSv4 - pseudo filesystem root -/export *(ro,fsid=0,crossmnt,no_subtree_check) +/export *(ro,fsid=0,no_root_squash,no_subtree_check,crossmnt) # Shares beneath the NFSv4 root /export/Common *(fsid=4,rw,no_subtree_check,insecure) -/export/Data *(fsid=5,rw,no_subtree_check,insecure,crossmnt,nohide) +/export/Data *(fsid=5,rw,no_subtree_check,insecure,crossmnt) +/export/Data/media/books *(fsid=51,rw,no_subtree_check,insecure,nohide) +/export/Data/media/comics *(fsid=52,rw,no_subtree_check,insecure,nohide) /export/Docker *(fsid=29,rw,no_root_squash,sync,no_subtree_check,insecure) /export/Green *(fsid=30,rw,no_root_squash,no_subtree_check,insecure) +/export/photos *(fsid=31,rw,no_root_squash,no_subtree_check,insecure) ``` -**Key options explained**: -- `crossmnt`: Allows NFS to cross filesystem mount boundaries -- `nohide`: Makes child mounts visible through parent exports -- `no_subtree_check`: Improves reliability and performance +**Key options:** -### Step 6: Configure NFS to Wait for ZFS at Boot +- `fsid=0` on `/export` — required for NFSv4 pseudo-root. Clients enumerate all exports from here. +- `crossmnt` — allows NFS to cross ZFS dataset boundaries when traversing the tree. +- `nohide` — required on `media/books` and `media/comics` because they are separate ZFS datasets mounted beneath the `vault/Data` export path. Without it clients see empty directories. +- `no_root_squash` — Docker and Green exports allow root writes. Required for container volume mounts. +- `insecure` — permits connections from unprivileged ports (>1024). Required for some Linux NFS clients and all macOS clients. +- `sync` on Docker — forces synchronous writes for container volume safety. -Create systemd override for NFS server: -```bash -sudo mkdir -p /etc/systemd/system/nfs-server.service.d/ -sudo nano /etc/systemd/system/nfs-server.service.d/override.conf -``` +### systemd Boot Order Override + +NFS is configured to wait for ZFS to fully mount before starting. + +`/etc/systemd/system/nfs-server.service.d/override.conf`: -Add this content: ```ini [Unit] After=zfs-import.target zfs-mount.service local-fs.target Requires=zfs-import.target zfs-mount.service ``` -This ensures NFS waits for ZFS to be fully mounted before starting. +Apply after any changes: -### Step 7: Reload and Restart Services ```bash sudo systemctl daemon-reload -sudo exportfs -ra sudo systemctl restart nfs-server ``` -### Step 8: Verify Configuration +### Autofs Disabled on Server + +Autofs is disabled on ZNAS itself. It must only run on NFS clients. Running autofs on the server creates recursive mount loops. -On the server: ```bash -# Check ZFS mounts -zfs list -r vault - -# Check what's actually mounted -mount | grep export - -# Verify NFS exports -sudo exportfs -v - -# Check content is visible -ls -la /export/Data/media/books/ +sudo systemctl stop autofs +sudo systemctl disable autofs ``` -On the client: +--- + +## Loopback Mount (Docker Swarm) + +ZNAS mounts its own NFS exports back to itself at `/data/nfs/znas`. This ensures containers scheduled to ZNAS by Docker Swarm access storage at the same NFS-backed paths as containers running on any other swarm member — consistent regardless of which node a service lands on. + +Swarm container volume mounts reference paths under `/data/nfs/znas/` rather than `/export/` directly. + +### The Timing Problem + +Getting this mount to survive reboots reliably was non-trivial. The loopback has a chicken-and-egg dependency chain: + +1. ZFS must import and mount pools before NFS server can export anything +2. NFS server must be fully started before the loopback mount can succeed +3. The loopback mount must be established before Docker Swarm containers start + +A plain `_netdev` fstab entry is not sufficient — `_netdev` only guarantees the network is up, not that the NFS server is ready. The mount would race against NFS startup and fail silently or hang. + +### Solution — fstab with x-systemd.after + +The loopback is established via `/etc/fstab` using the `x-systemd.after` option to explicitly declare the dependency on `nfs-server.service`: + +``` +localhost:/ /data/nfs/znas nfs4 defaults,_netdev,x-systemd.after=nfs-server.service 0 0 +``` + +`x-systemd.after=nfs-server.service` causes systemd-fstab-generator to automatically create a mount unit (`data-nfs-znas.mount`) with `After=nfs-server.service` in its `[Unit]` block. This guarantees the full dependency chain: + +``` +zfs-import.target + → zfs-mount.service + → nfs-server.service (via nfs-server override.conf) + → data-nfs-znas.mount (via x-systemd.after in fstab) + → remote-fs.target + → Docker Swarm containers +``` + +The generated unit (created automatically at runtime by systemd-fstab-generator — not a file on disk): + +```ini +# /run/systemd/generator/data-nfs-znas.mount +[Unit] +Documentation=man:fstab(5) man:systemd-fstab-generator(8) +SourcePath=/etc/fstab +After=nfs-server.service +Before=remote-fs.target + +[Mount] +What=localhost:/ +Where=/data/nfs/znas +Type=nfs4 +Options=defaults,_netdev,x-systemd.after=nfs-server.service +``` + +**Do not create a hand-written systemd mount unit for this.** systemd-fstab-generator handles it automatically from the fstab entry. A manual unit would conflict. + +### Verify Loopback is Active + ```bash -# Show available exports +mount | grep data/nfs/znas +# Should show: localhost:/ on /data/nfs/znas type nfs4 (...) + +systemctl status data-nfs-znas.mount +# Should show: active (mounted) +``` + +--- + +## Client Configuration + +All non-Swarm clients are Linux systems using autofs. + +### Autofs Configuration + +`/etc/auto.master` (relevant entry): + +``` +/data/nfs /etc/auto.nfs +``` + +`/etc/auto.nfs`: + +``` +znas -fstype=nfs4 192.168.5.10:/ +``` + +This mounts the full NFSv4 tree from ZNAS at `/data/nfs/znas` on demand — the same path used by the loopback mount on ZNAS itself. All swarm nodes (including ZNAS) access NAS storage via `/data/nfs/znas/`. + +**Note:** Autofs must be enabled on clients and disabled on the NFS server. Running autofs on the server creates recursive mount loops. + +### Adding a New Client + +```bash +# Install autofs if not present +sudo apt install autofs + +# Add to /etc/auto.master if not already present +echo "/data/nfs /etc/auto.nfs" | sudo tee -a /etc/auto.master + +# Create or update /etc/auto.nfs +echo "znas -fstype=nfs4 192.168.5.10:/" | sudo tee -a /etc/auto.nfs + +# Reload autofs +sudo systemctl reload autofs + +# Trigger mount by accessing the path +ls /data/nfs/znas/ +``` + +### Manual Mount (testing only) + +```bash +# Verify exports are visible from client showmount -e 192.168.5.10 -# Mount NFS share +# Test manual mount +sudo mkdir -p /mnt/znas sudo mount -t nfs4 192.168.5.10:/ /mnt/znas -# Verify content -ls -la /mnt/znas/Data/media/books/ +# Verify tree is accessible +ls /mnt/znas/Data/media/books/ + +# Unmount after testing +sudo umount /mnt/znas ``` +--- + ## Adding New Datasets -When creating new datasets that need to be exported via NFS: +When creating a new ZFS dataset that needs to be NFS-accessible: + ```bash -# Create dataset with correct mountpoint from the start +# Create with the correct mountpoint from the start sudo zfs create -o mountpoint=/export/Data/new_folder vault/Data/new_folder - -# The dataset will automatically mount and be visible via NFS -# due to the crossmnt and nohide options on the parent export - -# Verify it's visible -ls -la /export/Data/new_folder/ ``` -**No need to**: -- Modify `/etc/exports` (unless you need special permissions) -- Create bind mounts in `/etc/fstab` -- Restart NFS (it will see the new mount automatically) +The dataset will be automatically visible via NFS due to `crossmnt` and `nohide` on the parent — no changes to `/etc/exports` needed unless the new dataset requires different access controls. + +If different permissions are required, add an explicit entry to `/etc/exports` and reload: + +```bash +sudo exportfs -ra +``` + +--- + +## Current Export List + +Verified via `showmount -e 127.0.0.1`: + +``` +/export/photos * +/export/Green * +/export/Docker * +/export/Data/media/comics * +/export/Data/media/books * +/export/Data * +/export/Common * +/export * +``` + +--- + +## Known Gotchas + +**Loopback mount races NFS at boot** — This was the hardest problem to solve. A plain `_netdev` fstab entry only guarantees the network interface is up, not that the NFS server is ready to accept connections. The loopback mount would attempt before NFS finished starting and fail silently or hang. The fix is `x-systemd.after=nfs-server.service` in the fstab options, which causes systemd-fstab-generator to emit an `After=nfs-server.service` dependency in the generated mount unit. The full required boot chain is: `zfs-import.target` → `zfs-mount.service` → `nfs-server.service` → `data-nfs-znas.mount`. Each link must be explicit. + +**Do not hand-write a systemd mount unit for the loopback** — systemd-fstab-generator creates `data-nfs-znas.mount` automatically from the fstab entry at runtime (in `/run/systemd/generator/`, not `/etc/systemd/system/`). Creating a manual unit in `/etc/systemd/system/` will conflict with the generated one. + +**Autofs must be disabled on the server** — Running autofs on ZNAS itself creates a recursive mount loop. Autofs belongs on clients only. If autofs is accidentally re-enabled on ZNAS it will fight with the fstab loopback mount. + +**NFSv4 pseudo-root is required** — The `/export` entry with `fsid=0` is mandatory for NFSv4 clients. Without it clients cannot enumerate the export tree. Do not remove it even though it looks redundant. + +**`nohide` on sub-datasets** — `vault/Data/media_books` and `vault/Data/media_comics` are separate ZFS datasets mounted beneath the `vault/Data` export path. NFS does not cross filesystem boundaries by default. Without `nohide` clients see empty directories at those paths even though the data is present. + +**Do not use bind mounts for ZFS datasets** — Configure ZFS mountpoints directly to `/export/*`. Bind mounts in fstab for ZFS datasets cause ordering problems and are unnecessary. + +**Always set mountpoints when creating new datasets** — If a dataset is created without an explicit mountpoint it will inherit the parent's path and may not be visible or exportable correctly. Set `mountpoint=` at creation time. + +--- ## Troubleshooting ### Datasets not visible via NFS + ```bash # Verify dataset is mounted zfs list | grep dataset_name @@ -173,71 +290,104 @@ zfs list | grep dataset_name # Check NFS can read it sudo -u nobody ls -la /export/path/to/dataset/ -# Restart NFS +# Reload exports sudo exportfs -ra sudo systemctl restart nfs-server ``` ### Client shows empty directories + ```bash -# On client, clear NFS cache +# Clear NFS cache and remount sudo umount -f /mnt/znas sudo mount -t nfs4 192.168.5.10:/ /mnt/znas -# Or mount with no caching to test +# Test without caching to isolate the problem sudo mount -t nfs4 -o noac,lookupcache=none 192.168.5.10:/ /mnt/znas ``` ### After reboot, exports are empty + ```bash -# Verify ZFS mounted before NFS started +# Confirm ZFS mounted before NFS started systemctl status zfs-mount.service systemctl status nfs-server.service -# Check the override is in place -systemctl cat nfs-server.service | grep -A5 Unit +# Confirm override is in place +systemctl cat nfs-server.service | grep -A5 "\[Unit\]" ``` -## Important Notes +### Loopback mount not working for Swarm containers -- **Do not use bind mounts** in `/etc/fstab` for ZFS datasets - let ZFS handle mounting directly -- **Keep autofs disabled** on the NFS server to prevent recursive mount loops -- **Child datasets** must have their mountpoints explicitly set to be visible -- **The `crossmnt` and `nohide` options** are critical for NFSv4 to traverse ZFS dataset boundaries -- **Always set mountpoints when creating datasets** to avoid having to fix them later +```bash +# Check mount unit status +systemctl status data-nfs-znas.mount + +# Verify full dependency chain is satisfied +systemctl status zfs-mount.service +systemctl status nfs-server.service +systemctl status data-nfs-znas.mount + +# Verify loopback is mounted +mount | grep data/nfs/znas + +# If missing, mount manually to test +sudo mount -t nfs4 127.0.0.1:/ /data/nfs/znas + +# Check container can see the path +docker run --rm -v /data/nfs/znas/Data:/data alpine ls /data +``` + +If the unit fails at boot, confirm the fstab entry includes `x-systemd.after=nfs-server.service` — without this the mount races against NFS startup and loses. A plain `_netdev` entry is not sufficient. + +--- ## Configuration Files Reference -### /etc/fstab +### /etc/exports -Should **NOT** contain bind mounts for `/export/*`. ZFS handles mounting directly. - -Only keep system mounts: -```bash -# / was on /dev/nvme0n1p2 during curtin installation -/dev/disk/by-uuid/40c60952-0340-4a78-81f9-5b2193da26c6 / btrfs defaults 0 1 -# /boot was on /dev/nvme0n1p3 during curtin installation -/dev/disk/by-uuid/4abb4efa-0b2b-4e4a-bcaf-78227db4628f /boot ext4 defaults 0 1 -/dev/disk/by-uuid/d07437a0-3d0e-417a-a88e-438c603c2237 none swap sw 0 0 -# /srv was on /dev/nvme0n1p5 during curtin installation -/dev/disk/by-uuid/c66e81ff-436e-4d6f-980b-6f4875ea7c8e /srv btrfs defaults 0 1 +``` +/export *(ro,fsid=0,no_root_squash,no_subtree_check,crossmnt) +/export/Common *(fsid=4,rw,no_subtree_check,insecure) +/export/Data *(fsid=5,rw,no_subtree_check,insecure,crossmnt) +/export/Data/media/books *(fsid=51,rw,no_subtree_check,insecure,nohide) +/export/Data/media/comics *(fsid=52,rw,no_subtree_check,insecure,nohide) +/export/Docker *(fsid=29,rw,no_root_squash,sync,no_subtree_check,insecure) +/export/Green *(fsid=30,rw,no_root_squash,no_subtree_check,insecure) +/export/photos *(fsid=31,rw,no_root_squash,no_subtree_check,insecure) ``` ### /etc/systemd/system/nfs-server.service.d/override.conf + ```ini [Unit] After=zfs-import.target zfs-mount.service local-fs.target Requires=zfs-import.target zfs-mount.service ``` -### /etc/exports -```bash -# NFSv4 - pseudo filesystem root -/export *(ro,fsid=0,crossmnt,no_subtree_check) +### /etc/fstab (ZNAS system mounts only) -# Shares beneath the NFSv4 root -/export/Common *(fsid=4,rw,no_subtree_check,insecure) -/export/Data *(fsid=5,rw,no_subtree_check,insecure,crossmnt,nohide) -/export/Docker *(fsid=29,rw,no_root_squash,sync,no_subtree_check,insecure) -/export/Green *(fsid=30,rw,no_root_squash,no_subtree_check,insecure) -``` \ No newline at end of file +ZFS datasets are not listed here — ZFS handles its own mounting. Only system partitions appear: + +``` +# / - btrfs on nvme0n1p2 +/dev/disk/by-uuid/40c60952-0340-4a78-81f9-5b2193da26c6 / btrfs defaults 0 1 +# /boot - ext4 on nvme0n1p3 +/dev/disk/by-uuid/4abb4efa-0b2b-4e4a-bcaf-78227db4628f /boot ext4 defaults 0 1 +# swap +/dev/disk/by-uuid/d07437a0-3d0e-417a-a88e-438c603c2237 none swap sw 0 0 +# /srv - btrfs on nvme0n1p5 +/dev/disk/by-uuid/c66e81ff-436e-4d6f-980b-6f4875ea7c8e /srv btrfs defaults 0 1 +``` + +--- + +## Command Reference + +- Show active exports: `sudo exportfs -v` +- Reload exports: `sudo exportfs -ra` +- Show available exports (from any host): `showmount -e 192.168.5.10` +- Restart NFS: `sudo systemctl restart nfs-server` +- Check NFS status: `systemctl status nfs-server` +- Verify ZFS mounts: `mount | grep export` +- Verify loopback: `mount | grep data/nfs`