traveler/Netgrimoire
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs: update Netgrimoire/Storage/ZNAS_NFS_Exports

Browse source
This commit is contained in:
Administrator 2026-02-23 21:58:20 +00:00 • committed by John Smith
parent 77ed0db95d
commit 40724dad35
1 changed files with 287 additions and 137 deletions
Download patch file Download diff file Expand all files Collapse all files

424
Netgrimoire/Storage/ZNAS_NFS_Exports.md
View file

@ -2,170 +2,287 @@
title: ZFS-NFS-Exports title: ZFS-NFS-Exports
description: Exporting NFS shares from ZFS datasets description: Exporting NFS shares from ZFS datasets
published: true published: true
date: 2026-02-01T20:45:40.210Z date: 2026-02-23T21:58:11.949Z
tags: tags:
editor: markdown editor: markdown
dateCreated: 2026-02-01T20:45:40.210Z dateCreated: 2026-02-01T20:45:40.210Z
--- ---
# NFS + ZFS Configuration Fix # NFS Configuration
## Problems Identified ## Overview
1. **Boot order issue**: NFS was starting before ZFS mounted the datasets ZNAS exports storage via NFSv4. All exports are ZFS datasets mounted directly to `/export/*` — no bind mounts. NFS is configured to wait for ZFS at boot via a systemd override.
2. **Autofs recursive loop**: The server was NFS-mounting its own exports back to itself via autofs, creating conflicts
3. **Mountpoint mismatch**: ZFS datasets were mounting to `/srv/vault/*` but NFS was trying to export from `/export/*` using bind mounts that didn't work properly
4. **Child dataset mountpoints**: When the parent dataset mountpoint changed, child datasets still had old mountpoints and weren't visible
## Complete Solution ZNAS also mounts its own NFS exports back to itself at `/data/nfs/znas`. This is intentional: Docker Swarm containers scheduled to ZNAS need to access NAS storage at the same paths as containers running on other swarm members. The loopback mount provides a consistent NFS-backed path regardless of which node a container lands on.
### Step 1: Stop and Disable Autofs on NFS Server All other clients are Linux systems using autofs.
```bash
sudo systemctl stop autofs ---
sudo systemctl disable autofs
## Server Configuration
### ZFS Mountpoints
ZFS datasets mount directly to `/export/*`. No bind mounts are used.
```
vault → /export
vault/Common → /export/Common
vault/Data → /export/Data
vault/Data/media_books → /export/Data/media/books
vault/Data/media_comics → /export/Data/media/comics
vault/Docker → /export/Docker
vault/Green → /export/Green
vault/Green/Pocket → /export/Green/Pocket
vault/Photos → /export/Photos
``` ```
**Note**: Autofs should only run on NFS clients, not on the server itself. Verify at any time:
### Step 2: Set ZFS Datasets to Mount Directly to `/export/*`
Instead of using bind mounts, configure ZFS to mount directly to the export paths:
```bash ```bash
sudo zfs set mountpoint=/export vault mount | grep export
sudo zfs set mountpoint=/export/Data vault/Data
sudo zfs set mountpoint=/export/Green vault/Green
sudo zfs set mountpoint=/export/Docker vault/docker
sudo zfs set mountpoint=/export/Common vault/Common
``` ```
### Step 3: Configure Child Dataset Mountpoints ### /etc/exports
For any child datasets (subdirectories under parent datasets), set their mountpoints explicitly:
```bash
sudo zfs set mountpoint=/export/Data/media/books vault/Data/media_books
sudo zfs set mountpoint=/export/Data/media/comics vault/Data/media_comics
``` ```
### Step 4: Unmount and Remount Datasets
Unmount children first, then parents, then remount all:
```bash
# Unmount child datasets first
sudo zfs unmount vault/Data/media_books
sudo zfs unmount vault/Data/media_comics
# Unmount parent datasets
sudo zfs unmount vault/Data
sudo zfs unmount vault/Green
sudo zfs unmount vault/docker
sudo zfs unmount vault/Common
# Remount all
sudo zfs mount -a
```
### Step 5: Configure NFS Exports
Edit `/etc/exports`:
```bash
sudo nano /etc/exports
```
Content:
```bash
# /etc/exports: the access control list for filesystems which may be exported
# to NFS clients. See exports(5).
# NFSv4 - pseudo filesystem root # NFSv4 - pseudo filesystem root
/export *(ro,fsid=0,crossmnt,no_subtree_check) /export *(ro,fsid=0,no_root_squash,no_subtree_check,crossmnt)
# Shares beneath the NFSv4 root # Shares beneath the NFSv4 root
/export/Common *(fsid=4,rw,no_subtree_check,insecure) /export/Common *(fsid=4,rw,no_subtree_check,insecure)
/export/Data *(fsid=5,rw,no_subtree_check,insecure,crossmnt,nohide) /export/Data *(fsid=5,rw,no_subtree_check,insecure,crossmnt)
/export/Data/media/books *(fsid=51,rw,no_subtree_check,insecure,nohide)
/export/Data/media/comics *(fsid=52,rw,no_subtree_check,insecure,nohide)
/export/Docker *(fsid=29,rw,no_root_squash,sync,no_subtree_check,insecure) /export/Docker *(fsid=29,rw,no_root_squash,sync,no_subtree_check,insecure)
/export/Green *(fsid=30,rw,no_root_squash,no_subtree_check,insecure) /export/Green *(fsid=30,rw,no_root_squash,no_subtree_check,insecure)
/export/photos *(fsid=31,rw,no_root_squash,no_subtree_check,insecure)
``` ```
**Key options explained**: **Key options:**
- `crossmnt`: Allows NFS to cross filesystem mount boundaries
- `nohide`: Makes child mounts visible through parent exports
- `no_subtree_check`: Improves reliability and performance
### Step 6: Configure NFS to Wait for ZFS at Boot - `fsid=0` on `/export` — required for NFSv4 pseudo-root. Clients enumerate all exports from here.
- `crossmnt` — allows NFS to cross ZFS dataset boundaries when traversing the tree.
- `nohide` — required on `media/books` and `media/comics` because they are separate ZFS datasets mounted beneath the `vault/Data` export path. Without it clients see empty directories.
- `no_root_squash` — Docker and Green exports allow root writes. Required for container volume mounts.
- `insecure` — permits connections from unprivileged ports (>1024). Required for some Linux NFS clients and all macOS clients.
- `sync` on Docker — forces synchronous writes for container volume safety.
Create systemd override for NFS server: ### systemd Boot Order Override
```bash
sudo mkdir -p /etc/systemd/system/nfs-server.service.d/ NFS is configured to wait for ZFS to fully mount before starting.
sudo nano /etc/systemd/system/nfs-server.service.d/override.conf
``` `/etc/systemd/system/nfs-server.service.d/override.conf`:
Add this content:
```ini ```ini
[Unit] [Unit]
After=zfs-import.target zfs-mount.service local-fs.target After=zfs-import.target zfs-mount.service local-fs.target
Requires=zfs-import.target zfs-mount.service Requires=zfs-import.target zfs-mount.service
``` ```
This ensures NFS waits for ZFS to be fully mounted before starting. Apply after any changes:
### Step 7: Reload and Restart Services
```bash ```bash
sudo systemctl daemon-reload sudo systemctl daemon-reload
sudo exportfs -ra
sudo systemctl restart nfs-server sudo systemctl restart nfs-server
``` ```
### Step 8: Verify Configuration ### Autofs Disabled on Server
Autofs is disabled on ZNAS itself. It must only run on NFS clients. Running autofs on the server creates recursive mount loops.
On the server:
```bash ```bash
# Check ZFS mounts sudo systemctl stop autofs
zfs list -r vault sudo systemctl disable autofs
# Check what's actually mounted
mount | grep export
# Verify NFS exports
sudo exportfs -v
# Check content is visible
ls -la /export/Data/media/books/
``` ```
On the client: ---
## Loopback Mount (Docker Swarm)
ZNAS mounts its own NFS exports back to itself at `/data/nfs/znas`. This ensures containers scheduled to ZNAS by Docker Swarm access storage at the same NFS-backed paths as containers running on any other swarm member — consistent regardless of which node a service lands on.
Swarm container volume mounts reference paths under `/data/nfs/znas/` rather than `/export/` directly.
### The Timing Problem
Getting this mount to survive reboots reliably was non-trivial. The loopback has a chicken-and-egg dependency chain:
1. ZFS must import and mount pools before NFS server can export anything
2. NFS server must be fully started before the loopback mount can succeed
3. The loopback mount must be established before Docker Swarm containers start
A plain `_netdev` fstab entry is not sufficient — `_netdev` only guarantees the network is up, not that the NFS server is ready. The mount would race against NFS startup and fail silently or hang.
### Solution — fstab with x-systemd.after
The loopback is established via `/etc/fstab` using the `x-systemd.after` option to explicitly declare the dependency on `nfs-server.service`:
```
localhost:/ /data/nfs/znas nfs4 defaults,_netdev,x-systemd.after=nfs-server.service 0 0
```
`x-systemd.after=nfs-server.service` causes systemd-fstab-generator to automatically create a mount unit (`data-nfs-znas.mount`) with `After=nfs-server.service` in its `[Unit]` block. This guarantees the full dependency chain:
```
zfs-import.target
→ zfs-mount.service
→ nfs-server.service (via nfs-server override.conf)
→ data-nfs-znas.mount (via x-systemd.after in fstab)
→ remote-fs.target
→ Docker Swarm containers
```
The generated unit (created automatically at runtime by systemd-fstab-generator — not a file on disk):
```ini
# /run/systemd/generator/data-nfs-znas.mount
[Unit]
Documentation=man:fstab(5) man:systemd-fstab-generator(8)
SourcePath=/etc/fstab
After=nfs-server.service
Before=remote-fs.target
[Mount]
What=localhost:/
Where=/data/nfs/znas
Type=nfs4
Options=defaults,_netdev,x-systemd.after=nfs-server.service
```
**Do not create a hand-written systemd mount unit for this.** systemd-fstab-generator handles it automatically from the fstab entry. A manual unit would conflict.
### Verify Loopback is Active
```bash ```bash
# Show available exports mount | grep data/nfs/znas
# Should show: localhost:/ on /data/nfs/znas type nfs4 (...)
systemctl status data-nfs-znas.mount
# Should show: active (mounted)
```
---
## Client Configuration
All non-Swarm clients are Linux systems using autofs.
### Autofs Configuration
`/etc/auto.master` (relevant entry):
```
/data/nfs /etc/auto.nfs
```
`/etc/auto.nfs`:
```
znas -fstype=nfs4 192.168.5.10:/
```
This mounts the full NFSv4 tree from ZNAS at `/data/nfs/znas` on demand — the same path used by the loopback mount on ZNAS itself. All swarm nodes (including ZNAS) access NAS storage via `/data/nfs/znas/`.
**Note:** Autofs must be enabled on clients and disabled on the NFS server. Running autofs on the server creates recursive mount loops.
### Adding a New Client
```bash
# Install autofs if not present
sudo apt install autofs
# Add to /etc/auto.master if not already present
echo "/data/nfs /etc/auto.nfs" | sudo tee -a /etc/auto.master
# Create or update /etc/auto.nfs
echo "znas -fstype=nfs4 192.168.5.10:/" | sudo tee -a /etc/auto.nfs
# Reload autofs
sudo systemctl reload autofs
# Trigger mount by accessing the path
ls /data/nfs/znas/
```
### Manual Mount (testing only)
```bash
# Verify exports are visible from client
showmount -e 192.168.5.10 showmount -e 192.168.5.10
# Mount NFS share # Test manual mount
sudo mkdir -p /mnt/znas
sudo mount -t nfs4 192.168.5.10:/ /mnt/znas sudo mount -t nfs4 192.168.5.10:/ /mnt/znas
# Verify content # Verify tree is accessible
ls -la /mnt/znas/Data/media/books/ ls /mnt/znas/Data/media/books/
# Unmount after testing
sudo umount /mnt/znas
``` ```
---
## Adding New Datasets ## Adding New Datasets
When creating new datasets that need to be exported via NFS: When creating a new ZFS dataset that needs to be NFS-accessible:
```bash ```bash
# Create dataset with correct mountpoint from the start # Create with the correct mountpoint from the start
sudo zfs create -o mountpoint=/export/Data/new_folder vault/Data/new_folder sudo zfs create -o mountpoint=/export/Data/new_folder vault/Data/new_folder
# The dataset will automatically mount and be visible via NFS
# due to the crossmnt and nohide options on the parent export
# Verify it's visible
ls -la /export/Data/new_folder/
``` ```
**No need to**: The dataset will be automatically visible via NFS due to `crossmnt` and `nohide` on the parent — no changes to `/etc/exports` needed unless the new dataset requires different access controls.
- Modify `/etc/exports` (unless you need special permissions)
- Create bind mounts in `/etc/fstab` If different permissions are required, add an explicit entry to `/etc/exports` and reload:
- Restart NFS (it will see the new mount automatically)
```bash
sudo exportfs -ra
```
---
## Current Export List
Verified via `showmount -e 127.0.0.1`:
```
/export/photos *
/export/Green *
/export/Docker *
/export/Data/media/comics *
/export/Data/media/books *
/export/Data *
/export/Common *
/export *
```
---
## Known Gotchas
**Loopback mount races NFS at boot** — This was the hardest problem to solve. A plain `_netdev` fstab entry only guarantees the network interface is up, not that the NFS server is ready to accept connections. The loopback mount would attempt before NFS finished starting and fail silently or hang. The fix is `x-systemd.after=nfs-server.service` in the fstab options, which causes systemd-fstab-generator to emit an `After=nfs-server.service` dependency in the generated mount unit. The full required boot chain is: `zfs-import.target` → `zfs-mount.service` → `nfs-server.service` → `data-nfs-znas.mount`. Each link must be explicit.
**Do not hand-write a systemd mount unit for the loopback** — systemd-fstab-generator creates `data-nfs-znas.mount` automatically from the fstab entry at runtime (in `/run/systemd/generator/`, not `/etc/systemd/system/`). Creating a manual unit in `/etc/systemd/system/` will conflict with the generated one.
**Autofs must be disabled on the server** — Running autofs on ZNAS itself creates a recursive mount loop. Autofs belongs on clients only. If autofs is accidentally re-enabled on ZNAS it will fight with the fstab loopback mount.
**NFSv4 pseudo-root is required** — The `/export` entry with `fsid=0` is mandatory for NFSv4 clients. Without it clients cannot enumerate the export tree. Do not remove it even though it looks redundant.
**`nohide` on sub-datasets** — `vault/Data/media_books` and `vault/Data/media_comics` are separate ZFS datasets mounted beneath the `vault/Data` export path. NFS does not cross filesystem boundaries by default. Without `nohide` clients see empty directories at those paths even though the data is present.
**Do not use bind mounts for ZFS datasets** — Configure ZFS mountpoints directly to `/export/*`. Bind mounts in fstab for ZFS datasets cause ordering problems and are unnecessary.
**Always set mountpoints when creating new datasets** — If a dataset is created without an explicit mountpoint it will inherit the parent's path and may not be visible or exportable correctly. Set `mountpoint=` at creation time.
---
## Troubleshooting ## Troubleshooting
### Datasets not visible via NFS ### Datasets not visible via NFS
```bash ```bash
# Verify dataset is mounted # Verify dataset is mounted
zfs list | grep dataset_name zfs list | grep dataset_name
@ -173,71 +290,104 @@ zfs list | grep dataset_name
# Check NFS can read it # Check NFS can read it
sudo -u nobody ls -la /export/path/to/dataset/ sudo -u nobody ls -la /export/path/to/dataset/
# Restart NFS # Reload exports
sudo exportfs -ra sudo exportfs -ra
sudo systemctl restart nfs-server sudo systemctl restart nfs-server
``` ```
### Client shows empty directories ### Client shows empty directories
```bash ```bash
# On client, clear NFS cache # Clear NFS cache and remount
sudo umount -f /mnt/znas sudo umount -f /mnt/znas
sudo mount -t nfs4 192.168.5.10:/ /mnt/znas sudo mount -t nfs4 192.168.5.10:/ /mnt/znas
# Or mount with no caching to test # Test without caching to isolate the problem
sudo mount -t nfs4 -o noac,lookupcache=none 192.168.5.10:/ /mnt/znas sudo mount -t nfs4 -o noac,lookupcache=none 192.168.5.10:/ /mnt/znas
``` ```
### After reboot, exports are empty ### After reboot, exports are empty
```bash ```bash
# Verify ZFS mounted before NFS started # Confirm ZFS mounted before NFS started
systemctl status zfs-mount.service systemctl status zfs-mount.service
systemctl status nfs-server.service systemctl status nfs-server.service
# Check the override is in place # Confirm override is in place
systemctl cat nfs-server.service | grep -A5 Unit systemctl cat nfs-server.service | grep -A5 "\[Unit\]"
``` ```
## Important Notes ### Loopback mount not working for Swarm containers
- **Do not use bind mounts** in `/etc/fstab` for ZFS datasets - let ZFS handle mounting directly ```bash
- **Keep autofs disabled** on the NFS server to prevent recursive mount loops # Check mount unit status
- **Child datasets** must have their mountpoints explicitly set to be visible systemctl status data-nfs-znas.mount
- **The `crossmnt` and `nohide` options** are critical for NFSv4 to traverse ZFS dataset boundaries
- **Always set mountpoints when creating datasets** to avoid having to fix them later # Verify full dependency chain is satisfied
systemctl status zfs-mount.service
systemctl status nfs-server.service
systemctl status data-nfs-znas.mount
# Verify loopback is mounted
mount | grep data/nfs/znas
# If missing, mount manually to test
sudo mount -t nfs4 127.0.0.1:/ /data/nfs/znas
# Check container can see the path
docker run --rm -v /data/nfs/znas/Data:/data alpine ls /data
```
If the unit fails at boot, confirm the fstab entry includes `x-systemd.after=nfs-server.service` — without this the mount races against NFS startup and loses. A plain `_netdev` entry is not sufficient.
---
## Configuration Files Reference ## Configuration Files Reference
### /etc/fstab ### /etc/exports
Should **NOT** contain bind mounts for `/export/*`. ZFS handles mounting directly. ```
/export *(ro,fsid=0,no_root_squash,no_subtree_check,crossmnt)
Only keep system mounts: /export/Common *(fsid=4,rw,no_subtree_check,insecure)
```bash /export/Data *(fsid=5,rw,no_subtree_check,insecure,crossmnt)
# / was on /dev/nvme0n1p2 during curtin installation /export/Data/media/books *(fsid=51,rw,no_subtree_check,insecure,nohide)
/dev/disk/by-uuid/40c60952-0340-4a78-81f9-5b2193da26c6 / btrfs defaults 0 1 /export/Data/media/comics *(fsid=52,rw,no_subtree_check,insecure,nohide)
# /boot was on /dev/nvme0n1p3 during curtin installation /export/Docker *(fsid=29,rw,no_root_squash,sync,no_subtree_check,insecure)
/dev/disk/by-uuid/4abb4efa-0b2b-4e4a-bcaf-78227db4628f /boot ext4 defaults 0 1 /export/Green *(fsid=30,rw,no_root_squash,no_subtree_check,insecure)
/dev/disk/by-uuid/d07437a0-3d0e-417a-a88e-438c603c2237 none swap sw 0 0 /export/photos *(fsid=31,rw,no_root_squash,no_subtree_check,insecure)
# /srv was on /dev/nvme0n1p5 during curtin installation
/dev/disk/by-uuid/c66e81ff-436e-4d6f-980b-6f4875ea7c8e /srv btrfs defaults 0 1
``` ```
### /etc/systemd/system/nfs-server.service.d/override.conf ### /etc/systemd/system/nfs-server.service.d/override.conf
```ini ```ini
[Unit] [Unit]
After=zfs-import.target zfs-mount.service local-fs.target After=zfs-import.target zfs-mount.service local-fs.target
Requires=zfs-import.target zfs-mount.service Requires=zfs-import.target zfs-mount.service
``` ```
### /etc/exports ### /etc/fstab (ZNAS system mounts only)
```bash
# NFSv4 - pseudo filesystem root
/export *(ro,fsid=0,crossmnt,no_subtree_check)
# Shares beneath the NFSv4 root ZFS datasets are not listed here — ZFS handles its own mounting. Only system partitions appear:
/export/Common *(fsid=4,rw,no_subtree_check,insecure)
/export/Data *(fsid=5,rw,no_subtree_check,insecure,crossmnt,nohide) ```
/export/Docker *(fsid=29,rw,no_root_squash,sync,no_subtree_check,insecure) # / - btrfs on nvme0n1p2
/export/Green *(fsid=30,rw,no_root_squash,no_subtree_check,insecure) /dev/disk/by-uuid/40c60952-0340-4a78-81f9-5b2193da26c6 / btrfs defaults 0 1
``` # /boot - ext4 on nvme0n1p3
/dev/disk/by-uuid/4abb4efa-0b2b-4e4a-bcaf-78227db4628f /boot ext4 defaults 0 1
# swap
/dev/disk/by-uuid/d07437a0-3d0e-417a-a88e-438c603c2237 none swap sw 0 0
# /srv - btrfs on nvme0n1p5
/dev/disk/by-uuid/c66e81ff-436e-4d6f-980b-6f4875ea7c8e /srv btrfs defaults 0 1
```
---
## Command Reference
- Show active exports: `sudo exportfs -v`
- Reload exports: `sudo exportfs -ra`
- Show available exports (from any host): `showmount -e 192.168.5.10`
- Restart NFS: `sudo systemctl restart nfs-server`
- Check NFS status: `systemctl status nfs-server`
- Verify ZFS mounts: `mount | grep export`
- Verify loopback: `mount | grep data/nfs`