Setting Up TrueNAS on Proxmox VE with ZFS on a Split NVMe SSD

This post will guide you through splitting a 1TB NVMe SSD into two partitions, setting up ZFS on these partitions, and using one to host a TrueNAS VM on Proxmox VE. This setup efficiently utilizes high-speed NVMe storage, ensuring robust data integrity and performance.

Prerequisites

• A Proxmox VE installation
• A 1TB NVMe SSD installed in your server
• Basic knowledge of Linux command line and ZFS

Note

• In this guide, we used ZFS storage. You can use ext4 or xfs.

Step 1: Partitioning the NVMe SSD

First, identify your NVMe device and partition it into two equal parts using fdisk or parted. Here, we assume the device is identified as /dev/nvme0n1.

sudo fdisk /dev/nvme0n1

Step 2: Create First Partition

1. Type n to create a new partition.
2. Select p the primary partition.
3. Enter 1 when prompted for the partition number.
4. Press Enter to accept the default first sector.
5. Type +512G Set the partition size to 512GB, or press Enter to use exactly half of the disk if it's a 1TB drive.

Step 3: Create Second Partition

1. Type n to create another new partition.
2. Select p for primary.
3. Enter 2 for the partition number.
4. Press Enter to accept the default first sector.
5. Press Enter again to use the remaining disk space for the second partition.

Step 4: Write Changes and Exit

• After creating the partitions, type w to write the changes to the disk and exit fdisk.

Note:

• Partition 1: 500GB for the TrueNAS VM
• Partition 2: 500GB for additional storage or other VMs

Step 5: Creating ZFS Pools

Once the partitions are ready, create ZFS pools on these new partitions. This step involves wiping and initializing each partition with ZFS:

sudo wipefs -a /dev/nvme0n1p1
sudo wipefs -a /dev/nvme0n1p2
sudo zpool create zfs_swd /dev/nvme0n1p1
sudo zpool create zfs_share /dev/nvme0n1p2

Here, zfs_swd is used for general VM storage and zfs_share is intended for shared storage or specific uses like NAS.

Step 6: Verify and Configure ZFS

After creating the pools, use the zfs list and zpool status Commands to ensure everything is configured correctly:

zfs list
zpool status

Step 7: Add ZFS as Storage in Proxmox

To use the ZFS pools in Proxmox for VM storage, you need to add them to Proxmox’s storage configuration. Here’s how to do that:

1. Open Proxmox Web Interface: Log in to your Proxmox VE web interface.
2. Go to Datacenter: Click on “Datacenter” in the top left.
3. Select the Proxmox Node: Click on the node where you want to add storage.
4. Go to the Storage Tab: Click “Storage” in the right pane.
5. Click ‘Add’ -> ‘ZFS’: This will open a configuration dialog.

• Pool: Select zfs_swd or zfs_share from the dropdown, depending on which one you want to configure first.
• Content: Select the types of data this storage will hold, such as Virtual Disks, containers, and so on.

Step 8: Installing and Configuring TrueNAS

• Create a TrueNAS VM: Deploy TrueNAS on Proxmox using the first ZFS pool (zfs_swd). Allocate the necessary resources (CPU, RAM) and use a ZFS volume for the VM’s disk.
• Install TrueNAS: Proceed with the standard installation of TrueNAS within the VM.

Step 9: Attaching Additional Storage to TrueNAS

After TrueNAS is set up, the next step is to attach additional storage from the zfs_share pool:

• Create a ZFS volume (zvol) within zfs_share For NAS purposes:

sudo zfs create -V 400G zfs_share/nas_storage

Step 10: Configuring Persistent Storage Attachments

Due to potential device path changes after reboots, use persistent identifiers for the ZFS volume to ensure stability:

• Find the UUID of the ZFS volume:
  (It would be something like /dev/zd192

ls -l /dev/zvol/zfs_share
blkid | grep /dev/zd192

• Update the TrueNAS VM configuration to use a persistent identifier (PARTUUID or UUID): Edit the VM’s configuration file to replace the device path with its UUID or PARTUUID.

qm set 123 -scsi1 /dev/disk/by-partuuid/your-partuuid-here

• You can also edit the VM configuration file and add the UUID path.

nano /etc/pve/qemu-server/123.conf
scsi1: /dev/disk/by-partuuid/your-partuuid-here

Step 11: Managing and Monitoring

• Start the TrueNAS VM and verify the configuration. Ensure that the attached ZFS volume is recognized by TrueNAS and properly mounted.
• Configure network file-sharing services in TrueNAS (e.g., NFS, SMB) to utilize the attached ZFS volume for storage needs.

Conclusion

This setup maximizes the capabilities of your NVMe SSD by utilizing ZFS for both performance and reliability and integrates TrueNAS for robust data storage solutions. Through this approach, you create a scalable, high-performance storage solution suitable for a variety of applications.