{"id":2196,"date":"2025-02-16T21:14:02","date_gmt":"2025-02-17T02:14:02","guid":{"rendered":"https:\/\/enterpriseadmins.org\/blog\/?p=2196"},"modified":"2025-02-16T21:14:02","modified_gmt":"2025-02-17T02:14:02","slug":"how-to-set-up-a-minimal-nfs-and-iscsi-storage-solution-using-ubuntu-24-04","status":"publish","type":"post","link":"https:\/\/enterpriseadmins.org\/blog\/virtualization\/how-to-set-up-a-minimal-nfs-and-iscsi-storage-solution-using-ubuntu-24-04\/","title":{"rendered":"How to Set Up a Minimal NFS and iSCSI Storage Solution Using Ubuntu 24.04"},"content":{"rendered":"\n

In my lab, I often need different types of storage to test various scenarios. For example, just last week someone asked about using New-Datastore<\/code> with a specific version of VMFS and I needed to quickly perform a quick syntax check. I’ve found that having a nested storage appliance, like a Open Filer or Free NAS available is helpful. However, these appliances offer way more features than I need and typically have higher resource requirements. Sometimes, setting up specific storage protocols like NFS or iSCSI is crucial for testing and development, but existing solutions can be overly complex or resource-heavy for lab environments. In this post I’ll outline how I solved this problem with a few utilities added to an existing Ubuntu 24.04 template.<\/p>\n\n\n\n

Storage Protocols<\/h2>\n\n\n\n

With this project I wanted to have a single storage target that could provide both NFS and iSCSI storage protocols. For purposes of my lab, nearly all of my storage testing the ‘client’ system will be ESXi. This post will provide some output\/examples in that context. For example, ESXi supports block storage (such as iSCSI) and file storage, specifically NFS3 and NFS4.1. Ideally, I want to provide all three of these options with this single appliance, so we’ll show examples of using the appliance in all three of those ways.<\/p>\n\n\n\n

Setting Up the Test Appliance<\/h2>\n\n\n\n

I deployed an Ubuntu 24.04 VM, using the image\/customization spec described here: https:\/\/enterpriseadmins.org\/blog\/scripting\/ubuntu-24-04-packer-and-vcenter-server-customization-specifications\/<\/a><\/p>\n\n\n\n

The template VM has a single 50GB disk, so I added an additional 50GB disk to use as the backing for the storage server. We’ll format this disk as btrfs<\/code> and mount it as \/data<\/code>. This will be covered in the following code block:<\/p>\n\n\n\n

sudo mkdir \/data\nsudo mkfs.btrfs \/dev\/sdb\necho \"\/dev\/sdb \/data btrfs defaults 0 0\" | sudo tee -a \/etc\/fstab\nsudo systemctl daemon-reload\nsudo mount \/data<\/code><\/pre>\n\n\n\n
The above code block is creating a folder, formatting our second disk in the system, adding an entry to the fstab<\/code> file so the filesystem mounts when the system boots, and finally mounts our new disk.  After the above is complete, running df -h \/data<\/code> should return the mounted disk and its size, just as a confirmation that everything worked successfully.<\/p>\n\n\n\n
Configuring NFS on Ubuntu 24.04<\/h2>\n\n\n\n
I’ll start with NFS as this is a problem I’ve previously solved for using a Photon OS (https:\/\/enterpriseadmins.org\/blog\/virtualization\/vmware-workstation-lab-photon-os-container-host-and-nfs-server\/<\/a>).  The only difference this time is that I planned to use Ubuntu 24.04, which has a slightly different package name for the NFS Server components.<\/p>\n\n\n\n
sudo apt install nfs-kernel-server -y\nsudo mkdir \/data\/nfs\necho \"\/data\/nfs *(rw,async,no_root_squash,insecure_locks,sec=sys,no_subtree_check)\" | sudo tee -a \/etc\/exports\nsudo systemctl daemon-reload\nsudo systemctl reload nfs-server<\/code><\/pre>\n\n\n\n
The above code block installs our NFS server package, creates a subfolder to export as NFS, adds an entry to the NFS exports configuration file, then reloads the configuration to take effect.  On our client system (ESXi), we can confirm that our work was successful by creating a datastore.  I’ll complete that task in PowerCLI below:<\/p>\n\n\n\n
New-Datastore -VMHost h316-vesx-64* -Name 'nfs326' -Nfs -NfsHost 192.168.10.26 -Path \/data\/nfs -Confirm:$false\n\nName                               FreeSpaceGB      CapacityGB\n----                               -----------      ----------\nnfs326                                  14.994          15.000<\/code><\/pre>\n\n\n\n
As we can see, the test was a success and returned our mount point size in the command results.  The above example resulted in an NFS 3 mount of the NFS folder.  I created a subfolder (test41<\/code>) and executed a similar test to confirm this could work for NFS 4.1 as well.<\/p>\n\n\n\n
New-Datastore -VMHost h316-vesx-64* -Name 'nfs326-41' -Nfs -FileSystemVersion 4.1 -NfsHost 192.168.10.26 -Path \/data\/nfs\/test41 -Confirm:$false\n\nName                               FreeSpaceGB      CapacityGB\n----                               -----------      ----------\nnfs326-41                               14.994          15.000<\/code><\/pre>\n\n\n\n
As we can see in the vCenter web interface, one of these datastores is NFS 3 and the other is NFS 4.1, both showing the same capacity and free space.<\/p>\n\n\n\n
\"\"<\/a><\/figure>\n\n\n\n
This confirms that we were able to successfully connect to our NFS Server service using ESXi with both NFS 3 and NFS 4.1 connections.  Next, we’ll look at setting up iSCSI storage, which requires a slightly different approach.<\/p>\n\n\n\n
iSCSI<\/h2>\n\n\n\n
The iSCSI testing was a bit more interesting.  Looking around, I found a couple of ways to create an iSCSI target.  I ended up using targetcli<\/code> to create the iSCSI target.  There are plenty of tutorials around for this, including a video: https:\/\/www.youtube.com\/watch?v=OIpxwX6pTIU<\/a> and Ubuntu 24.04 specific article: https:\/\/www.server-world.info\/en\/note?os=Ubuntu_24.04&p=iscsi&f=1<\/a> that were very helpful.  I’ll document the steps below for completeness.  <\/p>\n\n\n\n
In this first code block we’ll install the service and create the folder where we’ll storage some images.<\/p>\n\n\n\n
sudo apt install targetcli-fb -y\nsudo mkdir \/data\/iscsi<\/code><\/pre>\n\n\n\n
The targetcli command can create image files to use as backing for our disks, but in my testing the sparse=true<\/code> switch does not create sparse files, so we’ll do this as two steps.  You’ll note that I’m specifying one image has having a 2T file size… but as you may have noticed in our NFS example, we only have 15GB of disk for our \/data<\/code> mount.  This doesn’t result in some ‘magic beans’ sort of free storage… once we write 15GB of stuff to this disk we’ll be out of capacity and run into problems.  This is only being done for illustration\/simulation purposes…. sometimes you’ll want the UI to show ‘normal’ sizes that you’d see with actual datastores.  One reason that the \/data<\/code> mount was configured with btrfs<\/code> instead of something like ext4<\/code> is so we can support image files larger than 16T.  This btrfs<\/code> filesystem will allow files over 62TB in size (62TB being the maximum supported size for VMFS 6).  Also in the code block output below, we’ll use du<\/code> to show that these disks are using 0 bytes on the file system, but have larger apparent sizes.<\/p>\n\n\n\n
sudo truncate -s 10G \/data\/iscsi\/disk0.img\nsudo truncate -s 10G \/data\/iscsi\/disk1.img\nsudo truncate -s 2T \/data\/iscsi\/disk2.img\n\ndu -h \/data\/iscsi\/*.img\n0       \/data\/iscsi\/disk0.img\n0       \/data\/iscsi\/disk1.img\n0       \/data\/iscsi\/disk2.img\n\ndu -h \/data\/iscsi\/*.img --apparent-size\n10G     \/data\/iscsi\/disk0.img\n10G     \/data\/iscsi\/disk1.img\n2.0T    \/data\/iscsi\/disk2.img<\/code><\/pre>\n\n\n\n
Once we have our files pre-staged, we can start working with targetcli<\/code>.<\/p>\n\n\n\n
sudo targetcli\n\n# this should enter the targetcli shell\n\ncd \/backstores\/fileio\ncreate disk0 \/data\/iscsi\/disk0.img sparse=true write_back=false\ncd disk0\nset attribute is_nonrot=1\ncd ..\ncreate disk1 \/data\/iscsi\/disk1.img sparse=true write_back=false\ncd disk1\nset attribute is_nonrot=1\ncd ..\ncreate disk2 \/data\/iscsi\/disk2.img sparse=true write_back=false\ncd disk2\nset attribute is_nonrot=1<\/code><\/pre>\n\n\n\n
The above code block creates the fileio references to each of our disks, and also set the is_nonrot<\/code> flag to tell the system that these are non-rotational (ie Flash) devices.<\/p>\n\n\n\n
Still in the targetcli<\/code> shell, we’ll start our iSCSI configuration.<\/p>\n\n\n\n
cd \/iscsi\ncreate iqn.2025-02.com.example.iscsi:target01 \ncd iqn.2025-02.com.example.iscsi:target01\/tpg1\/luns\ncreate \/backstores\/fileio\/disk0\ncreate \/backstores\/fileio\/disk1\ncreate \/backstores\/fileio\/disk2<\/code><\/pre>\n\n\n\n
This will create LUNs for each of our disks.  Finally, still in the targetcli<\/code> shell we’ll create an ACL to allow a specific host to access the target.  We’ll then delete it.  This puts the correct syntax in our command history, so we can refer back to it in the future (I plan to use this as a template for future tests).  <\/p>\n\n\n\n
cd \/iscsi\/iqn.2025-02.com.example.iscsi:target01\/tpg1\/acls\ncreate iqn.1998-01.com.vmware:host.lab.enterpriseadmins.org:3:65\ndelete iqn.1998-01.com.vmware:host.lab.enterpriseadmins.org:3:65\nexit<\/code><\/pre>\n\n\n\n
The exit will cause targetcli<\/code> to save our changes so they’ll persist a reboot.  For testing, we’ll go back into targetcli<\/code> and add a specific reference to allow our test host to the iSCSI target.<\/p>\n\n\n\n
sudo targetcli\ncreate iqn.1998-01.com.vmware:h316-vesx-64.lab.enterpriseadmins.org:394284478:65\nexit<\/code><\/pre>\n\n\n\n
On our test client system, we can add a dynamic target for the IP\/name & port 3260 of our storage appliance and then rescan for storage.  We should see the three disks that we created, with the sizes specified.<\/p>\n\n\n\n
\"\"<\/a><\/figure>\n\n\n\n
As another confirmation, we may want to make each of these disks a VMFS volume.  We can do that using syntax similar to the below code block:<\/p>\n\n\n\n
Get-ScsiLun -VmHost h316* |?{$_.Vendor -eq 'LIO-ORG'} | %{\n  New-Datastore -Host h316* -Name \"ds-vmfs-$($_.model)\" -Path $_.CanonicalName -Vmfs -FileSystemVersion 6 \n}\n\nName                               FreeSpaceGB      CapacityGB\n----                               -----------      ----------\nds-vmfs-disk1                            8.345           9.750\nds-vmfs-disk2                        2,046.312       2,047.750\nds-vmfs-disk0                            8.345           9.750<\/code><\/pre>\n\n\n\n
Looking in the vCenter web interface, we can see that all our storage has been presented.<\/p>\n\n\n\n
\"\"<\/a><\/figure>\n\n\n\n
Once we’ve placed filesystems on these disks, we can go back to the shell and see how much space is being used on disk.<\/p>\n\n\n\n
du -h \/data\/iscsi\/*.img\n29M     \/data\/iscsi\/disk0.img\n29M     \/data\/iscsi\/disk1.img\n62M     \/data\/iscsi\/disk2.img<\/code><\/pre>\n\n\n\n
We can see that the creation of a filesystem on these disks does consume some of the blocks (we are using several MB of disk, instead of the previous 0 bytes).  <\/p>\n\n\n\n
Adding extra LUNs to the iSCSI target is a straightforward process, requiring just a handful of commands.  An example can be found in the code block below:<\/p>\n\n\n\n
sudo truncate -s 10T \/data\/iscsi\/disk3.img\nsudo targetcli\n# this should enter the targetcli shell\n\ncd \/backstores\/fileio\ncreate disk3 \/data\/iscsi\/disk3.img sparse=true write_back=false\n\ncd \/iscsi\/iqn.2025-02.com.example.iscsi:target01\/tpg1\/luns\ncreate \/backstores\/fileio\/disk3\n\nexit<\/code><\/pre>\n\n\n\n
The above code block shows the creation of a 10TB disk image, entering the targetcli<\/code> shell, adding the newly created disk as a ‘fileio’ option, and mapping that disk to our iSCSI target.  Finally we exit, which by default will save the configuration and make it persistent.  Refreshing storage on ESXi should cause the new LUN to appear.  Since we didn’t set the is_nonrot<\/code> attribute, this device will appear as an HDD instead of a Flash device.<\/p>\n\n\n\n
Growing the \/data<\/code> btrfs filesystem<\/h2>\n\n\n\n
Our filesystem is currently backed by a 15GB disk.  We’ve allocated about 12TB of that, so it is grossly oversubscribed.  For a production system this would be a terrible idea, but for our lab\/illustration purposes this is probably fine.  At some point we may need to extend this filesystem to accommodate some growth.   I’ve grown ext3 and ext4 filesystems before but wanted to document how to do the extension for the btrfs<\/code> filesystem used in this example.  I choose btrfs<\/code> because it supports larger files, allowing us to create images as large as supported by ESXi (62TB).   The following code block will show how to extend this filesystem in the guest OS.  This assumes we have already increased the size of the disk in the vCenter web client, for illustration purposes we’ve extended the disk from 15GB to 20GB.<\/p>\n\n\n\n
df -h\n# shows that the filesystem did not autogrow\n\necho 1 | sudo tee \/sys\/class\/block\/sdb\/device\/rescan\n# rescans for disk changes\n\nsudo lsblk  \n# confirms disk is now seen as 20gb\n\nsudo btrfs device usage \/data\n# shows that device size is 20gb\n\nsudo btrfs filesystem resize max \/data\n# results in:\n# Resize device id 1 (\/dev\/sdb) from 15.00GiB to max\n\ndf -h \/data\n# confirm filesystem is now 15gb<\/code><\/pre>\n\n\n\n
The above commands rescanned our disk to be aware of the new size, then resized the filesystem to the size we defined in the hypervisor (20GB).<\/p>\n\n\n\n
To confirm this works as expected, we can refresh storage information for one of our NFS mounts.  The capacity should increase from 15GB to 20GB, as seen in the following screenshot.<\/p>\n\n\n\n
\"\"<\/a><\/figure>\n\n\n\n
Conclusion<\/h2>\n\n\n\n
Creating this storage server to support NFS 3, NFS 4.1, and iSCSI targets is relatively straightforward.  Having this pre-configured storage appliance can greatly streamline the process of testing various storage protocols, especially in virtual environments where quick deployment is key.<\/p>\n","protected":false},"excerpt":{"rendered":"
In my lab, I often need different types of storage to test various scenarios. For example, just last week someone asked about using New-Datastore with a specific version of VMFS and I needed to quickly perform a quick syntax check. … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":6,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[9,4],"tags":[],"class_list":["post-2196","post","type-post","status-publish","format-standard","hentry","category-lab-infrastructure","category-virtualization"],"_links":{"self":[{"href":"https:\/\/enterpriseadmins.org\/blog\/wp-json\/wp\/v2\/posts\/2196","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/enterpriseadmins.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/enterpriseadmins.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/enterpriseadmins.org\/blog\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/enterpriseadmins.org\/blog\/wp-json\/wp\/v2\/comments?post=2196"}],"version-history":[{"count":8,"href":"https:\/\/enterpriseadmins.org\/blog\/wp-json\/wp\/v2\/posts\/2196\/revisions"}],"predecessor-version":[{"id":2209,"href":"https:\/\/enterpriseadmins.org\/blog\/wp-json\/wp\/v2\/posts\/2196\/revisions\/2209"}],"wp:attachment":[{"href":"https:\/\/enterpriseadmins.org\/blog\/wp-json\/wp\/v2\/media?parent=2196"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/enterpriseadmins.org\/blog\/wp-json\/wp\/v2\/categories?post=2196"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/enterpriseadmins.org\/blog\/wp-json\/wp\/v2\/tags?post=2196"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}