How to Create a Pi-Hole Management Pack in Aria Operations

Two years ago, I documented the integration of Pi-Hole with VMware Aria Operations using the Aria Operations Management Pack Builder (MPB) and Pi-Hole’s API. Since then, both tools have undergone significant updates: MPB has a revamped 2.0 interface, and Pi-Hole 6 introduced a completely new API with formal documentation and session-based authentication. In this updated guide, I’ll walk you through building a Pi-Hole Management Pack using these modern tools, highlighting key changes and providing a detailed, hands-on tutorial for tech enthusiasts and system admins.

What’s New with the Pi-Hole API?

Formal Documentation and Accessibility

With the release of Pi-Hole 6, the API has been overhauled to be more robust and user-friendly. Official documentation is now available at http://pi.hole/api/docs, which resolves to your local Pi-Hole server to ensure version compatibility. If you haven’t installed Pi-Hole yet, you can explore the documentation for all branches online, such as the Pi-Hole API documentation (master branch). Additionally, Pi-Hole provides a built-in UI at https://YourPi-holeServerName/api/docs/ where you can test API requests directly—a fantastic feature for developers.

Session ID Authentication: A Shift from Tokens

One major change in Pi-Hole 6 is the move from token-based to session ID-based authentication. I discovered this shift when my existing Management Pack stopped collecting data after upgrading to version 6. The API authentication documentation outlines several authentication options, I opted for the App Password method. Here’s how to set it up:

  1. In the Pi-Hole Admin UI, navigate to the Web Interface / API section of Settings, make sure the interface is toggled from Basic to Expert and click Configure App Password.
  2. The UI will display a one-time App Password—copy it immediately, as it won’t be shown again. Store it securely (e.g., in a password manager).

To verify the new authentication method, I used the API UI (https://YourPi-holeServerName/api/docs/) to test a request with my App Password. After clicking Try, I received a session.sid, confirming successful authentication.

I then tested the /stats/summary endpoint (used in my original article) with the session ID, ensuring it returned the expected data. I now had a basic enough understanding of the API to apply it to building a Management Pack.

Introducing Aria Operations Management Pack Builder 2.0

Released in October 2024, version 2.0 of the Aria Operations Management Pack Builder brings a refreshed UI and improved functionality. While MPs from version 1 can be imported, the Pi-Hole API changes require a fresh build for this example. I’ll assume you’ve already deployed the MPB appliance (refer to the official MPB deployment guide) and are ready to start.

Step 1: Configure the Connection in MPB

First, connect MPB to your Aria Operations server:

  1. Access the MPB Web UI.
  2. Navigate to the VMware Aria Operations Connection tab.
  3. Create and test a connection to your Aria Operations server.

With the connection established, you’re ready to build your Management Pack.

Step 2: Define the Source

The Source defines the environment (your Pi-Hole server) that MPB will interact with. Here’s how to set it up:

  1. In the Designs tab, click Create.
  2. Name your Management Pack (e.g., “Pi-Hole MP”) and add a description, then click Save.
  3. Configure the Source:
    • Enter the Hostname, Port, SSL Configuration, and Base API Path of your Pi-Hole server.
    • Click Next.

Set Up Authentication

Pi-Hole 6 uses session-based authentication, so we’ll configure MPB accordingly:

  1. On the Authentication tab, change the dropdown from Basic to Custom.
  2. Create a label called “App Password” and paste the App Password you generated earlier into the value field.
  3. Check the Sensitive box to secure the password, then click Add Field. This creates a variable ${authentication.credentials.app_password} for use in the MP.
  4. Enable the Session Authentication toggle and click Next.

Configure the Session API Request

From the Pi-Hole API documentation, we know the authentication endpoint is a POST request to /auth. Configure it on the Get Session API Request tab as follows:

  1. Select POST from the dropdown HTTP Method list
  2. In the API Path field, enter auth.

We saw in the API documentation that we need to pass our App Password in the Body of our request in the JSON Format of:

{"password": "${authentication.credentials.app_password}"}

So on the Get Session Request Advanced tab we will use the variable created earlier to pass the App Password (note the use of double-quotes):

3. Click Request to test the configuration. If successful, you’ll see a session.sid in the response.

4. Select the session.sid field from the response body and copy the session ID variable for later use

The Pi-Hole API documentation lists four ways to use the session ID when making requests.

We’ll use the X-FTL-SID header option.

  1. In the Global Request Settings tab, add a header:
    • Header Name: X-FTL-SID
    • Header Value: ${session.sid}
      This ensures the session ID is included in all API requests.

In the Release Session Request tab:

  • Enable the toggle to request a release
  • Choose an HTTP Method of DELETE
  • Set the API Path to auth

We do not add any additional values on the “Release Session Request Advanced” tab. Click Next.

On the Make Release Session Request tab click the Request button to verify a successful session release.

To verify the setup, test the /stats/summary endpoint:

  1. On the Test Connection Request tab, set the API Path to stats/summary.
  2. Click Request and confirm that summary data (e.g., total queries, blocked queries) is returned.
  3. Click Save to complete the Source configuration.

Step 3: Create API Requests

With a Source section successfully tested against our reference environment, we move to the Requests section of the design. This is where we define the API requests to collect the data that our MP uses. Since the point is to just demonstrate a concept, we are going to create just two requests, one named “summary” and one named “hostname”.

From the previous blog we determined that the /stats/summary API endpoint will return most of the data that we would use in our MP. And we used it in our Source section to test our reference environment. If you had read the previous blog you will recall that there was no endpoint that would return the pi-hole server name. Thankfully that has been resolved with the /info/host endpoint so we will use that as our second request for this test.

Request 1: Summary Statistics

  1. In the Requests tab, click Add API Request.
  2. Skip chaining (click Next).
  3. Set the HTTP Method to GET, the API Path to stats/summary, and keep the default Request Name.
  4. No advanced options are needed—click Next.
  5. On the Preview screen, click Request to confirm data is returned (e.g., total queries, blocked queries).

Request 2: Hostname

  1. Repeat the steps above, setting the API Path to info/host.
  2. Test the request to ensure data (e.g., host.uname.nodename) is returned.

Step 4: Define Objects

In the Objects section, select the data from your API requests to include in the Management Pack:

  1. Click Add New Object.
  2. Set the Object Type to “Pi-Hole Server” and click Next.
  3. On the Attributes from the API Request tab:
    • Expand the stats/summary request and select key metrics like:
      • queries.total
      • queries.blocked
      • queries.percent_blocked
      • queries.cached
    • Expand the info/host request and select host.uname.nodename for the server name.
  4. On the Properties and Metrics tab, adjust labels and data types as needed (e.g., use “Count” for total queries, “%” for percent blocked). For more on KPIs, check VMware’s KPI documentation.
  5. Set the Object Instance Name and Object Identifiers to “Pi-Hole Server Name” for clarity.

Step 5: Configure and Build the Management Pack

With the design complete, move to the Configuration section to review settings like MP version and labels. Then:

  1. In the Build section, click Perform Collection to run a test collection and verify the results.
  2. Click Build to generate the .pak file.
  3. New in MPB 2.0: Deploy the Management Pack directly to your Aria Operations server by selecting it from the list and clicking Deploy.

Step 6: Configure the Management Pack in Aria Operations

Install the .pak file in Aria Operations as you would any other Management Pack:

  1. Navigate to Administration > Integrations > Repositories.
  2. Enable the new Management Pack
  3. Add an account for each Pi-Hole server, including the App Password as a credential in Aria Operations.

Within minutes, you’ll be tracking Pi-Hole performance metrics—like total queries and blocked queries—directly in Aria Operations.

Conclusion: Monitor Pi-Hole Like a Pro

By leveraging the updated Pi-Hole API and Aria Operations MPB 2.0, you can seamlessly integrate Pi-Hole metrics into your monitoring environment. This tutorial demonstrates a simple yet powerful Management Pack, but the possibilities are endless—what other applications would you like to monitor with Aria Operations?

I’d love to hear your thoughts! Drop a comment below with your experiences building Management Packs or any other apps you’d like to integrate with Aria Operations. If you found this guide helpful, share it with your network and stay tuned for more tech tutorials.

Posted in Uncategorized | Tagged , , , | Leave a comment

Reviving an Old ESXi Host: USB to Local Disk Migration

I have an older Intel NUC in my lab, and although its aging, it still serves a purpose, and I plan to hang on to it for a little while longer. This post will outline some issues I encountered while recently migrating from a USB boot device to a more permanent option. As described extensively in this knowledge base article: https://knowledge.broadcom.com/external/article/317631/sd-cardusb-boot-device-revised-guidance.html, using USB devices is no longer a recommended boot media due to the endurance of the media. In addition, this host recently started throwing an error message:

Lost connectivity to the device mpx.vmhba32:C0:T0:L0 backing the boot filesystem /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0. As a result, host configuration changes will not be saved to persistent storage.

This message appeared on the host object in vCenter Server. I decided this would be a good time to move the boot device to a more durable media. The host had a local disk, which contained a single VMFS volume where I stored a VM containing some backups. I moved this VM to a shared datastore for safe keeping and proceeded to delete the VMFS volume. I wanted to re-install ESXi and specify this device, and not having a VMFS volume makes me more confident when selecting the disk during the ESXi install.

Creating the boot media

For this ESXi host, I knew that I would need the latest ESXi base image (8.0u3d), the Synology NFS Plug-in for VAAI, and the USB NIC Fling Driver. Instead of just installing ESXi and then adding packages, or using New-ImageBundle, I decided to turn to vCenter Server Lifecycle Manager for help. I first created a new, empty cluster object. I then selected the updates tab and selected ‘manage with a single image’, and then ‘setup image manually’. I selected the required ESXi version and additional components, save, and finally ‘finish image setup’. Once complete, I was able to select the ‘…’ and ‘Export’ options, pictured below.

This allowed me to export the image as an ISO image, pictured below:

With the ISO image in hand, I used Rufus to write the ISO image to a USB drive to use as the installation media.

Installing ESXi

Since I only needed to install ESXi on a single host, I decided to do so manually / interactively. Knowing that this was an old host, and the installed CPU was no longer supported on the HCL, I pressed SHIFT+o (the letter o, not the number zero) during bootup to add a couple of boot options:

systemMediaSize=min allowLegacyCPU=true

The systemMediaSize option limits the amount of space used on the boot media to 32GB (min) instead of 128GB (default). This is described more here: https://knowledge.broadcom.com/external/article/345195/boot-option-to-configure-the-size-of-esx.html. The allowLegacyCPU option allows ESXi installs to continue on unsupported CPUs. This is documented various places, including here: https://williamlam.com/2022/10/quick-tip-automating-esxi-8-0-install-using-allowlegacycputrue.html.

The install went well, I was able to select my empty local disk to use as an installation target and the system booted up fine afterwards. I noticed I now had a datastore1 on this host which was 32GB smaller than the original VMFS volume.

Configuring USB NIC Fling Driver

My USB NIC was recognized immediately as well, since I had included the driver in the custom image. I added the host to a distributed virtual switch, and mapped uplinks to the appropriate physical NICs, but on reboot the vusb0 device was no longer in use by Uplink 2.

Some of my notes mentioned that I had previously added some lines to the /etc/rrc.local.d/local.sh script to handle this, although I didn’t list which commands. Thankfully I was able to get the system to boot from the failing USB device and review this file. I’ve included the code added below:

vusb0_status=$(esxcli network nic get -n vusb0 | grep 'Link Status' | awk '{print $NF}')
count=0
while [[ $count -lt 20 && "${vusb0_status}" != "Up" ]]
do
    sleep 10
    count=$(( $count + 1 ))
    vusb0_status=$(esxcli network nic get -n vusb0 | grep 'Link Status' | awk '{print $NF}')
done

esxcfg-vswitch -P vusb0 -V 308 30-Greenfield-DVS
/bin/vim-cmd internalsvc/refresh_network

The esxcfg-vswitch help states that the -P and -V options are used as follows:

 -V|--dvp=dvport             Specify a DVPort Id for the operation.
 -P|--add-dvp-uplink=uplink  Add an uplink to a DVPort on a DVSwitch.
                              Must specify DVPort Id.

The Physical uplink I wanted to add was vusb0 and the DV Port Id for the operation was 308, which could be found on the distributed switch > ports tab > when filtering he ‘connectee’ column for the specific host in question, pictured below:

Now on system reboot, the vusb0 uplink correctly connects to the expected distributed switch.

Lifecycle Manager – Host is not compatible with the image

Once I had the host networking situated, I wanted to verify that vCenter Lifecycle Manager agreed that my host was up to date with the latest image. I was surprised to see that the system said The CPU on the host is not supported by the image. Please refer to KB 82794 for more details.

I knew these CPUs were unsupported, but had expected a less significant The CPU on this host may not be supported in future ESXi releases, which is what I had observed prior to the host rebuild. After some searching, I found this thread: https://community.broadcom.com/vmware-cloud-foundation/discussion/syntax-for-an-upgrade-cmd-to-ignore-cpu-requirements, which proposed edits to the /bootbank/boot.cfg, specifically adding the allowLegacyCPU=true flag to be added to the end of the kernelopt= line. This resolved my issue and allows me to continue functioning with this older system.

Conclusion

This migration process highlights the challenges of maintaining older ESXi hosts while ensuring compatibility. Moving from USB-based boot devices to more durable storage is a critical step, especially as support is phased out for USB/SD boot devices. Leveraging vCenter Lifecycle Manager simplifies image management, though workarounds (such as allowLegacyCPU=true) may be needed for legacy hardware.

Posted in Lab Infrastructure, Virtualization | Leave a comment

vCenter Server 7 to 8 Upgrade: Avoiding Pitfalls with Skyline Health Diagnostics

I recently upgraded a vCenter Server instance in a lab from version 7 to 8. In this post I’ll walk through some issues encountered and lessons learned during the process.

Pre-Checks with Skyline Health Diagnostics

In Skyline Health Diagnostics, there is an analysis option for vCenter Upgrade Pre-Check Plugins. This can be found under New Analysis by selecting either product VMware vSphere or VMware vCenter Server. In the current version of Skyline Health Diagnostics (4.0.8 as of this post), this analysis will check a total of 32 possible issues that could impact a vCenter Server upgrade.

In my case, the scan identified a warning named ‘LookupServiceCheck.’ The recommendation was to use lsdoctor to remove some duplicate service registrations. The tools usage is described in this KB article: https://knowledge.broadcom.com/external/article/320837/using-the-lsdoctor-tool.html. The detailed output of Skyline Health Diagnostics also identified specifically which service registrations need to be validated.

After taking a snapshot of the VM, I followed the KB article to remove the duplicate registrations. I then re-ran the vCenter Upgrade Pre-Check Plugins scan, confirmed the issue was resolved, and verified that no new issues were found.

Upgrading vCenter Server

The upgrade from vCenter Server 7.0 to 8.0 was uneventful — which is exactly how I like to describe upgrades. A new vCenter Server Appliance was deployed, the configuration copied over, and logins to the new system, even using active directory users, worked without issue.

It’s difficult to know if the Skyline Health Diagnostics recommendation to run lsdoctor directly contributed to the smooth upgrade. However, the tool was easy to use, and I would definitely include it in my upgrade checklist for future upgrades.

Clearing the arp-cache

Looking in the inventory of the upgraded vCenter Server, I did notice that two test hosts were disconnected. While disconnected test hosts aren’t uncommon, I recalled both hosts being online prior to the upgrade. Interestingly I was able to ping these hosts from my jump box, but not from the vCenter Server appliance.

These hosts were connected to an older consumer grade-switch (TP Link T1600G-28TS) in my lab. I had previously encountered an issue where devices on this switch failed to recognize IP address changes, which I resolved by clearing the arp table. Given that the vCenter Server upgrade workflow involves assigning a temporary IP before reverting to the original, I suspected a similar issue.

I referenced my old notes, which instructed me to SSH into the switch and run a few commands. However, when I attempted to connect using: ssh admin@192.168.10.1, I was met with this error:

Unable to negotiate with 192.168.10.1 port 22: no matching key exchange method found. Their offer: diffie-hellman-group1-sha1

A quick search helped me overcome that error. However, I encountered a new error, and then another error, until I found a working connection string:

ssh -oKexAlgorithms=+diffie-hellman-group1-sha1 -oHostKeyAlgorithms=+ssh-dss -c aes256-cbc -m hmac-md5 admin@192.168.10.1

Once connected, I ran the following commands:

enable
configure
clear arp-cache

Immediately, the two disconnected hosts came back online. This was a reminder that aging network equipment can introduce unexpected issues. Newer operating systems and SSH implementations have stricter security requirements, and this switch hasn’t kept pace.

Conclusion

The best way to ensure a successful vCenter Server upgrade is preparation. Skyline Health Diagnostics provides an effective way to validate the environment before upgrading and offers guidance on any required remediation steps. Additionally, this experience reinforced the importance of keeping network hardware up to date—technical debt can surface at the most unexpected times, even during routine maintenance.

Posted in Lab Infrastructure, Virtualization | Leave a comment

Automated Deployment of Storage Server

In this post, I’ll walk through the process of automating the deployment of a storage server using Aria Automation. Specifically, I’ll show how to add an option to deploy a storage server dynamically in your lab environment and troubleshoot an iSCSI duplicate LUN ID issue.

In a recent post, I documented the creation of a storage server that could provide NFS or iSCSI storage for use in my lab. To make consuming this appliance easier, I wanted to add a check box to my Aria Automation request form to determine if a storage server was needed. If this box was checked, then a storage server would be included with the deployment.

To achieve this, I added an additional vSphere.Machine object to my Assember Template and connected it to the same network as the nested ESXi host. I then added a boolean input to the template. Finally, the ‘count’ property of the machine object is set using the following logic:

      count: ${input.deployStorageServer == true ? 1:0 }

This logic says that if the box is checked (boolean true) then the count is 1, otherwise it is 0. Here is a screenshot of the canvas after adding the machine:

I now have a toggle to deploy (or not) an external storage appliance when deploying nested ESXi hosts.

Duplicate Device ID issue

After creating a couple of test deployments and adding hosts to vCenter Server, I ran into an interesting issue. For deployment #1, I formatted LUN0 as VMFS. For deployment #2, I tried to format LUN0 but it didn’t appear in the GUI. Looking a bit more closely at the devices, I realized that the unique identifiers (naa.* value) of each LUN was the same. This makes sense as each appliance was cloned with devices presented and all. Next, I attempted to unmap and remap the iSCSI device from the appliance. Below are the commands I used.

sudo targetcli

# Remove the iscsi LUN mapping
cd /iscsi/iqn.2025-02.com.example.iscsi:target01/tpg1/luns
delete 0

# Remove the disk mapping & recreate
cd /backstores/fileio/
delete disk0
create disk0 /data/iscsi/disk0.img sparse=true write_back=false

# Recreate the iscsi LUN mapping
cd /iscsi/iqn.2025-02.com.example.iscsi:target01/tpg1/luns
create /backstores/fileio/disk0

After doing this and rescanning the host, I confirmed that the unique identifier (naa ID) changed and I was able to format LUN0 as VMFS.

Longer term, I decided that I should not have the iSCSI mappings pre-created in my storage appliance. I removed the configuration (clearconfig confirm=True) from the appliance, and instead placed the following script as: /data/iscsi/setup.sh:

targetcli backstores/fileio create disk0 /data/iscsi/disk0.img sparse=true write_back=false
targetcli backstores/fileio/disk0 set attribute is_nonrot=1
targetcli backstores/fileio create disk1 /data/iscsi/disk1.img sparse=true write_back=false
targetcli backstores/fileio/disk1 set attribute is_nonrot=1
targetcli backstores/fileio create disk2 /data/iscsi/disk2.img sparse=true write_back=false
targetcli backstores/fileio/disk2 set attribute is_nonrot=1

targetcli /iscsi create iqn.2025-02.com.example.iscsi:target01 
targetcli /iscsi/iqn.2025-02.com.example.iscsi:target01/tpg1/luns create /backstores/fileio/disk0
targetcli /iscsi/iqn.2025-02.com.example.iscsi:target01/tpg1/luns create /backstores/fileio/disk1
targetcli /iscsi/iqn.2025-02.com.example.iscsi:target01/tpg1/luns create /backstores/fileio/disk2

targetcli /iscsi/iqn.2025-02.com.example.iscsi:target01/tpg1/acls create iqn.1998-01.com.vmware:h316-vesx-64.lab.enterpriseadmins.org:394284478:65

targetcli saveconfig

Now, after deploying the storage appliance, assuming I need an iSCSI target I can run sudo /data/iscsi/setup.sh and have new identifiers generated at runtime. This eliminates the identifier duplication and gets the automation ~80% of the way there.

With the steps outlined above, I was able to automate the deployment of storage servers and resolve the issue with duplicate LUN IDs. This process saves time and ensures each deployment is consistent. Going forward, I’ll continue automating aspects of my lab environment to increase efficiency.

Posted in Lab Infrastructure, Virtualization | Leave a comment

How to Set Up a Minimal NFS and iSCSI Storage Solution Using Ubuntu 24.04

In my lab, I often need different types of storage to test various scenarios. For example, just last week someone asked about using New-Datastore with a specific version of VMFS and I needed to quickly perform a quick syntax check. I’ve found that having a nested storage appliance, like a Open Filer or Free NAS available is helpful. However, these appliances offer way more features than I need and typically have higher resource requirements. Sometimes, setting up specific storage protocols like NFS or iSCSI is crucial for testing and development, but existing solutions can be overly complex or resource-heavy for lab environments. In this post I’ll outline how I solved this problem with a few utilities added to an existing Ubuntu 24.04 template.

Storage Protocols

With this project I wanted to have a single storage target that could provide both NFS and iSCSI storage protocols. For purposes of my lab, nearly all of my storage testing the ‘client’ system will be ESXi. This post will provide some output/examples in that context. For example, ESXi supports block storage (such as iSCSI) and file storage, specifically NFS3 and NFS4.1. Ideally, I want to provide all three of these options with this single appliance, so we’ll show examples of using the appliance in all three of those ways.

Setting Up the Test Appliance

I deployed an Ubuntu 24.04 VM, using the image/customization spec described here: https://enterpriseadmins.org/blog/scripting/ubuntu-24-04-packer-and-vcenter-server-customization-specifications/

The template VM has a single 50GB disk, so I added an additional 50GB disk to use as the backing for the storage server. We’ll format this disk as btrfs and mount it as /data. This will be covered in the following code block:

sudo mkdir /data
sudo mkfs.btrfs /dev/sdb
echo "/dev/sdb /data btrfs defaults 0 0" | sudo tee -a /etc/fstab
sudo systemctl daemon-reload
sudo mount /data

The above code block is creating a folder, formatting our second disk in the system, adding an entry to the fstab file so the filesystem mounts when the system boots, and finally mounts our new disk. After the above is complete, running df -h /data should return the mounted disk and its size, just as a confirmation that everything worked successfully.

Configuring NFS on Ubuntu 24.04

I’ll start with NFS as this is a problem I’ve previously solved for using a Photon OS (https://enterpriseadmins.org/blog/virtualization/vmware-workstation-lab-photon-os-container-host-and-nfs-server/). The only difference this time is that I planned to use Ubuntu 24.04, which has a slightly different package name for the NFS Server components.

sudo apt install nfs-kernel-server -y
sudo mkdir /data/nfs
echo "/data/nfs *(rw,async,no_root_squash,insecure_locks,sec=sys,no_subtree_check)" | sudo tee -a /etc/exports
sudo systemctl daemon-reload
sudo systemctl reload nfs-server

The above code block installs our NFS server package, creates a subfolder to export as NFS, adds an entry to the NFS exports configuration file, then reloads the configuration to take effect. On our client system (ESXi), we can confirm that our work was successful by creating a datastore. I’ll complete that task in PowerCLI below:

New-Datastore -VMHost h316-vesx-64* -Name 'nfs326' -Nfs -NfsHost 192.168.10.26 -Path /data/nfs -Confirm:$false

Name                               FreeSpaceGB      CapacityGB
----                               -----------      ----------
nfs326                                  14.994          15.000

As we can see, the test was a success and returned our mount point size in the command results. The above example resulted in an NFS 3 mount of the NFS folder. I created a subfolder (test41) and executed a similar test to confirm this could work for NFS 4.1 as well.

New-Datastore -VMHost h316-vesx-64* -Name 'nfs326-41' -Nfs -FileSystemVersion 4.1 -NfsHost 192.168.10.26 -Path /data/nfs/test41 -Confirm:$false

Name                               FreeSpaceGB      CapacityGB
----                               -----------      ----------
nfs326-41                               14.994          15.000

As we can see in the vCenter web interface, one of these datastores is NFS 3 and the other is NFS 4.1, both showing the same capacity and free space.

This confirms that we were able to successfully connect to our NFS Server service using ESXi with both NFS 3 and NFS 4.1 connections. Next, we’ll look at setting up iSCSI storage, which requires a slightly different approach.

iSCSI

The iSCSI testing was a bit more interesting. Looking around, I found a couple of ways to create an iSCSI target. I ended up using targetcli to create the iSCSI target. There are plenty of tutorials around for this, including a video: https://www.youtube.com/watch?v=OIpxwX6pTIU and Ubuntu 24.04 specific article: https://www.server-world.info/en/note?os=Ubuntu_24.04&p=iscsi&f=1 that were very helpful. I’ll document the steps below for completeness.

In this first code block we’ll install the service and create the folder where we’ll storage some images.

sudo apt install targetcli-fb -y
sudo mkdir /data/iscsi

The targetcli command can create image files to use as backing for our disks, but in my testing the sparse=true switch does not create sparse files, so we’ll do this as two steps. You’ll note that I’m specifying one image has having a 2T file size… but as you may have noticed in our NFS example, we only have 15GB of disk for our /data mount. This doesn’t result in some ‘magic beans’ sort of free storage… once we write 15GB of stuff to this disk we’ll be out of capacity and run into problems. This is only being done for illustration/simulation purposes…. sometimes you’ll want the UI to show ‘normal’ sizes that you’d see with actual datastores. One reason that the /data mount was configured with btrfs instead of something like ext4 is so we can support image files larger than 16T. This btrfs filesystem will allow files over 62TB in size (62TB being the maximum supported size for VMFS 6). Also in the code block output below, we’ll use du to show that these disks are using 0 bytes on the file system, but have larger apparent sizes.

sudo truncate -s 10G /data/iscsi/disk0.img
sudo truncate -s 10G /data/iscsi/disk1.img
sudo truncate -s 2T /data/iscsi/disk2.img

du -h /data/iscsi/*.img
0       /data/iscsi/disk0.img
0       /data/iscsi/disk1.img
0       /data/iscsi/disk2.img

du -h /data/iscsi/*.img --apparent-size
10G     /data/iscsi/disk0.img
10G     /data/iscsi/disk1.img
2.0T    /data/iscsi/disk2.img

Once we have our files pre-staged, we can start working with targetcli.

sudo targetcli

# this should enter the targetcli shell

cd /backstores/fileio
create disk0 /data/iscsi/disk0.img sparse=true write_back=false
cd disk0
set attribute is_nonrot=1
cd ..
create disk1 /data/iscsi/disk1.img sparse=true write_back=false
cd disk1
set attribute is_nonrot=1
cd ..
create disk2 /data/iscsi/disk2.img sparse=true write_back=false
cd disk2
set attribute is_nonrot=1

The above code block creates the fileio references to each of our disks, and also set the is_nonrot flag to tell the system that these are non-rotational (ie Flash) devices.

Still in the targetcli shell, we’ll start our iSCSI configuration.

cd /iscsi
create iqn.2025-02.com.example.iscsi:target01 
cd iqn.2025-02.com.example.iscsi:target01/tpg1/luns
create /backstores/fileio/disk0
create /backstores/fileio/disk1
create /backstores/fileio/disk2

This will create LUNs for each of our disks. Finally, still in the targetcli shell we’ll create an ACL to allow a specific host to access the target. We’ll then delete it. This puts the correct syntax in our command history, so we can refer back to it in the future (I plan to use this as a template for future tests).

cd /iscsi/iqn.2025-02.com.example.iscsi:target01/tpg1/acls
create iqn.1998-01.com.vmware:host.lab.enterpriseadmins.org:3:65
delete iqn.1998-01.com.vmware:host.lab.enterpriseadmins.org:3:65
exit

The exit will cause targetcli to save our changes so they’ll persist a reboot. For testing, we’ll go back into targetcli and add a specific reference to allow our test host to the iSCSI target.

sudo targetcli
create iqn.1998-01.com.vmware:h316-vesx-64.lab.enterpriseadmins.org:394284478:65
exit

On our test client system, we can add a dynamic target for the IP/name & port 3260 of our storage appliance and then rescan for storage. We should see the three disks that we created, with the sizes specified.

As another confirmation, we may want to make each of these disks a VMFS volume. We can do that using syntax similar to the below code block:

Get-ScsiLun -VmHost h316* |?{$_.Vendor -eq 'LIO-ORG'} | %{
  New-Datastore -Host h316* -Name "ds-vmfs-$($_.model)" -Path $_.CanonicalName -Vmfs -FileSystemVersion 6 
}

Name                               FreeSpaceGB      CapacityGB
----                               -----------      ----------
ds-vmfs-disk1                            8.345           9.750
ds-vmfs-disk2                        2,046.312       2,047.750
ds-vmfs-disk0                            8.345           9.750

Looking in the vCenter web interface, we can see that all our storage has been presented.

Once we’ve placed filesystems on these disks, we can go back to the shell and see how much space is being used on disk.

du -h /data/iscsi/*.img
29M     /data/iscsi/disk0.img
29M     /data/iscsi/disk1.img
62M     /data/iscsi/disk2.img

We can see that the creation of a filesystem on these disks does consume some of the blocks (we are using several MB of disk, instead of the previous 0 bytes).

Adding extra LUNs to the iSCSI target is a straightforward process, requiring just a handful of commands. An example can be found in the code block below:

sudo truncate -s 10T /data/iscsi/disk3.img
sudo targetcli
# this should enter the targetcli shell

cd /backstores/fileio
create disk3 /data/iscsi/disk3.img sparse=true write_back=false

cd /iscsi/iqn.2025-02.com.example.iscsi:target01/tpg1/luns
create /backstores/fileio/disk3

exit

The above code block shows the creation of a 10TB disk image, entering the targetcli shell, adding the newly created disk as a ‘fileio’ option, and mapping that disk to our iSCSI target. Finally we exit, which by default will save the configuration and make it persistent. Refreshing storage on ESXi should cause the new LUN to appear. Since we didn’t set the is_nonrot attribute, this device will appear as an HDD instead of a Flash device.

Growing the /data btrfs filesystem

Our filesystem is currently backed by a 15GB disk. We’ve allocated about 12TB of that, so it is grossly oversubscribed. For a production system this would be a terrible idea, but for our lab/illustration purposes this is probably fine. At some point we may need to extend this filesystem to accommodate some growth. I’ve grown ext3 and ext4 filesystems before but wanted to document how to do the extension for the btrfs filesystem used in this example. I choose btrfs because it supports larger files, allowing us to create images as large as supported by ESXi (62TB). The following code block will show how to extend this filesystem in the guest OS. This assumes we have already increased the size of the disk in the vCenter web client, for illustration purposes we’ve extended the disk from 15GB to 20GB.

df -h
# shows that the filesystem did not autogrow

echo 1 | sudo tee /sys/class/block/sdb/device/rescan
# rescans for disk changes

sudo lsblk  
# confirms disk is now seen as 20gb

sudo btrfs device usage /data
# shows that device size is 20gb

sudo btrfs filesystem resize max /data
# results in:
# Resize device id 1 (/dev/sdb) from 15.00GiB to max

df -h /data
# confirm filesystem is now 15gb

The above commands rescanned our disk to be aware of the new size, then resized the filesystem to the size we defined in the hypervisor (20GB).

To confirm this works as expected, we can refresh storage information for one of our NFS mounts. The capacity should increase from 15GB to 20GB, as seen in the following screenshot.

Conclusion

Creating this storage server to support NFS 3, NFS 4.1, and iSCSI targets is relatively straightforward. Having this pre-configured storage appliance can greatly streamline the process of testing various storage protocols, especially in virtual environments where quick deployment is key.

Posted in Lab Infrastructure, Virtualization | 1 Comment