Enterprise Admins.org

In a previous post (https://enterpriseadmins.org/blog/virtualization/nested-vmware-workstation-lab-using-linked-clones/) I mentioned a nested ‘smash lab’ using VMware Workstation. This post will focus on a Photon OS VM with docker and nfs-server services enabled.

Occasionally there is a service that I may temporarily need in my lab, such as an SMTP server or web server, and those are readily available as docker container images. Therefore I decided to have a virtual machine available pre-configured for running docker images. After setting this up, I realized that it would also be convenient to have access to an NFS server that could be used as a shared datastore if needed. There were a couple of container images available for NFS server, but I decided to run NFS directly on this VM instead.

I started by downloading Photon OS 4.0 Rev2 from https://packages.vmware.com/photon/4.0/Rev2/ova/photon-ova_uefi-4.0-c001795b80.ova with virtual hardware v13 (UEFI Secure Boot). There were a few other versions available, but this was the latest available at the time.

Double clicking the OVA downlad starts the VMware Workstation import wizard, and asks for a name for the virtual machine. I went with lab-dock-14 since this was going to be running docker and I planned to assign a static IP ending in .14. I adjusted the VM to use ‘VMnet10’ which is the isolated lab network and removed the floppy disk drive. I also decided to add a 50GB disk on SCSI 1:0 to use as a mount for the NFS server. Once the settings were how I wanted, I powered on the VM.

The initial login to the OVA uses root/changeme and forces a password change. I set a password then configured networking by manually typing in the following:

cat > /etc/systemd/network/10-static-en.network << "EOF"

[Match]
Name=eth0

[Network]
Address=172.16.10.14/24
Gateway=172.16.10.1
DNS=172.16.10.1
EOF

This defines my static IP address, points the default gateway and DNS to the lab side of the domain controller/NAT router. Now I just need to change permissions for the file and restart networking:

chmod 644 /etc/systemd/network/10-static-en.network
systemctl restart systemd-networkd

From this point, I can use ssh root@172.16.10.14 from my jump server to login to the VM and start using copy/paste to do the rest of my configuration. The first step was to make sure all the packages were up to date with tdnf update.

I then defined my hostname with the command hostnamectl set-hostname lab-dock-14.example.org and made sure this DNS A record was defined in my DNS server.

Docker is preinstalled on the Photon OS appliance, it just needs started & enabled at boot, so I did that with:

systemctl start docker
systemctl enable docker

I wanted to make a /data mount to use as an NFS server and potentially to store container configuration if needed. I confirmed which device I wanted to use by running fdisk -l to list disks and confirmed that /dev/sdb was the unused 50GB disk I added when configuring the VM. I ran mkfs -t ext3 /dev/sdb to put a filesystem on the drive, created a directory with mkdir /data, opened a text editor with vi /etc/fstab and appended /dev/sdb /data ext3 defaults 0 0 to the end of the file. After saving/exiting that file I typed mount /data and confirmed that the new mount was available with df -h.

With the mount defined, I next needed to make it available via NFS. To do this, I first needed to install nfs-utils with tdnf install nfs-utils. I then opened a text editor with vi /etc/exports and added a single line to the file that says /data *(rw,async,no_root_squash,insecure_locks,sec=sys,no_subtree_check). This says I wanted to make the /data mount available to all hosts over NFS. After saving the file I ran the following commands:

exportfs -ra # this should return nothing, assuming the /etc/exports file is correct
systemctl start nfs-server.service
systemctl enable nfs-server.service

After doing this I attempted to mount the NFS export as a datastore on my nested ESXi host but was unsuccessful. After more troubleshooting than I care to admit, I realized that the firewall was enabled on my Photon OS appliance. For my purposes, I don’t really need the firewall enabled, so I disabled it and set it not to start automatically with:

systemctl stop iptables
systemctl disable iptables

This solved the problem where I wasn’t able to mount the datastore. Depending on your environment it might be worthwhile to define the ports necessary for NFS, but as I was using NFS3 and that was many ports, this wholesale disabling of the firewall was quicker.

Once this was complete, I rebooted to ensure everything came up as expected (IP & hostname correct, Docker & NFS running, etc) and shutdown with shutdown -h now. Once the VM was powered off, I created a new snapshot so this could be used as a parent virtual machine for future linked clones.

In a previous post (https://enterpriseadmins.org/blog/virtualization/nested-vmware-workstation-lab-using-linked-clones/) I mentioned a nested ‘smash lab’ using VMware Workstation. This post will focus on a couple of the component VMs: lab-esxi-02 and lab-esxi-03, which are nested ESXi 7.0.3 and ESXi 8.0.0 hosts, each containing a corresponding vCenter Server Appliance.

These two nested ESXi hosts only differ in the version of ESXi that is installed. Each has 2 vCPU, 20GB RAM, a 16GB SCSI 0:0 device (for ESXi install) and a 100GB SCSI 1:0 device (for a VMFS datastore). I decided to manually install ESXi, so that I could choose specific builds. I installed 1 patch prior to the latest available, just in case I had a need to attempt an upgrade I wanted to ensure that an upgrade was available. Other than the default next > next > finish installs, I only made 2 changes to these hosts:

1. Configured networking from the DCUI. This involved setting the IP address to 172.16.10.2 or 172.16.10.3, where the last octet matches the host name, as well as setting the default gateway and DNS server IP to 172.16.10.1, which is the lab side of the domain controller.
2. Create a VMFS datastore named local-hdd that used the 100GB SCSI 1:0 device. I could have automated this, but since it was a super simple task I decided to just knock it out in the UI.

Once ESXi was installed, I added a deployed corresponding vCenter Server Appliances to the local datastore. For this I first created DNS records for the appliances with associated IP addresses, created a copy of the <cd-rom>\vcsa-cli-installer\templates\install\embedded_vCSA_on_ESXi.json file, specified values for hostname, datastore name, etc and then deployed through the command line with .\vcsa-deploy install C:\tmp\lab-vcsa-13.json --accept-eula --acknowledge-ceip --no-ssl-certificate-verification and waited until the process completed. I ran into two different challenges with this. First, running a nested 64-bit guest requires that “Virtualize Intel T-x/EPT or AMD-V/RVI” be selected on the processor configuration. Credential Guard was enabled on my system and had to be disabled to allow for the VCSA to start. The other challenge that I encountered was that by default ntp_servers is defined as time.nist.gov in the JSON configuration file. I didn’t change this, but deployed the VCSA when my laptop could not reach the internet. The VCSA startup failed and reviewing log files showed all time stamps with a 1970-01-01 timestamp. I remembered that NTP was set to an internet address, so I tried to deploy again after updating the JSON file to point to time.example.org, which was a CNAME configured to the DNS server lab-mgmt-01.example.org, and this worked without error.

Once the VCSA was running, I debated on whether or not I should create inventory (like a new Data Center object, containing a Cluster with my nested ESXi host, etc) but decided to leave the VCSA completely unconfigured. This will allow me to address the configuration each time the environment is deployed. In the past I’ve created a minimal config, so time will tell which route is best. If having a minimal configuration is more practical, I can address that issue and create a new snapshot if needed.

With my ESXi host & VCSA deployed, I powered down the VCSA and ESXi host. Once the Workstation ESXi VM was powered off, I created a new snapshot so this could be used as a parent virtual machine for future linked clones.

In a previous post (https://enterpriseadmins.org/blog/virtualization/nested-vmware-workstation-lab-using-linked-clones/) I mentioned a nested ‘smash lab’ using VMware Workstation. This post will focus on one of the component VMs: lab-mgmt-01, which functions as the management console/GUI, domain controller, DNS Server, Certificate Authority, and NAT gateway.

This VM is a Windows Server 2022 Standard virtual machine with a very generic install. A lot of services will end up running on this VM and its likely that it will be used by nearly every test. Therefore, I decided to try and document the configuration in PowerShell, in case I ever wanted to update it to a newer version of Windows via re-deploy.

This first section of code will rename the computer

# Set IP address for interface #2; see previous post for network diagram.
New-NetIPAddress -InterfaceAlias 'Ethernet1' -IPAddress 172.16.10.1 -PrefixLength 24 -Confirm:$false

# Set the computer name
Rename-Computer -NewName 'lab-mgmt-01' -Restart:$true

### We need a reboot after a name change before a dcpromo, this should happen automatically as part of above Rename-Computer setp ###

The second set of code will focus on installing the active directory components and promoting this system to a domain controller for a new forest named example.org. I like using example domain names as these are specifically reserved by RFC 2606 – Reserved Top Level DNS Names (ietf.org) and make documentation/screenshots look nice.

# Install AD and DNS roles
Install-WindowsFeature -name AD-Domain-Services,DNS -IncludeManagementTools

# Make me a new AD Forest
Import-Module ADDSDeployment
Install-ADDSForest `
-CreateDnsDelegation:$false `
-DatabasePath "C:\Windows\NTDS" `
-DomainMode "WinThreshold" `
-DomainName "example.org" `
-DomainNetbiosName "EXAMPLE" `
-ForestMode "WinThreshold" `
-InstallDns:$true `
-LogPath "C:\Windows\NTDS" `
-NoRebootOnCompletion:$false `
-SysvolPath "C:\Windows\SYSVOL" `
-Force:$true

After the system is promoted to a domain controller it will automatically reboot. When the system comes back up there are a few more services we need to install like the Certificate Authority and Routing components.

### PART 2, AFTER AD REBOOT ###
Install-WindowsFeature Routing,Adcs-Cert-Authority,Adcs-Web-Enrollment -IncludeManagementTools


# Configure RRAS
Install-RemoteAccess -VpnType RoutingOnly
$ExternalInterface="Ethernet0"
$InternalInterface="Ethernet1"

cmd.exe /c "netsh routing ip nat install"
cmd.exe /c "netsh routing ip nat add interface $ExternalInterface"
cmd.exe /c "netsh routing ip nat set interface $ExternalInterface mode=full"
cmd.exe /c "netsh routing ip nat add interface $InternalInterface"

# Configure Certificate Authority
Install-AdcsCertificationAuthority -CAType EnterpriseRootCA -CACommonName rootca.example.org -ValidityPeriod:Years -ValidityPeriodUnits 10 -Confirm:$false
Install-AdcsWebEnrollment -Confirm:$false

There were a few more steps that I completed manually.

1. In the Certification Authority console > right click the CA rootca.example.org > Security tab > select the Administrators group > check the box for Request Certificates. This will allow the default admin user to be able to request certificates.
2. Create a DNS record for time.example.org to be a CNAME back to the domain controller. This allows the domain controller to provide time to the ESXi hosts & VCSA and allows things to work as expected even when disconnected from the internet.
   Note: any DNS edits will need to happen in this parent VM to be available for other lab exercises. In addition to this time record that was initially created, it might be helpful to create extra records that might point at the container host, for services like SMTP.
3. Configured DNS to disable root hints and set forwarder to home network DNS Server (could have pointed to Google or CloudFlare).
4. Installed PowerCLI module and configured some common settings: Install-Module vmware.powercli; Set-PowerCLIConfiguration -InvalidCertificateAction:Prompt -ParticipateInCeip:$true -Scope:AllUsers
5. Ran VMware Horizon OS Optimization Tool to disable services like screensaver and Windows Update.
6. Configured autologin under VM Options > Autologin so that we automatically login as the domains Administrator account.

I rebooted the VM a couple of times to make sure that Autologin worked, services would startup and everything was working as expected. I finally powered down the VM and created a new snapshot so this could be used as a parent virtual machine for future linked clones.

I like to have easy access to a variety of lab environments. I keep a fairly active home lab which has a focus on persistent virtual machines — like running copies of various vCenter Server releases, vRealize Suite, Horizon, etc. I like to consider these sort of ‘production’ as when they break I will typically troubleshoot and repair them in place. However, I also like to have very disposable environments that can be destroyed and easily recreated to iterate through some testing. I’ve used an environment before that referred to these as ‘smash labs’ because you could snapshot and smash things as necessary. I’ve been using VMware Workstation and linked clones for a while to provide this sort of ‘smash lab’ environment. I recently had a need to switch PCs and decided to rebuild this environment and document the process. The next few blog posts will focus on various aspects of this project, starting with the end result, and then posts covering the builds for each individual VM.

1. lab-mgmt-01: The management console/GUI that also acts as a domain controller, DNS Server, Certificate Authority, and NAT gateway.
2. lab-esxi-02 and lab-esxi-03: I’ll cover these two VMs in one post because they are very similar. One is a nested ESXi 7.0.3 host and the other is a nested ESXi 8.0.0 host, each containing a corresponding vCenter Server Appliance.
3. lab-dock-14: A Photon OS VM with docker and nfs-server services enabled.

We can see all four of these VMs in the following screenshot. You can see that I named them with a parent_ prefix, and that is because I don’t typically power on these VMs, but instead create linked clones that are disposable. This allows me to create various instances of these VMs and switch between them as needed.

For example, if I need to test something, like an SSL certificate replacement script for vCenter Server 7.0.3, I would create linked clones of parent_lab-mgmt-01 and parent_lab-esxi-02 by right clicking the VM > Manage > Clone. In the wizard that pops up I would select “An existing snapshot (powered off only)” and then selecting “Create a linked clone” on the next page. After giving the VM a name, the clone operation is completed nearly instantly. Relevant screenshots are included below:

These linked clones can be powered on and used as needed. The VMs have static IP addresses, so the way networking is configured I can only power on one linked clone copy at a time, but due to resource limitations on my laptop this hasn’t been a problem.

While we are talking about IP addresses, it is probably helpful to understand the topology that we’ve built. The following image should capture how the VMs interact with each other and the outside network.

As you can see, the lab-mgmt-01 virtual machine is serving as a gateway between the lab network and rest of the network. If we need to test something as if we are in an airgapped network, we can simply disable the NIC1 (Ethernet0) interface on the management server. Without this adapter working, the rest of the lab becomes isolated from the internet and any other services.

The parent VMs could have a few snapshots. As shown in the following example, one of the snapshots is in use and therefore is locked and cannot be deleted.

Once the test is complete, I could power off my two temporary clones and either delete them or keep them for a couple of days (in case I need to refer back to logs or such).

The next few posts will focus on configuration of the individual component VMs that make up this ‘smash lab’ environment.