How to Build an Ubuntu 24.04 Template using Packer and vCenter Server Customization Specifications

I’ve heard from several people who have highly recommended Packer (https://packer.io) to create standardized images, and with the recent release of Ubuntu 24.04 I decided now would be a good time to build a new template to try it out. This template will be used for most Linux VMs deployed in my vSphere based home lab so in addition to the base install of Ubuntu, there are a handful of agents/customizations that I wanted to have available:

• Aria Operations for Logs agent
• Aria Automation Salt Stack Config minion
• Trusts my internal root CA
• Joined to Active Directory for centralized authentication

I wanted to make this process as low touch as possible, so I needed to automate several agent installations and other customizations. I ended up with a set of packer configuration files and a customization spec that did exactly what I wanted. With each install or customization, I had to decide if it would be best to include the automation in the base image (executed by Packer) or the customization spec (executed by the customization script). Some of this came down to personal preference, and I might revisit the choices in the future. For example, I’ve placed the code to trust my internal CA into the base template. I may want to evaluate removing that from the template and use multiple customization specs to allow for different certificate-related options.

Following are the details and download of my final results and the notes about the steps that I followed with some troubleshooting steps that I had to perform.

My Final Packer Configuration

The Packer Configuration spans several files. I’ve described each file below and provide a link to a zip file with my working configuration here.

• http\meta-data – this is an empty file but is expected by Packer.
• http\user-data – this file contains a listing of packages installed automatically and some commands that get run automatically during the template creation, for example VMware Tools customization or to execute custom scripts.
• setup\setup.sh.txt – this script runs in the template right before it is powered off. It contains some cleanup code and agent installs. You’ll need to rename this file to remove the .txt extension if you want it to execute.
• ubuntu.auto.pkr.hcl – contains variable declarations and then defines all the virtual machine settings which are created.
• variables.pkrvars.hcl – contains shared code (vCenter Server details, credentials, Datacenter, Datastore, etc) which may be consumed by multiple templates.

Creating the Configuration Files

I downloaded the Ubuntu 24.04 ISO image, placed it on a vSphere datastore, and noted the path to this file for the next step.

I had previously saved some sample configuration files for Ubuntu 22.04 (unfortunately I didn’t bookmark the original source) to use as my starting point. I cleaned up the files a bit, removing some declared variables that weren’t in use. You will need to edit the iso_paths property in the ubuntu.auto.pkr.hcl file and other credential/environmental values in the variables.pkrvars.hcl to customize them to your environment. Also review the setup.sh script for any changes you want to make.

I then downloaded the Packer binary for my OS (from: https://developer.hashicorp.com/packer/install?product_intent=packer) and placed it in my working directory.

I knew there were two commands to run:
The init command initializes packer, this will download the vSphere plugin we’ve specified. The build command will actually kick off the template build.

./packer.exe init .
./packer.exe build -var-file="variables.pkrvars.hcl" .

The first time I ran build, I got an error that said could not find a supported CD ISO creation command (the supported commands are: xorriso, mkisofs, hdiutil, oscdimg). Since I was running packer on Windows, I downloaded and ran ADKsetup.exe from https://go.microsoft.com/fwlink/?linkid=2243390 . During the setup, I unchecked all components except Deployment Tools, which will install oscdimg. After installation, I rebooted and had to add the path to oscdimg (“C:\Program Files (x86)\Windows Kits\10\Assessment and Deployment Kit\Deployment Tools\amd64\Oscdimg” on my machine) to my System Environment Variable.

The initial build created a vSphere template in about 6 minutes however, the first deployment of the template failed. Reviewing the var/log/vmware-imc/toolsDeployPkg.log file, the message ERROR: Path to hwclock not found. hwclock was observed. There was a KB article for this (https://knowledge.broadcom.com/external/article?legacyId=95091) related to Ubuntu 23.10, which mentioned that the util-linux-extra package was needed. I added this to the definition of packages in the user-data file and rebuilt the template using packer build. This resolved the issue and future deployments were successful.

I noticed that the resulting virtual machine had two CD ROM devices. I looked around and found a PR (link) stating that an option existed to control this behavior as of the vSphere 1.2.4 plugin. I updated the required_plugins mapping in the ubuntu.auto.pkr.hcl file to state this 1.2.4 version is the minimum required. I then added reattach_cdroms = 1 later in the file with the other CD ROM related settings.

One other thing that I noticed was that it would have been helpful to have a date/time stamp either in the VM name or the notes field, just to know when that instance of a template was created. I discovered how to get a timestamp and added notes = "Template created ${formatdate("YYYY-MM-DD", timestamp())}" property to my ubuntu.auto.pkr.hcl file.

After making the above fixes, I deployed a VM from the latest template and applied a customization spec which contained a customization script to do a few final customization tasks (update /etc/sudoers, generate a new openssh-server key, complete the domain join, make a change to the sssd configuration and finally restart ssd services. This script failed to execute. Reviewing the /var/log/vmware-imc/toolsDeployPkg.log I noticed the message user defined scripts execution is not enabled. To enable it, please have vmware tools v10.1.0 or later installed and execute the following cmd with root privilege: 'vmware-toolbox-cmd config set deployPkg enable-custom-scripts true'. Back in my user-data configuration file, in the late-commands section, I added this command to enable custom scripts in the template.

After rebuilding the template to enable custom scripts, I deployed a new VM. This did not complete the domain join as I had hoped. All of my commands were running in a precustomization period, before the virtual machine was on the network. I found the following KB article: https://knowledge.broadcom.com/external/article?legacyId=74880 which described how to run some commands in precustomization and others during postcustomization. Moving the domain join to postcustomization solved this issue, as the VM was on the network when the domain join ran.

I wanted the template to trust my internal CA, so I added a few commands to the setup.sh script to download the certificate file from an internal webserver and run update-ca-certificates.

The next task I wanted to complete was the installation of the Aria Automation Config (aka Salt Stack Config) minion. In the past I had used the salt-project version of the minion, but reviewing VMware Tools documentation (https://docs.vmware.com/en/VMware-Tools/12.4.0/com.vmware.vsphere.vmwaretools.doc/GUID-373CD922-AF80-4B76-B19B-17F83B8B0972.html) I found an alternative way. I added the open-vm-tools-salt-minion as a package in the user-data file and had Packer add additional configuration_parameters to the template to specify the salt_minion.desiredstate and salt_minion.args values.

I also wanted the template to include the Aria Operations for Logs (previously vRealize Log Insight) agent. The product documentation showed how to pass configuration during install (https://docs.vmware.com/en/VMware-Aria-Operations-for-Logs/8.16/Agents-Operations-for-Logs/GUID-B0299481-23C1-482D-8014-FAC1727D515D.html). However, I was having problems automating the download of the agent. Trying to do a wget of the link from the agent section of the Aria Ops for Logs console the resulting file was an HTML redirect. I found this article: https://michaelryom.dk/getting-log-insight-agent which provided an API link to download the package and I was able to wget this file. I placed the wget and install commands in the setup.sh script that runs right before the new template is powered down.

After rebuilding the template with packer, I deployed another test VM. I confirmed that:

• SSH worked
• AD Authentication worked
• The Aria Ops for Logs agent sent logs
• My internal CA was trusted
• The Aria Automation Config minion was reporting (the key needed accepted in the console)

To repackage the template VM takes about 6 minutes. To deploy & customize the template takes about 2 minutes, and everything is setup exactly as I want it to be.

vSphere Customization Specification > Customization script

As described above, I chose to perform several tasks in a vSphere Customization Specification, such as how to name the VM, the time zone, network settings, etc. The part of this script which really helps with completing some of the desired customizations was the customization script. This took a bit of trial and error so I’ve included my final script below as reference. This code runs as part of the virtual machine deployment and is unique to each VM.

#!/bin/sh
if [ x$1 = x"precustomization" ]; then
    echo "Do Precustomization tasks"
    # append group to sudoers with no password
    echo '%lab\ linux\ sudoers ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers
elif [ x$1 = x"postcustomization" ]; then
    echo "Do Postcustomization tasks"
    # generate new openssh-server key
    test -f /etc/ssh/ssh_host_dsa_key || dpkg-reconfigure openssh-server
    # make home directories automatically at login
    /usr/sbin/pam-auth-update --enable mkhomedir
    # do a domain join and then modify the sssd config
    echo "VMware1!" | /usr/sbin/realm join lab.enterpriseadmins.org -U svc-windowsjoin --computer-ou "ou=services,ou=lab servers,dc=lab,dc=enterpriseadmins,dc=org"
    sed -i -e 's/^#\?use_fully_qualified_names.*/use_fully_qualified_names = False/g' /etc/sssd/sssd.conf
    systemctl restart sssd.service
fi