Directory partition has not been backed up [DC=virtualcenter,DC=vmware,DC=int]

With the help of Alan Renouf’s vCheck (Daily Report) script I recently started noticing an error in my Windows event log on a vCenter server.

This directory partition has not been backed up since at least the following number of days. Directory partition: DC=virtualcenter,DC=vmware,DC=int 'Backup latency interval' (days): 90 It is recommended that you take a backup as often as possible to recover from accidental loss of data. However if you haven't taken a backup since at least the 'backup latency interval' number of days, this message will be logged every day until a backup is taken. You can take a backup of any replica that holds this partition. By default the 'Backup latency interval' is set to half the 'Tombstone Lifetime Interval'. If you want to change the default 'Backup latency interval', you could do so by adding the following registry key. 'Backup latency interval' (days) registry key: System\CurrentControlSet\Services\NTDS\Parameters\Backup Latency Threshold (days)

This error only occurred in one of the three vCenter instances I manage. After checking with my enterprise backup team and confirming this server does get backed up on a regular basis I opened an support request (SR) with VMware Tech Support. Two weeks later VMware technical support finally contacted me about my SR. The response: since the error occurs as part of the ADAM/LDAP instance that comes with the underlying operating system please contact your Operating System vendor for support. After waiting as long as I did for an initial response I had higher hopes for an actual solution.

I started Google-ing for solutions on backing up the ADAM/ADLDS instance on my own. I found an excellent article on how to do this in powershell at http://www.open-a-socket.com/index.php/2009/04/21/schedule-backups-of-your-ad-lds-instance-using-dsdbutil-2/. Here is the actual script that I now run as a scheduled task:

[cc lang=”powershell”]
$vcenter = “vcenter.host.name”
$date = Get-Date -Format yyyy-MM-dd
$backpath = “D:\ESXiBackup\$vcenter\$date\”

#Create a backup of the ADAM Instance VMwareVCMSDS
#http://www.open-a-socket.com/index.php/2009/04/21/schedule-backups-of-your-ad-lds-instance-using-dsdbutil-2/
md “$backpath#VMwareVCMSDS”
$cmd = $env:SystemRoot + “\system32\dsdbutil.exe `”ac i VMwareVCMSDS`” ifm `”create full $backpath\#VMwareVCMSDS`” q q”
Invoke-Expression $cmd
[/cc]

Now that I have my own VCMSDS backup — and no errors in the event log — I can sleep better at night.

Posted in Scripting, Virtualization | 2 Comments

ADAM Instance VMwareVCMSDS index may be corrupt

VMware vSphere vCenter Server installs an Active Directory Application Mode (ADAM) [aka Active Directory Lightweight Directory Service (ADLDS)] instance called VMwareVCMSDS. I have had several problems with this service — one issue with the contents of the directory (see vCenter Migration) and a couple with the actual ADAM/ADLDS instance. In a recent upgrade I moved a lab environment from vSphere 4.0 to 4.1 and started getting the following event log message:

VMwareVCMSDS (2428) ADAMDSA: Database 'D:\Program Files\VMware\Infrastructure\VirtualCenter Server\VMwareVCMSDS\adamntds.dit': The secondary index 'INDEX_00000003' of table 'datatable' may be corrupt. If there is no later event showing the index being rebuilt, then please defragment the database to rebuild the index.

The solution to this problem was rather simple — but required working directly with the ADAM instance. This took a little trial and error but here is a description of the solution in case you run into the same error:

Stop the VMwareVCMSDS service:

net stop ADAM_VMwareVCMSDS

Enter the directory service database utility application:

dsdbutil

The following commands get you to the file configuration settings:

Activate Instance VMwareVCMSDS
Files
info

Here is my test systems file configuration:

Drive Information:

C:\ NTFS (Fixed Drive  ) free(5.4 Gb) total(23.9 Gb)
D:\ NTFS (Fixed Drive  ) free(2.6 Gb) total(3.9 Gb)

DS Path Information:

Database   : D:\Program Files\VMware\Infrastructure\VirtualCenter Server\VMwareVCMSDS\adamntds.dit - 4.1 Mb
Backup dir : D:\Program Files\VMware\Infrastructure\VirtualCenter Server\VMwareVCMSDS\dsadata.bak
Working dir: D:\Program Files\VMware\Infrastructure\VirtualCenter Server\VMwareVCMSDS
Log dir    : D:\Program Files\VMware\Infrastructure\VirtualCenter Server\VMwareVCMSDS - 30.0 Mb total
edbres00002.jrs - 10.0 Mb
edbres00001.jrs - 10.0 Mb
edb.log - 10.0 Mb

To compact/defrag the database type the following command in the dsdbutil screen:

compact to d:\temp\defragged

Here are the results of the compact operation:

Initiating DEFRAGMENTATION mode...
Source Database: D:\Program Files\VMware\Infrastructure\VirtualCenter Server\VMwareVCMSDS\adamntds.dit
Target Database: d:\temp\defragged\adamntds.dit

Defragmentation  Status (% complete)

0    10   20   30   40   50   60   70   80   90  100
|----|----|----|----|----|----|----|----|----|----|
...................................................

It is recommended that you immediately perform a full backup
of this database. If you restore a backup made before the
defragmentation, the database will be rolled back to the state
it was in at the time of that backup.

Compaction is successful. You need to:
copy "d:\temp\defragged\adamntds.dit" "D:\Program Files\VMware\Infrastructure\VirtualCenter Server\VMwareVCMSDS\adamntds.dit"
and delete the old log files:
del D:\Program Files\VMware\Infrastructure\VirtualCenter Server\VMwareVCMSDS\*.log

I created a zip file contianing all the files in: “D:\Program Files\VMware\Infrastructure\VirtualCenter Server\VMwareVCMSDS” before proceeding with the given instructions.

copy "d:\temp\defragged\adamntds.dit" "D:\Program Files\VMware\Infrastructure\VirtualCenter Server\VMwareVCMSDS\adamntds.dit"

Answering yes to the overwrite files confirmation.

del "D:\Program Files\VMware\Infrastructure\VirtualCenter Server\VMwareVCMSDS\*.log"

After completing the defragmentation I started the VMwareVCMSDS service again but had problems authenticating back to vCenter. Instead of restarting the vCenter service I went ahead and rebooted the server — which seems to have cleared up all my errors.

Posted in Virtualization | Leave a comment

Active Directory: Find recent NO_CLIENT_SITE subnets

From time to time I try to verify that all of my subnet boundaries are properly defined in AD Sites and Services. At some locations this is a little more challenging — especially when a site has many domain controllers. I recently threw together this PowerShell script to look at a site with over a dozen DCs:

Foreach ($server in get-content servers.txt) {Tail -10 \\$server\admin$\debug\netlogon.log | %{"$server,$_"}}

I had a file (servers.txt) that contained the name of each domain controller on a separate line. I ran the script using an Enterprise Admin account as it has permissions to the admin$ on each DC.

If you wanted to get really creative, you could add some select logic to get a list of all domain controllers. The get-date cmdlet could even compare the date format listed in the netlogon.log (MM-dd) and only give you the current days events. Throw in a scheduled task and now you could really be on top of this 🙂

Posted in Scripting | Leave a comment

Create a bunch of snapshots with PowerCLI

Last week I was working with snapshots in my lab. I came up with this one liner to create a bunch of snapshots on the same virtual machine:

for ($i=1; $i -le 5; $i++) {get-vm "snaptest" |new-snapshot -name "snapshot00$i" -Description (get-date -format "'Created: 'yyyy-MM-dd HH:mm") -memory:$true}

As you can see I’m using the virtual machine snaptest and the snapshots will be named snapshot001-snapshot005 and have a description containing the date/time the snapshot was created.

Posted in Scripting, Virtualization | Leave a comment

Install VMware Tools in openfiler

I have a couple of lab type vSphere setups that use Openfiler virtual machines for shared storage. For several months these have been running with out of date VMware tools. Several weeks ago I was able to upgrade these tools and wanted to share the required steps.

The first required task was to install a C Compiler — specifically gcc. While at it I updated all other components and shutdown the virtual machine:

conary updateall
conary update gcc
shutdown -h now

After the machine was powered down I added a virtual CD-ROM drive. This drive will be used to mount the VM tools ISO. After adding a CD-ROM drive the virtual machine can be powered back on. Select install/upgrade tools from the menu.

mkdir /mnt/cdrom
mount /dev/cdrom /mnt/cdrom
cp VMwareTools* /mnt
tar zxvf /tmp/VMwareTools*
cd vmware-tools-distrib
./vmware-install.pl

I accepted the defaults on the vmware-install.pl configuration script.

Special thanks to this communities article for a head start: http://communities.vmware.com/docs/DOC-9332;jsessionid=A7F5565895191AE1D161E4749AF86B12

Posted in Virtualization | 3 Comments