In my home network, I have a Raspberry Pi4 which provides DNS (pi-hole) and NTP (chrony). Its a device that I don’t touch often and is a ‘production’ type service — in my lab I don’t mind blowing up / breaking things… but this device needs to be stable. If DNS goes offline the family can’t stream shows and it’s a real production down sort of situation. Systems in my lab consume NTP from this device, and regular devices in my home network rely on it providing DNS (for ad blocking as well as conditional forwarding of lab domains to DNS servers in the lab). A few days ago, I noticed that this system was down — it wasn’t answering DNS requests and SSH/VNC wasn’t working. After power cycling the system, I was also no longer able to ping the device. After a bit of troubleshooting, I realized that the SD card used as boot media had failed. The system had been running 24×7 for ~5 years, logging DNS requests and such, probably more write IO than anyone should expect from a consumer SD card.
To resolve the issue I ordered a new SD card… but I realized that this system had about 5 years of various configurations. I’m going to attempt to document the configuration (at least what I remember about it) below.
OS Installation
The previous Raspberry Pi used the Raspbian OS with a GUI. However, I never really used the GUI and primarily access this system remotely. Since most other systems I manage use Ubuntu (specifically 24.04), I decided to install that OS using the server instructions from here: https://ubuntu.com/tutorials/how-to-install-ubuntu-on-your-raspberry-pi#1-overview.
I used the Raspberry Pi Imager for Windows, which allowed me to customize the username/password, hostname, etc of the OS so that it booted up and I could connect via ssh.
Once I was logged into the system, the first thing I did was make sure it was up to date using sudo apt update && sudo apt upgrade. This installed a bunch of updates, so I rebooted for good measure.
Lab Certificate
In rare cases, I’ll access something from my lab from the Raspberry Pi. To make this work without certificate warnings, I installed the lab CA certificate. This is just two commands, one to copy the file and another to update the certs.
sudo wget http://www.example.com/build/rootca-example-com.crt -P /usr/local/share/ca-certificates
sudo update-ca-certificatesInstall extra packages
I had a handful of extra packages that I installed. I’ll discuss each of these later, but for now we’ll install them all in one pass.
sudo apt install sssd-ad sssd-tools realmd adcli chrony tinyproxyProxy Server
For some occasional testing, I’ll use a proxy server in my lab. This was running in a dedicated VM, but while I’m revisiting things, I decided to co-locate it on this appliance.
# configure proxy
sudo nano /etc/tinyproxy/tinyproxy.conf
# change LogLevel from Info to Warning
# Allow 192.168.0.0/16 by removing comment
sudo systemctl reload tinyproxyNTP (chrony)
I prefer having NTP servers running on physical devices. Since I don’t have many of those in the lab, I use the Raspberry Pi as a locally accessible NTP server. I’m using the chrony service to do this and allow anything in the lab to query this device for time.
# configure NTP
sudo nano /etc/chrony/chrony.conf
# append the following comment / allow lines to the file
# Define the subnets that can use this host as an NTP server
allow 192.168.0.0/16
sudo systemctl restart chrony.servicePi-Hole
The reason I first purchased this Raspberry Pi was to block ads on my home network using pi-hole.
curl -sSL https://install.pi-hole.net | bash
# create a custom config file for various forward/reverse domain forwarding:
sudo nano /etc/dnsmasq.d/05-custom.conf
# contents of above new file
server=/lab.enterpriseadmins.org/192.168.127.30
server=/lab.enterpriseadmins.org/192.168.32.30
server=/example.com/192.168.127.30
server=/example.com/192.168.32.30
server=/168.192.in-addr.arpa/192.168.127.30
server=/168.192.in-addr.arpa/192.168.32.30
# from web UI, restart resolver.
# Update pihole settings > DNS, change from recommended allow only local requests to 'permit all origins' so that all lab subnets can resolve names.
# enable php for non-pihole /admin locations
sudo lighttpd-enable-mod fastcgi fastcgi-php
sudo service lighttpd reload
# Create redirect page for / to /admin
echo '<head> <meta http-equiv="Refresh" content="0; URL=/admin" /> </head>' | sudo tee /var/www/html/index.html
# Create 'get-hostname.php' file in /var/www/html as well, this is for Aria Ops management pack. The contents of the file should be:
<?php echo '{"hostname":"' . gethostname() . '"}'; ?>Active Directory Join
Most Ubuntu boxes in my lab are joined to Active Directory for common logins. I configured the same for the Raspberry Pi, although it is not really required.
# configure AD
echo '%lab\ linux\ sudoers ALL=(ALL) NOPASSWD:ALL' | sudo tee -a /etc/sudoers
sudo /usr/sbin/pam-auth-update --enable mkhomedir
sudo /usr/sbin/realm join lab.enterpriseadmins.org -U svc-windowsjoin --computer-ou "ou=services,ou=lab servers,dc=lab,dc=enterpriseadmins,dc=org"
sudo sed -i -e 's/^#\?use_fully_qualified_names.*/use_fully_qualified_names = False/g' /etc/sssd/sssd.conf
sudo systemctl restart sssd.serviceStatic IP
Once everything was configured/ready, I decided to put the device in service by changing the IP from the DHCP address originally obtained to the static IP address I have configured on most devices.
network:
version: 2
ethernets:
eth0:
match:
macaddress: "dc:a6:32:aa:aa:aa"
dhcp4: no
addresses: [192.168.127.53/24]
routes:
- to: default
via: 192.168.127.254
nameservers:
addresses: [192.168.127.53,192.168.32.53]To make the new network settings active, we must apply those file changes with sudo netplan apply.
Cleanup
The Raspberry Pi Imager utility used cloud-init to do some customizations. This was running at each startup and left a few messages on the system console. Since we no longer need cloud-init after the system is online, we’ll just remove the package with:
sudo apt purge cloud-initConclusion
The Raspberry Pi in my lab has been running for about 5 years with little to no maintenance. Other than this one failed SD card things have been very reliable. The steps here are mostly notes for future reference if I need to rebuild the device again. Hopefully you’ll find the notes helpful.