Running Debian 12 on Raspberry Pi
Due to general dissatisfaction with the distro I had running on a Pi (along with Red Hat’s source rugpull), I decided to try Debian 12 on it. It worked so well that I converted my other Pi a few weeks later. This is an account of the experience I had with doing so and what I learned along the way.
Background
In my last entry, I mentioned that I was going to convert my Raspberry Pi 4 running Arch Linux ARM to Debian 12 ARM64 as a way to test the viability of Debian to potentially replace a RHEL-like on my primary server. It’s been quite a while since I ran any kind of Debian, whether one of its many variants or the OS by itself.
To elaborate further on my history with Debian, I vaguely remember running sid at my first job on my workstation. I’m not sure if I started on slink or potato, why I used it or why I stopped using it..my long-term memory can be very fuzzy, especially about that time in my life. I remember not liking Debian all that much, preferring my first distro, Slackware (so old school they’re not using https..), and feeling disdainful towards “The Debian Way” (and maybe resentful about all the online boasting about apt and how easy it made it to maintain packages and upgrade systems when every other distro was struggling with these processes). Considering all of this, it’s surprising to me now that I gave it a chance back then.
Installation
Moving forward to the present (which I might be able to remember somewhat more clearly) I got a new MicroSD card and used dd to write a bookworm image to it from the site Raspberry Pi Debian images. I also assembled a list of software that was running on the target system, noting what was in Debian’s stable repository, what was in external repos, and what I’d have to compile myself. I made a final backup, powered off the Pi, switched the cards so I could use the old system’s data if needed, hooked up a keyboard and monitor (this Pi is wifi only), and powered it on.
As the documentation notes, the root password is blank; your first order of business should be setting one and then adding a separate user that’ll use sudo (or opendoas, if you’d like) as needed.
Getting WiFi running was straightforward using the Debian WiFi documentation, but it was not persisting after a reboot. Since the WiFi MAC was the same, it came up with the same IP4 address that the old system did, which was something I didn’t have to change in local DNS but it wasn’t getting an IP6 address at all. It took some troubleshooting and experimentation, but I came up with a successful method to get everything working properly:
# The wpa_* programs are in the wpasupplicant package, which is included with the
# images from raspi.debian.net.
# First, use wpa_passphrase to create the config file, filling in the ssid you
# want to connect to:
wpa_passphrase <ssid> > wpa_supplicant-wlan0.conf
# This will prompt for the SSID password.
# Move wpa_supplicant-wlan0.conf to /etc/wpa_supplicant.
# Set it to root:root 0640.
# Edit it to add:
ctrl_interface=/run/wpa_supplicant
update_config=1
# It should now look something like:
network={
ssid="your_ssid"
psk=$pre_shared_key
# See https://www.rfc-editor.org/rfc/rfc6617 for further details
# about pre-shared keys.
}
ctrl_interface=/run/wpa_supplicant
update_config=1
# Edit /etc/network/interfaces.d/wlan0:
allow-hotplug wlan0
iface wlan0 inet dhcp
iface wlan0 inet6 dhcp # Use this for dhcpv6. If SLAAC is available, use inet6 auto.
# If this Pi is only using WiFi, then remove the line "auto eth0"
# from the top of /etc/network/interfaces.d/eth0
# The files in interfaces.d are also root:root 0640.
# In order to make these changes persist across reboots, run:
systemctl disable --now wpa_supplicant
systemctl enable --now wpa_supplicant@wlan0
# This will ensure that the wlan0 config will run instead of..
# whatever else it was doing when I was troubleshooting it.
# Run
systemctl restart networking
# And check to see if the configuration is what you're looking for.
# This method can also be used before booting the Pi by mounting the SD card
# after writing the image and chrooting into it. This can also be used to set up
# users/passwords/etc. Either way should lead to the same destination,
# so use whatever you're comfortable with.
At this point, the system should be on the network and ready to start installing/updating software. IP6 was working for me now, though the address had changed..ah well, I couldn’t avoid changing DNS after all. After taking care of that, I used something like this combination as root to get things going:
export PATH=$PATH:/usr/sbin # For dpkg-reconfigure and visudo.
apt update
apt upgrade
apt remove nano # *glares in fox*
apt install vim sudo locales debconf
dpkg-reconfigure locales
visudo
# I added my unprivileged user to the sudo group at this point.
# After this, I could use sudo to do the rest.
# Depending on what image you use, apt upgrade might bring in a kernel update,
# which apt usually seems to note as someting to reboot for.
And now, I was off and running..
Configuration
Most of the configuration/databases/other files that I copied over after installing the corresponding programs worked right away (various permissions had to be adjusted, though, as role accounts can have different names across distributions.) One example of this that required a lot of troubleshooting was a case where Grafana was loading very inconsistently through its nginx reverse proxy. One thing that helped was getting the default nginx.conf from Debian’s package; this showed me that www-data is the proper user to run nginx as in Debian and to use for assigning permissions. Using the debugging console in a browser showed me that a few files were not being loaded consistently; when they weren’t, the site didn’t work at all. After finally figuring out that it was most likely a permissions issue, I then realized that…I didn’t set up error logging properly in nginx. After taking care of that, the permissions issue was fixed very quickly.
(Update: Grafana recently changed the gpg.key file at https://apt.grafana.com/gpg.key and the –dearmor method didn’t work but a straightforward copy did. Thus, I’ve updated this section.)
Another issue I ran into was making apt trust an external repository; grafana in this case. The usual solution is to take the key file and add it to /etc/apt/trusted.gpg.d with root:root 0644 permissions and an .asc extension. It’s possible that it won’t be in the format that apt is looking for, though. In that case, run:
gpg --output grafana.asc --dearmor gpg.key
After adding the source list to /etc/apt/sources.list.d, you should now be able to install packages from the repo.
A lot of the other things I had to do to get everything running properly are specific to my setup/didn’t require anything I couldn’t find by searching, but I want to note how to set up the ARM side libraries that are used for getting low-level information on a Pi. These appear to be installed by default in Raspberry Pi OS, (formerly known as Raspbian), but to get them on Debian or another distro running on a Pi, run:
# Install git if needed
git clone https://github.com/raspberrypi/userland.git
cd userland
./buildme --aarch64
This should install everything needed to /opt/vc. While not strictly necessary to run Linux on a Pi, I’ve found these to be useful for monitoring temperatures and other useful bits of information.
Thoughts on using Debian, then and now
When I had my first go-round with Debian, I remember it feeling very different configuration-wise from Slackware and Red Hat (pre-RHEL). APT had far better updating/dependency tracking capabilities than any other Linux distro I knew of at that time, which was its main selling point. Being able to upgrade from one major release to another with only a reboot also seemed incredible as “backup, reformat, reinstall, and restore” was the standard method. And yet, it still didn’t click with me..which seems silly in retrospect, though I was rather obstinate and stubborn in my views as a young fox, so I don’t think I gave Debian a fair chance.
After converting one Pi to Debian, I had such a good experience with it, I planned and executed another conversion on my other Pi (which was a lot easier to get going as it had Ethernet available and I didn’t need to configure Wifi) and was a lot of fun to do, overall. Having a reference system was very useful and I was able to simplify the configuration/setup I had had on my other Pi previously; getting rid of unnecessary things always helps in terms of having fewer “moving parts”/things to potentially go wrong.
While I’m still learning/figuring out the intricacies of managing Debian as a server, having it on systemd has helped me adjust more quickly. (I know it can use other init systems but I’m fine with the default.) It seems odd to me that /etc/init.d is still around and packages put files in there, including packages you wouldn’t expect to, such as sudo. (sudo’s entry has a reference to restorecon, which is odd for me to see outside of RHEL but selinux is available in Debian, along with firewalld, chrony, and other programs associated with RHEL.)
When I think about it, though, having /etc/init.d present is for other init programs and Debian doesn’t assume that systemd is always going to be used. It sets defaults such as systemd and apparmor, but they can be changed if needed. Same with their network configuration defaults; it can be changed to NetworkManager or something else if desired. A RHEL veteran such as myself can use familiar programs..or something else entirely; Debian doesn’t seem to force anything the way other distros can.
That fits with their slogan of “The Universal Operating System”, which I think applies to this method of having a considerable variety of packages available along with how many architectures are supported. Run it on whatever you want however you want.
The following statement is not a slight on Arch Linux ARM at all, as they are a small team of volunteers separate from the main Arch project working hard to keep their packages and processes updated as Arch keeps rolling forward (Arch Linux 32 is another such standalone port) without any noticeable support from main. There’s a significant difference between that and Debian ARM64, which is officially part of the Debian project and is updated/maintained alongside the other supported architectures. Arch ARM was mostly stable for me, but when it broke, it broke badly. I’m glad I was able to help with one such incident, but that wasn’t the first or last time something like that happened. Granted, it’s Arch and that’s standard operating procedure with that distro, but I found myself wanting that particular Pi to be a lot more stable. Debian ARM64 has done just that.
Various issues/observations
I have had some issues with it, as is to be expected. The Rosetta Stone on Arch Wiki has been invaluable in figuring out how to run the equivalents of the package query commands I know from RHEL. Yet, the output for the commands rpm -qf and dpkg -S can be inconsistent.
For example, if I run rpm -qf on a given file on a RHEL-like system, the output is as expected:
$ which dd
/usr/bin/dd
$ rpm -qf /usr/bin/dd
coreutils-8.30-15.el8.x86_64
If I run dpkg -S on Debian, though:
$ which dd
/usr/bin/dd
$ dpkg -S /usr/bin/dd
dpkg-query: no path found matching pattern /usr/bin/dd
# What the?!
# Since /bin is symlinked to /usr/bin, perhaps..
$ dpkg -S /bin/dd
coreutils: /bin/dd
This makes sense after listing the Debian coreutils package and seeing that dd is listed under /bin. After looking into it further, this occurs because of /usr/bin being listed before /bin in the PATH environmental variable, so the which command returns /usr/bin/dd instead.
So, what happens if I try the reverse on RHEL?
$ rpm -qf /bin/dd
coreutils-8.30-15.el8.x86_64
This is with dd listed under /usr/bin in the RHEL package. This isn’t a major issue, of course, but I tend to notice things like this.
I had more significant issues with a pair of packages, alertmanager and nala. I had trouble with the former’s web interface, something that I expected to be included as it has been every other time I’ve installed it. In the Debian version, though, it’s optional and needs to be set up with a script included in the package, /usr/share/prometheus/alertmanager/generate-ui.sh. Running this script kicks off an apt process that wants to install about 151 packages (on a configured system) or 259 (on a minimal container), mostly golang ones. I balked at what, to me, is an excessive amount of packages to install just to build a component. Thus, I used a container to build the web ui.
My troubles were only beginning, though. The first problem was
Compiling source code...
-- ELM VERSION MISMATCH ----------------------------------------------- elm.json
Your elm.json says this application needs a different version of Elm.
It requires 0.19.1, but you are using 0.19.0 right now.
To fix this, I had to manually download 0.19.1, extract it, add a line in the script to copy it to $TMPDIR, comment out the lines that download the package..and then run it. Already, this is way too much work to do just to activate a feature. How did the alertmanager package ship with such a broken script?
# For the record, the changes I made:
$ wget https://github.com/elm/compiler/releases/download/0.19.1/binary-for-linux-64-bit.gz
$ gunzip binary-for-linux-64-bit.gz
$ mv binary-for-linux-64-bit elm
# Then in the script:
#echo "Downloading Elm tools..." >&2
#curl --location $ELMDISTURL | tar xz -C $TMPDIR
cp elm $TMPDIR
But that’s not all. After copying the results of the elm compilation onto the system, it failed to load properly. The culprit was an incorrect symlink; according to my notes:
* Fix a bug where a symlink was made to /usr/share/nodejs/bootstrap/dist via bootstrap4 when the
ui is looking for bootstrap.
It still wasn’t working, so after more troubleshooting, I had to run
mkdir -p /usr/share/prometheus/alertmanager/ui/lib/elm-datepicker/css
chmod o+r -R /usr/share/prometheus/alertmanager/ui/lib/
To add a directory that it was looking for and fix permissions. Then, finally, I had to copy /usr/share/gocode/src/github.com/prometheus/alertmanager/ui/app/lib/elm-datepicker/css/elm-datepicker.css out of the container and into the css directory that I had just made.
Only after all of that did the web interface work.
True, the bugs aside from the script might have stemmed from moving the build from a container and the script has been fixed in this bug, though it has yet to make its way to the backports for bookworm. Still, I’m astonished that this wasn’t in a separate package. After digging through the package site, I found a patch that removes the embedded webui to conform with Debian policy. This is fine, but I don’t think that should preclude having it as a separate package, even if it has to be in contrib or otherwise outside of the main distribution. prometheus-alertmanager-webui, depends on prometheus-alertmanager, done.
As for nala..well, I accidentally created a memory leak (it nearly crashed the Pi it was on), was able to reliably reproduce it, and then I reported it in this bug, but there has been no action on it as of this writing. The leak stems from nala being confused about being unable to read its history file in /var/lib/nala/history.json, which happens if the command “nala history” is run as a user that doesn’t have read access on this file, such as an unprivileged user with that file set to 0640. I don’t remember setting it to that, and fresh test installs in containers and such set that file to 0644..so it’s possible that I changed it and then promptly forgot about it..(wouldn’t be the first time).
Still, being unable to read a file should not get the program caught in a loop of systematically grabbing memory and not releasing it until physical memory is nearly exhausted and the oomkiller has to come out to save the system. All it has to do is something like
# Nala is currently written in Python, thus:
print('ERROR: Cannot read history file at ' + NALA_HISTORY + '. Exiting.')
exit(255)
..and it’s fixed. I’m so used to programs erroring out when they can’t read a file that I’m still incredulous that a) this program doesn’t know how to handle it and b) it handles it in the worst way possible. This is an edge case, but it can cause a denial of service given that specific setup. Given my general lack of familiarity with “The Debian Way”, I’m not sure what the proper procedure is to escalate this bug, if there even is one.
Final Thoughts
Despite these issues, I am very satisfied overall with Debian so far; its reputation as a very solid and stable distribution has proven itself in my (limited) experience with it. It no longer feels like some strange, alien take on Linux as it seemed to me so long ago. I’m not sure if it’s because I’m that much more experienced in Linux or if it’s that Debian has changed that much in 20+ years, but either way, I feel a lot more comfortable with it now. These Pis don’t require much work now that they’re set up properly and running everything I want them to run..though I’m sure I still have a lot more to learn about “The Debian Way”..and I’ll see how it goes if/when I want to add new software and/or make any significant changes.
(All art on this page by Jasmine at myfurryart.com)