Thoughts on FreeBSD
It had been quite a while since I’d tried out FreeBSD, and I figured, why not give it a fresh look for my latest project? This time, I liked what I saw…for the most part.
Background
I’ve known about FreeBSD since the 90s and I’ve tried it out from time to time ever since. I still have a FreeBSD 4.1 CDROM set that I ordered a long time ago from Walnut Creek (link to archive.org) that I likely never installed on a system. Between then and now, I made FreeBSD virtual machines and kicked their tires every so often. Yet, it never felt ‘right’ to me, for whatever reason.
In January 2023, when I decided to make this blog a reality instead of one of the many theoretical possibilities running around in my head, I could have used almost any Linux distro (especially an RHEL-like one) and it would have been quick and easy to set up. But when I thought about it, I figured I could use FreeBSD instead and give myself an educational challenge. I managed to succeed in both facets, but it did take an enormous amount of time and effort, which I’ll keep in mind for similar challenges in the future. Still, it was a lot of fun (for the most part) and I’ll consider FreeBSD going forward for the right use cases.
Compared to its past iterations, FreeBSD 13 feels a lot more comfortable to me. I’m not sure if it’s due more to changes in the OS itself or if I’ve become more open-minded/capable of navigating its nuances and not trying to force it to use any of the “Linux-isms” I’ve learned.
Getting Started and Packet Filtering
Fortunately, the cloud provider had a 13.1 instance available (without having to manually create and upload one). After deploying it, I was off and running (after doing all the non-OS specific work like registering the domain and setting up DNS, outbound email, and Let’s Encrypt).
The first order of business was to set up FreeBSD’s version of pf, which I had prior experience with from when I used OpenBSD for firewalling and routing. Immediately after I deployed it, my instance was getting bombarded with illicit ssh attempts so I set up an allow table for inbound ssh traffic while blocking the rest. For example:
# Set a variable for the internet-facing network interface
ext_if = "xl0"
# Fill this in with IP addresses and/or network ranges
table <allow> { 10.10.10.0/23, 192.168.34.0/24, 172.16.1.1 }
# Scrub and block everything by default, then add exceptions
scrub in all
block all
# Pass in from the allow table to ssh
pass in quick on $ext_if proto tcp from <allow> to port 22
While port 22 is still going to get a lot of bad actors knocking on it, that door isn’t going to open for them.
This method has the issue of only being able to ssh from a limited set of IPs, which can be a problem if you need to access your instance from various places. There are alternatives to using an allowlist, such as port knocking, setting up automatic blocking for IPs that make too many attempts, and using a different port for the ssh daemon. These are all viable, depending on your use case(s), but I prefer an allowlist, as I can use a software-defined mesh VPN such as Zerotier or Tailscale to get access from anywhere to any of my systems.
(Always use a public/private keypair, disable password authentication, and don’t allow any remote logins as root.)
I’ll get back to pf.conf later, when ipv6 enters the room.
doas and sudo
I’ve been using sudo for…well, as long as I can remember using UNIX and Unix-like systems (roughly the mid-90s or so) and for my entire IT career. I initially heard about doas when it was released as a replacement for sudo in OpenBSD 5.8, as an example of an OpenBSD subproject that was created from scratch as an alternative to an existing standard/protocol/program. Others include CARP, LibreSSL, pf, tmux, and, most famously, OpenSSH. As is also the case with most of this software, configuration/management of doas is simpler than sudo and works quite well on an instance between one and a few users. (However, for more complicated setups/infrastructures/using LDAP to store sudo configuration/etc., sudo is the only reasonable choice.)
I also want to note that there are two primary ports of doas used outside of OpenBSD; doas (which I’ll refer to as doas-js after the author to reduce confusion) and OpenDoas. The primary difference between them is that OpenDoas allows for the use of the persist keyword, which allows for doas to be used without a password for a time, similar to sudo, while doas-js does not; the author explains their reasoning in this comment. Not having this available when I’m so used to it from sudo was jarring to use at first, but I’ve managed to adjust. As mentioned in the comment, doas -s can be used if several commands need elevated privileges or when troubleshooting.
After installing doas, setting it up is as simple as creating its config file (usually at /usr/local/etc/doas.conf) and putting in something like
permit jsilverfox as root
In sudo, it would be
jsilverfox ALL=(ALL:ALL) ALL
To allow a user in doas to run commands without needing a password, the configuration is
permit nopass jsilverfox as root cmd /full/path/to/command
The equivalent in sudo is
jsilverfox ALL=(ALL:ALL) NOPASSWD: /full/path/to/command
True, the syntax isn’t that much simpler, but it seems a bit more like something a person would say instead of…whatever sudo has. I also like that doas doesn’t have the lecture feature that I see often in my work:
We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:
#1) Respect the privacy of others.
#2) Think before you type.
#3) With great power comes great responsibility.
doas also has doasedit and vidoas as the counterparts of sudoedit and visudo, respectively. These are all useful tools for editing files with elevated privileges and editing the configuration file with syntax checking.
Unfortunately, I recently tried editing doas.conf without vidoas and found out how much of a pain it is to fix a system with a broken doas.conf and no sudo installed. I had to boot into single user mode, set / to read-write, fix doas.conf, and then reboot.
(Here’s the procedure to do so, should you ever find yourself in a similar situation):
# For UFS
fsck -p
mount -u /
mount -a # To mount every filesystem in /etc/fstab, if needed
# For ZFS
zfs mount -a
zfs readonly=off zroot/ROOT/default # Given the default ZFS poolname
Overall, I like doas so far and might find use cases for it on Linux. I’m still going to default to sudo, though (decades of muscle memory), and I know I’ll type it frequently on FreeBSD before I remember that I need to use doas instead.
Packages, ports, and poudriere
I’ve been managing and building packages for a long time, and I’ve always liked how FreeBSD did it. The base OS is separated as much as possible from third-party software, which can be built from source or installed as binary packages.
Every Linux distro I’ve used doesn’t have such a visible separation between essential system packages and any add-ons. For example, trying to remove the glibc package from Linux should result in an error as almost every package depends on it directly or indirectly. (There should also be a message about how doing so would be cumbersome at best to recover from and serves no purpose whatsoever, but if you want to take a blunt force weapon to your system, feel free.) FreeBSD’s equivalent, libc, is part of the base system and not a package. (Yes, that makes removing it even more difficult to restore the system…but that’s not the point.)
In FreeBSD, precompiled packages/port trees can be obtained from either the quarterly or latest repositories, depending on how relatively stable or current you want your instance to be. Either way, keeping your system updated (especially any security patches) is always a best practice. For FreeBSD admins, the site https://www.freshports.org/ is near-indispensable for staying current and getting information on the thousands of ports/packages available.
Installing packages is straightforward, much like modern Linux distros.
doas pkg install package.name
The pkg command also handles searching, listing, and removing packages.
Managing ports, though…as far as I know, ports-mgmt/portmaster is the preferred tool for doing so, should you go that route. I tried a few ports at first with this particular system, but sorting through several options for every single dependency of a given package got old quickly. For those who like to customize every aspect of their software, this is perfect for them. (For the record, I was never into Gentoo Linux during its heyday, though the idea of having an optimized system was certainly tempting..)
While writing this post, I ran into a situation where the port had a security fix, but the package for git hadn’t been updated just yet. (To my knowledge, this happens naturally with FreeBSD packages; ports are almost always going to be more recent, barring a vulnerability that needs immediate attention.) Unfortunately, building git from the ports tree required compiling 84 (!) other ports. I balked at that rather quickly as having to manually select the options for each port was not appealing in the slightest.
I started looking for alternatives, recalling a package building system for FreeBSD that I couldn’t remember the name of. Then, it either came to me in an instant or I happened to stumble upon it while aimlessly searching. Knowing me, it was most likely the latter. (At this rate, I’m not going to remember anything as an old fox..)
poudriere is the name of the tool..it makes me think of poutine for some reason. Instead of a combination of french fries, cheese curds, and gravy, poudriere will not rearrange your arteries into a recreation of Chicago rush hour but it will make it much simpler to build packages from ports, whether one or many or some point in-between. The FreeBSD Handbook has a good introduction to this utility and there’s enough material about this application for a whole article about it. I’ll just summarize what I did, instead.
First, I installed the package and then edited the config file at /usr/local/etc/poudriere.conf. It can be used with ZFS, but I didn’t need it for this case, so I disabled it and pointed FREEBSD_HOST to https://download.FreeBSD.org.
Then, I ran these commands to set up the jail for the build and its ports tree:
# This sets up a jail called git-jail in /usr/local/poudriere/jails/git-jail based on 13.1.
# The jail is then updated to the most recent release.
doas poudriere jail -c -j git-jail -v 13.1-RELEASE
# Checks out a ports tree called local for the jail (/usr/local/poudriere/ports/local)
doas poudriere ports -c -p local -m git+https
This will take some time as the jail and its port tree are populated and updated. After this is done, however, building packages is no trouble at all:
# This was the only port I wanted to build, but more can be added to the same file as desired.
echo "devel/git" >> pkglist
doas poudriere bulk -f pkglist -p local -j git-jail
For a complicated port like git, this is going to take a long time to build everything but no intervention is needed. (I highly recommend putting any poudriere builds into tmux/screen to keep it going in case of a disconnection.) As the build interface notes, ctrl+t will provide a status update. There’s also an index.html that can be used (if you configure a web server for it) to watch the status from a browser.
After it’s all finished, the new package(s) will be in /usr/local/poudriere/data/packages/git-jail-local/All (in this case), along with packages for all the build dependencies. All I had to do was run
doas pkg install /usr/local/poudriere/data/packages/git-jail-local/All/git-2.39.2.pkg
And git was patched (..for now). For future builds, I just need to update the jail/ports tree and then start a build.
# Patch/update the jail.
doas poudriere jail -u -j git-jail
# Update the ports tree.
doas poudriere ports -u -p local -v
I had such an excellent experience with poudriere that I decided to remove /usr/ports and put a job in /etc/cron.d to update the poudriere tree, as I can use it now to build any ports I need to. It also made me more interested in FreeBSD’s jail mechanism, a light virtualization layer that’s complicated enough to require its own book. I had never used a jail before, but the concept is close enough to Linux containers that I can understand the basic premise of it more easily.
(Notes: if /usr/ports is removed, be sure to set DISTFILES_CACHE in poudriere.conf to another directory. Also, I figured out after doing that massive build that I could have used devel/git:tiny instead and it wouldn’t have taken nearly as long…heh. Various ports have flavors that can be applied to shrink the size of the port and/or build it against different toolkits; see https://docs.freebsd.org/en/books/porters-handbook/flavors/ for more details.)
As a final note in this section, one thing to be wary of is excessive customization of ports. This can lead to a case of a package looking for a certain dependency that it thinks is there (as the port for it is installed) but it doesn’t exist because that particular aspect was deselected as an option. That way lies dependency madness, also known as DLL hell and using rpms without a package manager.
Writing an rc.d script
Typically, having to write an rc.d script in FreeBSD is an unlikely possibility, as the great majority of packages and ports will provide them for all of their daemons. In the case of sysutils/loki, though, it did not (as of this writing) provide a script for promtail, which is used to feed logs into loki. It took a lot of trial and error (and irritated fox noises), but I managed to put a working shell script together.
#!/bin/sh
# PROVIDE: promtail
# REQUIRE: LOGIN
# KEYWORD: shutdown
# Add the following line to /etc/rc.conf to enable promtail
# promtail_enable="YES"
# promtail_enable (bool):
# Set it to YES to enable promtail
# Set to NO by default
# promtail_user (string):
# Set user that promtail will run under
# Default is "loki"
# promtail_positions (string):
# Set full path to positions file
# Default is "/var/db/promtail/positions.yaml"
# promtail_config (string)
# Set full path to config file
# Default is "/usr/local/etc/promtail.yaml"
# promtail_args (string)
# Set additional command line arguments
# Default is ""
. /etc/rc.subr
After the hashbang, the PROVIDE line is used to describe which dependencies this program provides. Most often, it will just be the name of the program itself, though others can be specified as needed. This isn’t needed in systemd, as, in this case, “promtail.service” would automatically be used for this purpose.
REQUIRE is used for any dependencies that this daemon might require, though, in most cases, LOGIN will be sufficient. This is used in much the same way as Wants or Requires in a systemd unit file. KEYWORD is used for rcorder, which attempts to execute scripts in a desired order. Setting this to shutdown will also suffice for the majority of rc.d scripts, which will shut down the daemons in the reverse order in which they were started.
The comments below are to inform an admin how to enable the daemon along with other parameters that can be set. As mentioned, adding the line promtail_enable=“yes” to /etc/rc.conf will configure the daemon to start after each boot. In FreeBSD, this also allows the start, stop, status, and restart commands to be used. If a daemon is not enabled in rc.conf yet, onestart, onestop, onestatus, and onerestart, etc., need to be used instead.
Also, the command
doas sysrc promtail_enable="yes"
will automatically add this (and any other) lines to /etc/rc.conf.
. /etc/rc.subr has every necessary function for an rc.d script, so be sure to include it or your script will break in various non-mysterious ways.
name=promtail
rcvar=promtail_enable
load_rc_config $name
: ${promtail_enable:="no"}
: ${promtail_user:="loki"}
: ${promtail_group:="loki"}
: ${promtail_config:="/usr/local/etc/promtail.yaml"}
: ${promtail_positions:="/var/db/promtail/positions.yaml"}
: ${promtail_logfile:="/var/log/loki/promtail.log"}
pidfile="/var/run/${name}/${name}.pid"
required_files="${promtail_config}"
procname="/usr/local/bin/promtail"
command="/usr/sbin/daemon"
command_args="-p ${pidfile} -t ${name} -o ${promtail_logfile} \
${procname} \
-config.file=${promtail_config} \
-positions.file=${promtail_positions} \
${promtail_args}"
The first lines set the name of the daemon and what rc.conf will look for to enable it. load_rc_config is a function from /etc/rc.subr that will source the variables in the script for use later on. The next lines set the default variables, which can be overwritten in /etc/rc.conf.
pidfile is self-explanatory, along with required_files; this should be set for config files along with anything else that’s essential to run this daemon. procname points to the path where the executable is installed, and command is the program used to daemonize, which will almost always be /usr/sbin/daemon.
command_args describes how the daemon will be run, though the first line is used by /usr/sbin/daemon to set the pidfile with -p, the title with -t, and the output/log file with -o. Everything after ${procname} will be the command itself.
promtail_start_precmd()
{
if [ ! -d "/var/run/${name}" ]; then
install -d -m 0750 -o ${promtail_user} -g ${promtail_group} "/var/run/${name}"
fi
if [ ! -d "/var/log/loki" ]; then
install -d -m 0750 -o ${promtail_user} -g ${promtail_group} "/var/log/loki/}"
fi
if [ ! -d "/var/db/promtail" ]; then
install -d -m 0750 -o ${promtail_user} -g ${promtail_group} "/var/db/${name}"
fi
}
run_rc_command "$1"
promtail_start_precmd() is a function that, in this case, is used to create directories for the pid file, promtail’s log, and the position yaml file, the latter of which is used to keep track of how far promtail has read in a given log. I realized that I hardcoded /var/run and /var/db while I was writing this post, though anyone who needs alternative directories for those files is also savvy enough to modify the rc.d script accordingly.
The last line will give the argument from $1 (stop/start/restart/etc.) to the run_rc_command function from /etc/run.subr.
For further information, see https://docs.freebsd.org/en/articles/rc-scripting/ and https://people.freebsd.org/~olivierd/porters-handbook/rc-scripts.html
I know that’s a lot to go over, but it was quite educational for me to figure all this out, as rarely as I’ll likely need to use it. I like how enabling and configuring command line arguments for daemons is all in /etc/rc.conf, instead of scattered in various directories under /etc, as is typical in Linux. systemd doesn’t have a central place to put these arguments (..yet..)
sendmail and postfix
These next two sections will go over the aspects of FreeBSD that…weren’t as much fun for me. First, I’ll look at sendmail, which is the default MTA in FreeBSD 13. It’s been a long time since I had to do it, but I have suffered through more than enough instances of troubleshooting m4 files while administering sendmail for one career.
It’s a…functional default, at least, but there was no way I wanted to set up outbound mail on it when postfix is readily available. I know the FreeBSD project likely has its reasons, but to me, sendmail feels like a dissonant echo of the past in an otherwise modern Unix-like OS.
To replace sendmail with postfix, first install the postfix_sasl package or set the port up however you need (I recommend the sasl package for its versatility). Then, run
doas sysrc sendmail_enable="no"
doas sysrc sendmail_submit_enable="no"
doas sysrc sendmail_outbound_enable="no"
doas sysrc sendmail_msp_queue_enable="no"
Next, stop sendmail and start postfix:
# If you remember using service on Linux in the pre-systemd days, this works exactly the same way.
doas service sendmail stop
doas service postfix start
Then, enable postfix on boot:
doas sysrc postfix_enable="yes"
The final part of this setup is to edit /etc/periodic.conf and set it to:
daily_clean_hoststat_enable="no"
daily_status_mail_rejects_enable="no"
daily_status_include_submit_mailq="no"
daily_submit_queuerun="no"
This ensures that these sendmail-specific routines won’t run when they don’t need to.
Also, edit /etc/mail/aliases and then create /etc/aliases.db by using /usr/local/bin/newaliases.
A reboot is a good idea after a change like this to make sure that everything is working as expected and you won’t get an unpleasant surprise if the system reboots unexpectedly. I know this from experience, especially on servers with gargantuan uptimes that haven’t been rebooted since “Gangnam Style” was popular. Such machines are dangerous, but that’s another article entirely.
Finally, postfix needs to be configured for whatever it’s to be used for, but none of that is FreeBSD-specific. The system should be free of sendmail completely now, though any MTA can be used if postfix doesn’t fit your needs/use case. In either case, https://docs.freebsd.org/en/books/handbook/mail/#mail-changingmta is the official reference for switching from sendmail to anything else.
Now comes the part that took the most time of all, more than the rest put together.
Trying to get an internet-routable ipv6 address up.
IPV6
There are far more informative guides out there as to how ipv6 works, how to configure it, what best practices to use, etc., etc. Here, I’m going to talk about my FreeBSD experience with it and why it’s something that hopefully will be a lot easier to set up in a future release.
The main issue is that the dhcp client in the base, dhclient, does not support ipv6. So, in a network where stateless address auto configuration is not available, FreeBSD cannot get a global address without the use of a port or package. (And even then, SLAAC configuration is not straightforward, from what I could tell.)
After extensive searching, troubleshooting, many failures, and angry fox noises, I finally came across a page with the solution: Thank you, whoever wrote this. This involves installing the net/dual-dhclient package, editing /etc/rc.conf like so:
# Substitute em0 with the appropriate interface
dhclient_program="/usr/local/sbin/dual-dhclient"
ifconfig_em0="DHCP"
ifconfig_em0_ipv6="inet6 DHCP accept_rtadv"
and then modifying /usr/local/sbin/dual-dhclient to the following:
#!/bin/sh
/usr/local/sbin/dhclient "$@"
/usr/local/sbin/dhclient -6 -nw -D LL "$@"
(I’m not a fan of changing already packaged files, which adds to my irritation with this procedure.)
Set /usr/local/etc/dhclient.conf to only have
send fqdn.fqdn = gethostname();
Finally, kill any dhclient processes, run dual-dhclient in a separate terminal, and then use ifconfig to check your network addresses. If configured properly, the interface should now be using ipv4 and ipv6. Thanks to the changes in rc.conf, this should run on every reboot (test this to be sure) and not require enabling a separate daemon.
Yet, that’s not all. To make sure that this address is renewed when the dhcpv6 lease runs out, pf.conf needs some changes:
table <ipv6_net> { your:inbound:ip6:network::/64 }
pass in quick on $ext_if proto tcp from <ipv6_net> to port 22
This allows for limiting ssh over ipv6, similar to above. Then comes a lot of ipv6-specific lines:
# Neighbor Discovery Protocol (NDP) (types 133-137):
# Router Solicitation (RS), Router Advertisement (RA)
# Neighbor Solicitation (NS), Neighbor Advertisement (NA)
# Route Redirection
icmp6_types="{ toobig, echoreq }" # packet too big, echo request (ping6)
icmp6_types_ndp="{ 128, 133, 134, 135, 136, 137 }"
pass in quick on $ext_if inet6 proto ipv6-icmp icmp6-type $icmp6_types keep state
pass in quick on $ext_if inet6 proto ipv6-icmp from any to { ($ext_if), ff02::1/16 } icmp6-type $icmp6_types_ndp keep state
# Allow the ISP or local router to send DHCPv6 packets
pass in on $wan inet6 proto udp from fe80::/10 port dhcpv6-server to fe80::/10 port dhcpv6-client no state
I’m not going to try to explain this in detail, as…I don’t know exactly how it works.
For the sake of comparison, I created an AlmaLinux 9.1 instance on the same cloud provider with the same ipv6 setup. It was able to get a global ipv6 address without any intervention needed on my part.
That’s what, in my opinion, dhclient on FreeBSD needs to be able to do. Granted, from what I’ve read, it might be easier to set up SLAAC/DHCPv6 when doing a media installation of FreeBSD. Yet, if it’s not readily available in a cloud image, that will deter usage for those who require ipv6 and will instead go for the easier solution, especially when pressed for time.
Overall, I’m glad it’s working now, though I don’t like how much duct tape I had to use. Hopefully, future releases will fix/improve upon this and I’ll be able to put that tape to use elsewhere.
Ends and Odds and a Conclusion
To get the setfacl command to work, ACLs need to be enabled on the root (or whichever) filesystem. For a non-root filesystem, unmounting it and running tunefs -a enable /path/to/fs should work. For the root filesystem, a reboot will be needed immediately, so only do this when it’s safe to.
# Best to switch to the root user before doing this.
mount -fur / # This is not meant to be a reference...really! ^^
tunefs -a enable /
reboot
This re-mounts / as read-only, enables ACLs, and then reboots. If it works, you should see the output
tunefs: POSIX.1e ACLs set
tunefs: file system reloaded
along with the string “acls” after running the mount command.
In pf.conf, the default block directive can be set as block log all to log every blocked packet to /var/log/pflog. This file can be viewed with tcpdump or any software that can read pcap files. It’s also possible to use tcpdump or similar software to watch the blocked packets in realtime:
doas tcpdump -n -e -ttt -i pflog0
Marvel as various IPs slam their heads into the brick wall you’ve put in place…and then keep doing it ad absurdum…
This is a useful command I put together for seeing which installed software came from ports at a glance:
pkg query -a '%R %o' | sort | grep unknown-repository
I found pkg autoremove useful for uninstalling packages that became orphaned for whatever reason. This is similar to apt autoremove, dnf autoremove, and pacman -Qdtq | pacman -Rs - in Arch. (Of course..)
Installing minor updates/security patches is straightforward; freebsd-update fetch followed by freebsd-update install if any updates are available. Running freebsd-update cron in a daily cronjob sent to the admin(s) is the intended way to be notified about minor patches. I set this up and it worked as expected, as I got a mail about the release of FreeBSD 13.1-RELEASE-p7. After a fetch, install, and reboot, everything was good again. (It’ll stay that way unless my curiosity gets the better of me again…but that’s another story…)
As for a conclusion (I was going to get here eventually…), I’m overall satisfied with my experience so far. Yes, it took a considerable amount of work and effort but I learned many new things and it was ultimately worth it. It’s similar to the parts of my job which involve a lot of trial and error to get a procedure/program to work properly. Documenting it as I go, though, makes it much easier to reproduce once I’ve figured it out (and it’s the only way I can remember it these days). I love that kind of work and I’m definitely in the right career for it. (Maybe a bit too much, but that’s also another story…)
For as long as I can remember, FreeBSD’s motto has been “The Power to Serve” and it does that quite well. Take Netflix, for example. Their FreeBSD-based appliances are part of this (likely incomplete) list of products using or based on FreeBSD.
In data centers/general infrastructure, though, FreeBSD currently has a catch-22 problem: it’s hard to get professional FreeBSD experience as an admin because it’s only in a small minority of data centers and trying to add it to an infrastructure is difficult because it’s hard to find someone with that niche expertise. Near the start of my career, Linux was in a somewhat similar situation (also due to how young it was at the time) but once it hit critical mass, that problem disappeared and it’s been snowballing ever since.
While FreeBSD may never break out of its niche despite its many technical merits, it still has its place in the world of computing, obscure as it is. I think I can relate to that.
(For more information about FreeBSD and its software, I highly recommend the series of books written by Michael W. Lucas, who combines strong technical knowledge with a very entertaining writing style.)
(All art on this page by Jasmine at myfurryart.com)