Upgrading jails in FreeNAS has always been a little problematic. It’s slightly less so now, but is still fraught with danger and never to be taken lightly. This was going to be the story of one such upgrade (my SSL-Proxy jail) but thinking about it over the past few weeks has turned it into more of an overview of the different ways of upgrading jails.

Although I think iocage was introduced into FreeNAS at some point during the 9.10 train, I didn’t use it until I’d upgraded to 11.1, which I blogged about here. Until the introduction of iocage, it wasn’t possible to upgrade the version of FreeBSD running inside the jail created with the Warden jail manager.

This was quite a big problem as it meant that over time the version of FreeBSD became end-of-life (EOL) and the applications running in the jail would eventually stop updating. There was a period of time where it was possible to override the warning messages, but this increased the chances of breaking something which at some point was inevitable if you wanted to try and stay up to date.

The only solution was to build a new jail and transfer as much configuration as possible or keep files and databases outside of the jails to make this as painless as possible. The other option was not to upgrade, although eventually, this creates problems of its own.

Iocage changed the game and made it possible for the version of FreeBSD running inside a jail to be upgraded to match the version running on FreeNAS, so once you’d upgraded FreeNAS to a new version of FreeBSD you could update the jails

To be fair, this isn’t a regular event as FreeNAS often uses the same base version of FreeBSD for a couple of years. 11.1 was the first to use FreeBSD 11 and that was released back in 2018. FreeNAS doesn’t move to the latest version of FreeBSD either, so the current release 11.2-U7 is actually running FreeBSD 11.2 which itself is now EOL!

11.2-U7 did introduce the ability to run a newer version of FreeBSD inside a jail, specifically FreeBSD 11.3. It’s not possible to go beyond this to the current branch of FreeBSD (12) and this won’t be in FreeNAS or available to the jails until FreeNAS 12 which isn’t planned for another 6 months.

11.3 is still supported though, so upgrading the jails to this version gets around the issues caused running 11.2 and having to force upgrades as it was EOL. Fortunately, it’s a relatively painless process, although requires some patience as it can take between 45-60 minutes for each jail. I have 11, so it took some time.

I’d already done this for all of my jails when I upgraded to 11.2-U7 from U5, although upgrading the version of FreeBSD does not update any of the packages running inside the jail. This is another step and can be more fraught with danger than upgrading the FreeBSD version!

When it comes up updating the jail packages, there are a number of things to take into account.

And before you consider any of them, I would always recommend taking a manual Snapshot of the jail and any external database mounts, so if anything does go wrong and breaks, you can simply rollback to before you made any changes!

The next important thing is the package update frequency. Now, this is a relatively new one on me and I’m not sure when this was introduced, but by default FreeNAS and any jails created in FreeNAS follow a Quarterly package release. Whilst this does provide a level of stability, it also means you’re at least 3 months behind the Latest release schedule.

So if you’d noticed a package had been updated, maybe from an alert you’d set at www.freshports.org but then you’d run pkg update and pkg upgrade and you didn’t get the lastest release listed, this is probably why!

Luckily, changing to the Latest schedule is pretty easy – from the Jail command line:

mkdir -p /usr/local/etc/pkg/repos
nano /usr/local/etc/pkg/repos/FreeBSD.conf

copy/paste following into FreeBSD.conf

FreeBSD: {
   url: "pkg+http://pkg.FreeBSD.org/${ABI}/latest"
}

After that, pkg update and pkg upgrade should fetch and install the latest packages installed in the jail.

So this is pretty much the standard methods of updating jails, both in terms of the jail ‘OS’ and the applications within the jails. But there are as few others.

Many applications or services have built-in update routines, and generally speaking, these are the easiest and least dangerous to update. For example, Nextcloud, WordPress and LimeSurvey all have their own, slightly different but relatively painless ways to update

WordPress is probably the easiest, as by default it will update itself, although that might not always be desirable. To be fair, WordPress is pretty well tested and I’ve never had s problem with a release itself breaking. Maybe a plugin might be incompatible for a short while, and that’s why I normally leave it a couple of weeks before updating, but otherwise, it’s pretty safe.

Nextcloud is also pretty straight forward to update using the built-in functionality. There’s sometimes a little delay between the release date and it being available in the updater, but I don’t mind this and actually prefer not being one of the early testers anyway. Everyone now and again I need to restart the updater as it stops at one of the steps, but it always works a second time and rarely takes more than a few minutes.

It’s also worth mentioning that the Apps within Nextcloud can also be updated very easily from within the App, although from time to time they will need dependancies updating within the jail itself, so they may require a specific version of PHP or MySQL

The LimeSurvey Comfort Update is also very slick, although it should be as it’s a paid feature and one of the ways LimeSurvey generates some income from their open-source software. I think it cost about $49 for 6 months, which gives you 60 updates. I used less than 10, and they don’t carry forward, but if you are using the software it certainly makes updating much easier.

Finally, a special mention to Home Assistant, or more specifically Troy’s method I used for installing it which can be found here on the Home Assistant Forum. It includes some bash scripts to update, and a simple menu to run them from. Not really built into the App, but it makes updating things pretty simple, and the support from Troy when problems occur or you have a specific update requirement (like Python from 2.7 to 3.7) is outstanding!

With everything else, you’re at the mercy of pkg update and upgrade, which gets me back to the example, which I’d almost forgotten about! I’d upgraded the ssl-proxy jail I use for generating all of my SSL certificates and then redirecting services connecting externally on port 443 to the appropriate jail or VM using NGINX as a reverse proxy. I’d forgotten that’s also where I’ve also consolidated all the weblogs and installed AWstats so I have some information about how my server is being accessed, as everything externally comes through this jail.

To be fair, I didn’t have too many issues with Certbot other than tweaking the scripts I use to automate the renewals. I also had to comment out some of the NGINX configurations which were depreciated in the latest version, but it completely broke AWstats and didn’t like the GeoIP dependency that it uses to try and provide country information based on the IP addresses accessing my services.

I still haven’t got around to fixing it, as it’s not really a deal-breaker and in all honesty doesn’t provide that much useful info. I’m using Google Analytics for that, but I will look to try and fix it at some point, and when I do it might be worth its own blog!