TrueNAS 13.3

Oct 27, 2024 | IT, Server, TrueNAS | 0 comments

Well, it’s been a busy month!

TrueNAS and FreeNAS (as it was named before that) have been one of, if not the, most blogged-about subjects in the last 250 blogs (yes, I made it to 250 and didn’t even notice) over the past 8 years (or at least it will be in November 2024).

I’ve just upgraded from TrueNAS 13.1-U6 to 13.3, which could be the last major release of TrueNAS Core, the FreeBSD-based version that was FreeNAS 9.2.8 when I started using it in 2013. 

Upgrading TrueNAS is usually pretty uneventful, even though it causes me no end of worry for weeks before and sometimes weeks afterwards.  This is one of these updates that I might still be fighting with in months to come!

I’ve already blogged about some of iXsystems focus changes here, with TrueNAS Scale based on Linux being the future direction of travel. What this means for TrueNAS Core is still a little unclear, but hopefully, there are still a few years before that becomes a major issue for me.

After updating TrueNAS (which is built on FreeBSD 13.3 rather than 13.1), the process of upgrading the jails begins.  This updates them to the base version of FreeBSD.  I successfully upgraded one jail from 13.1 to 13.3 and then updated and upgraded the packages (#pkg update & pkg upgrade), but it was only my Airsonic jail, which isn’t really used in anger anymore. I wouldn’t be upset if it broke and I had to destroy it.  

Lots of my jails are web-based applications running on a FEMP stack, or FreeBSD (as the operating system), Nginx (a web server pronounced Engine-X), MySQL (a database server, in my case Mariadb), and PHP (a programming language to process dynamic PHP content).  I’ve kept the jails running by upgrading the operating system between versions of TrueNAS using commands like #iocage upgrade -r 13.3-RELEASE <jailname> and then within the running jail #pkg update && pkg upgrade.  I needed to update PHP separately, which I blogged about here, but the one thing I neglected was the MySQL element.  My Nextcloud, WordPress, and LimeSurvey jails are all running Mariadb103, which, along with Mariadb104, are no longer supported.  The latest version of MariaDB is 10.11, so I’m pretty far behind.

I asked for some help from the TrueNAS forum, and as ever, I wasn’t let down with some excellent advice.  It turns out I can update the jail base to FreeBSD 13.4 on TrueNAS 13.3, so I’ll be doing that and then trying to update Mariadb103 to Mariadb105 as a first step.  I’ve struggled with this in the past and simply rolled back and found a way to update without touching the database.  In hindsight, that wasn’t a great idea, but at least now I have a problem that needs to be solved, and once I’ve solved it for one, it should be easy enough to apply to the others.  

It turned out to be a little more problematic than I thought, although that’s more down to my lack of System Administration skills than the technology. The first rule of updating databases is to ensure the old database has been shut down correctly and not aborted by trying to jump a step ahead and simply install the new one! I’m sure this will be in System Administration 101, which I think I also skipped.

I also learned about something called tmux, which lets you keep a session running on TrueNAS even when your SSH session is lost. How have I made it 10 years without knowing this? To be fair, they have only just removed the built-in shell console, which I tended to use more than the SSH method.

Anyway, I’m going to list some commands I’ve used to upgrade, which is more for me than anyone reading this:

#tmux new -t iocage (starts a tmux session called iocage)

#tmux a (restart the previous tmux session – I’m guessing you have could several running, although I’ve only tried with one)

#iocage fetch -r 13.4-RELEASE (downloads the release)

#iocage upgrade -r 13.4-RELEASE <jailname> (update the jail to the specified release – note, there are various ‘y’ prompts to confirm the process here, which is where tmux comes in handy.  There is also a step which appears to stop with a : prompt, and this is skipped by entering q several times)

#iocage restart <jailname> (restart jail after upgrading)

#iocage console <jailname> (console session into jail)

#pkg info (lists all installed packages and versions)

#service mysql-server stop (clean stop of database server)

#pkg install mariadb105-server (this will remove the old database server and then install the new one – 10.5 – along with any dependencies)

#service mysql-server start (this should start the new database server; check with #service mysql-server status)

#mysql_upgrade -u root -p (to upgrade the databases)

#pkg update && pkg upgrade (this will upgrade all of the other packages in the jail)

I’m not going to pretend I understand exactly what changed between mariadb103 and 105, but the main issue I encountered was the UNIX socket connection, which changed from /tmp/mysql.sock to /var/run/mysql/MySQL.sock. These changes were in the /usr/local/etc/mysql folder and ./conf.d folder.

So, while the database started and the upgrades worked, none of my web applications would start, as the configuration to the database was wrong.  So far I’ve fixed Nextcloud (/usr/local/www/nextcloud/config/config.php where you set ‘dbhost’ => ‘127.0.0.1’,) and WordPress (/usr/local/www/wordpress/wp-config.php where you set define(‘DB_HOST’, ‘127.0.0.1’);)

This got everything back up and running but after a reasonably stressful afternoon and several snapshot rollbacks where I’d broken things and had no idea what to do.  Thank goodness for TrueNAS snapshots and the incredibly helpful people who use their forum.  It’s still one of the best online communities I’ve come across, even though I visit less frequently since the forum platform changed.  I need to try and fix that!