TrueNAS 2025

Well, I didn’t manage to blog every month in 2024, but I did manage more than 12, so I think that’s not a bad effort, given the two-year break! I know that I have another TrueNAS blog to start this year (after not blogging for so long) here and here, but...

TrueNAS 13.3

Well, it’s been a busy month!

TrueNAS and FreeNAS (as it was named before that) have been one of, if not the, most blogged-about subjects in the last 250 blogs (yes, I made it to 250 and didn’t even notice) over the past 8 years (or at least it will be in November 2024).

I’ve just upgraded from TrueNAS 13.1-U6 to 13.3, which could be the last major release of TrueNAS Core, the FreeBSD-based version that was FreeNAS 9.2.8 when I started using it in 2013. 

Upgrading TrueNAS is usually pretty uneventful, even though it causes me no end of worry for weeks before and sometimes weeks afterwards.  This is one of these updates that I might still be fighting with in months to come!

I’ve already blogged about some of iXsystems focus changes here, with TrueNAS Scale based on Linux being the future direction of travel. What this means for TrueNAS Core is still a little unclear, but hopefully, there are still a few years before that becomes a major issue for me.

After updating TrueNAS (which is built on FreeBSD 13.3 rather than 13.1), the process of upgrading the jails begins.  This updates them to the base version of FreeBSD.  I successfully upgraded one jail from 13.1 to 13.3 and then updated and upgraded the packages (#pkg update & pkg upgrade), but it was only my Airsonic jail, which isn’t really used in anger anymore. I wouldn’t be upset if it broke and I had to destroy it.  

Lots of my jails are web-based applications running on a FEMP stack, or FreeBSD (as the operating system), Nginx (a web server pronounced Engine-X), MySQL (a database server, in my case Mariadb), and PHP (a programming language to process dynamic PHP content).  I’ve kept the jails running by upgrading the operating system between versions of TrueNAS using commands like #iocage upgrade -r 13.3-RELEASE <jailname> and then within the running jail #pkg update && pkg upgrade.  I needed to update PHP separately, which I blogged about here, but the one thing I neglected was the MySQL element.  My Nextcloud, WordPress, and LimeSurvey jails are all running Mariadb103, which, along with Mariadb104, are no longer supported.  The latest version of MariaDB is 10.11, so I’m pretty far behind.

I asked for some help from the TrueNAS forum, and as ever, I wasn’t let down with some excellent advice.  It turns out I can update the jail base to FreeBSD 13.4 on TrueNAS 13.3, so I’ll be doing that and then trying to update Mariadb103 to Mariadb105 as a first step.  I’ve struggled with this in the past and simply rolled back and found a way to update without touching the database.  In hindsight, that wasn’t a great idea, but at least now I have a problem that needs to be solved, and once I’ve solved it for one, it should be easy enough to apply to the others.  

It turned out to be a little more problematic than I thought, although that’s more down to my lack of System Administration skills than the technology. The first rule of updating databases is to ensure the old database has been shut down correctly and not aborted by trying to jump a step ahead and simply install the new one! I’m sure this will be in System Administration 101, which I think I also skipped.

I also learned about something called tmux, which lets you keep a session running on TrueNAS even when your SSH session is lost. How have I made it 10 years without knowing this? To be fair, they have only just removed the built-in shell console, which I tended to use more than the SSH method.

Anyway, I’m going to list some commands I’ve used to upgrade, which is more for me than anyone reading this:

 

#tmux new -t iocage (starts a tmux session called iocage)

#tmux a (restart the previous tmux session – I’m guessing you have could several running, although I’ve only tried with one)

#iocage fetch -r 13.4-RELEASE (downloads the release)

#iocage upgrade -r 13.4-RELEASE <jailname> (update the jail to the specified release – note, there are various ‘y’ prompts to confirm the process here, which is where tmux comes in handy.  There is also a step which appears to stop with a : prompt, and this is skipped by entering q several times)

#iocage restart <jailname> (restart jail after upgrading)

#iocage console <jailname> (console session into jail)

#pkg info (lists all installed packages and versions)

#service mysql-server stop (clean stop of database server)

#pkg install mariadb105-server (this will remove the old database server and then install the new one – 10.5 – along with any dependencies)

#service mysql-server start (this should start the new database server; check with #service mysql-server status)

#mysql_upgrade -u root -p (to upgrade the databases)

#pkg update && pkg upgrade (this will upgrade all of the other packages in the jail)

 

I’m not going to pretend I understand exactly what changed between mariadb103 and 105, but the main issue I encountered was the UNIX socket connection, which changed from /tmp/mysql.sock to /var/run/mysql/MySQL.sock. These changes were in the /usr/local/etc/mysql folder and ./conf.d folder.

So, while the database started and the upgrades worked, none of my web applications would start, as the configuration to the database was wrong.  So far I’ve fixed Nextcloud (/usr/local/www/nextcloud/config/config.php where you set ‘dbhost’ => ‘127.0.0.1’,) and WordPress (/usr/local/www/wordpress/wp-config.php where you set define(‘DB_HOST’, ‘127.0.0.1’);)

This got everything back up and running but after a reasonably stressful afternoon and several snapshot rollbacks where I’d broken things and had no idea what to do.  Thank goodness for TrueNAS snapshots and the incredibly helpful people who use their forum.  It’s still one of the best online communities I’ve come across, even though I visit less frequently since the forum platform changed.  I need to try and fix that!

 

I Loved FreeNAS

This blog is a bit of a con, as I drafted it to post on the iXsystems forum in the hope that I might get some ideas about what to do with my TrueNAS server. I’ve posted it below, but I still haven’t added it to the forum!

 

I loved FreeNAS, and since first installing it over ten years ago, I have learned so much about hosting and maintaining my own micro data centre, much of it from the old forums.

The death of the old forum and the focus on TrueNAS Scale have been quite disheartening, and I don’t know where I go from here. Obviously, everything is still working, so I don’t have to make any immediate decisions, but I feel I should at least be thinking about what the future holds and how I might get there.

I was hoping that if I described how I was currently using TrueNAS Core, I might gain some of the forum’s incredible wisdom to help reinvigorate my love for all things Free to TrueNAS.

For background, I started this journey running an HP N40L Microserver and about 8TB of RAW storage.  That was upgraded to an NL54 and 16TB, and then I built my own server in a Fractal Define R5 case with an Intel Xeon E3 and Supermicro motherboard, ECC RAM and 36TB storage.  See, I was paying attention!  This was upgraded about 4 years ago using the same case, an updated Xeon and Supermicro motherboard, 64GB ECC RAM and 64TB storage, with some SSDs for jails and VMs.  I still have the old one and replicate to it monthly (it used to run 24×7 until the cost of electricity went through the roof), so I have something I could play around with using Scale.

Obviously, my main use is storage.  I have two pools.  This main pool is 8x8TB RAIDZ2 WD HDD which is currently running about 75% (I know, too high really but I was considering upgrading the 8TB drives to 16TB over a few months).  The second pool is 2x1TB Mirror Crucial SSD which is currently running about 65%. The main pool is predominantly long-term storage, and the second pool is for the jails and VMs and associated databases.

I know Scale will do all the storage stuff, so that’s not my worry.  I’m worried about the jails and virtual machine I have running lots of externally facing services accessed through NGINX Proxy Manager, now running on a separate Raspberry Pi (it was running in a jail for a few years)

I have six jails of varying degrees of importance.  My Nextcloud jail is probably the most important and is used by family and a small charity I was a Trustee for.  The files and database are in separate datasets, and I know this could be run on Scale, but moving it and breaking it scares me!  I also have an emby jail used by around 15 family members and friends.  It has the emby_server folder mounted to a separate dataset, so again moving should be possible.  I’m assuming it is possible to mount datasets into containers or VMs similarly to jails, but not whether the structure of these will be the same cross-platform?  I also have a WordPress jail, which used to run my company website, but it’s now just my personal blog.  It’s a similar set-up with a dataset stored outside the jail.  I have three other jails running OpenVPN (so I can connect back to my network remotely), Limesurvey (running small surveys, which could be lost) and Airsonic (an old music streaming service, which has pretty much been replaced by emby).

I then have some VMs.  The first one I created, running Ubuntu 20.4, was to play around with Docker.  I’m still no expert, but I have become quite dependent on this VM as it is running Bitwarden, Teslamate, Calibre-web and a test instance of WordPress (for playing with new plugins and themes). It does have some mount points to a Docker dataset with more permanent storage, but it’s not as consistent as I’d like.  It became a little unwieldy at one point, as I was also trying to run other things, so eventually, some of these got moved to their own VMs.  I have an Ubuntu VM just running ONLYOFFICE Document Server, which is linked to the Nextcloud service for an Office 365-type experience.  I have another Ubuntu VM running Pi-Hole and another running Mattermost, although this is still pretty much a playground and not used in anger, so I wouldn’t be too upset to lose it.  The last one is a ubuntu 22.4 VM running CrashPlan, which backs up any irreplaceable data in the cloud.

Having learnt to set all this up and maintain it for the last 7-8 years gives me hope to learn the necessary skills to do it all over again.  My only worry is this time I have data I don’t want to lose, and services I’d be devastated to have to start again from scratch, which I didn’t have when I started my FreeNAS > TrueNAS journey.  I know ownCloud was rebuilt a few times, as was emby, before iocage allowed jails to be upgraded after the version of FreeBSD was!

 

 

 

TrueNAS 2022

This has now become a regular annual update, but I really didn’t think after the crazy year of 2020 (blogged about here), that we’d be in a very similar situation at the end of 2021. I’ve spent the whole of 2021 working from home, working for the...

Restarting TrueNAS and Portainer

Restarting TrueNAS and Portainer probably isn’t a great title for this blog, but it will pop-up as a reminder for me to simply restart the NFS service after I’ve done this and hopefully avoid a few wasted hours trying to work out the problem!

(more…)