TrueNAS is dead, long live FreeCORE

This moment feels like it’s been coming for some years now. I can’t remember exactly when iXsystems announced it was dropping support for TrueNAS Core, their FreeBSD-based NAS appliance, in favour of TrueNAS Scale, their newer Linux-based version. Having been using TrueNAS, and FreeNAS before that, for about 13 years, I wasn’t sure what to do.

The reality is I buried my head in the sand until that was no longer an option, thanks to the PHP vulnerability that’s been all over the news this week. It looks like that might have been a sensible strategy, given FreeCORE has only been an option for the last few months, although just typing that emphasises what an enormous leap of faith I’m taking by migrating this weekend. This blog is going to be that journey, along with some of the thinking that’s going along with it. Who knows, it might go horribly wrong and the blog this is intended for never comes back online (guess what, it did!)

I started writing this at 23:39 on Friday 23 October, and every service on my TrueNAS box is down. The box is running and set to snapshot all the critical things in about 20 minutes and then replicate those to the backup box overnight. Tomorrow morning I’ll check that backup ran successfully, the target has the latest files, and then switch the train from TrueNAS to FreeCORE and upgrade. Wish me luck….

I’M BACK

So, here we go. The first job was to check that replication worked, and it did. I used to run both machines 24×7, but when energy prices spiked because of the war in Ukraine, I reverted to monthly backups to the old FreeNAS1 box. I still have more regular backups running to CrashPlan, so this seemed like a reasonable trade-off. I re-read the instructions and put on my ‘big boy pants’ before starting the update (or is it an upgrade?)

First gotcha was DNS (surprise surprise), as I have a VM running on my TrueNAS box running Pi-hole and unbound. That’s off, so I needed to point it back to my router, and the router out to a secondary DNS server.  Easy enough.

Then I switched the TrueNAS update train to FreeCORE, which is pretty much the only config. After that, you just run the update as usual from TrueNAS and wait. It didn’t take as long as I’d expected, as I was going to head out to walk the dog so I didn’t feel the need to interrupt, but before I’d had a chance, it started back up looking exactly like TrueNAS, which I thought was strange. A quick refresh and some FreeCORE branding and some interface tweaks, but feeling very much the same.

I thought I’d stopped everything from auto-booting, but I’d missed a few jails, so I changed that and then thought I’d reboot so I was starting from where I’d planned.

The system is running FreeCORE-15.0-U2.4 on the FreeCORE-15.0-STABLE train. This is the recommended first step to check that everything is still working as expected.

I can connect to the SMB shares from my MacBook, which, given it’s a NAS, is pretty important. I checked the pools with zpool status (all 3 ONLINE) and the FreeBSD version with freebsd-version -ku, which reports 15.1-RELEASE-p5.

sysctl kern.features.compat_freebsd13 returns a 1 which means features are still compatible with 13, which is currently where all my jails are sitting. This is probably going to be the biggest job, updating them to 15.1

I tested one worked (Nextcloud) and I could access through the NPM proxy, but then quickly closed that down as this is one of the PHP issue jails.

Next was the big one – would the Portainer/Docker VM start? YES, and other than dropping the Portainer license, which I had saved in Bitwarden, it looks like most, but not all of the containers came up. Teslamate didn’t and it looks like all of the Stacks in Portainer are unavailable.

So with a little help from Claude I managed to get to the bottom of this. It turns out three of my fourteen NFS mounts hadn’t come up, because the VM booted faster than the NAS and nothing retried them. One of those was the share holding all my Docker data. That is also the answer to why everything else started: the other eleven mounts were fine, and when Docker found its directory missing it quietly created an empty one on the VM’s own disk, so the containers came up happily on nothing at all. Portainer started on a blank database, which is why it looked like a fresh install. Teslamate was the only one that couldn’t paper over it and actually said so. The fstab entries only had nfsvers=3 and now have _netdev, nofail and a retry, and Docker itself has been told to wait for its storage rather than invent it.

I also started up the Pi-hole VM and some of the jails and everything appears to be working just fine.

I was going to leave upgrading jails from 13.5 until Sunday, but given how well things are going I’m going to start now. I’d got Claude to create me a ‘playbook’ which included the jail migration, but the new FreeCORE forum also had a HOW TO guide for this, so I tried to use the best of both.

Nextcloud was first and the hop to 14.5 took well over an hour, with long stretches where it looked like it had hung. I’d been using Claude to check a few of the messages and mentioned the time it was taking. Claude suggested trimming freebsd-update's components, because it was reinstalling the entire FreeBSD source tree one file at a time. Nothing in these jails needs that, and trimming it brought the second hop, to 15.1, down to about fifteen minutes. WordPress then did both hops at the faster rate and updated perfectly, so the two jails actually affected by the PHP security flaw were both done on the first day. And since this blog runs on that WordPress jail, the thing I was most worried about on Friday night turned out fine — after all, you’re reading it!

LimeSurvey is the same, so I’m planning to do that one and then stop for the night, but I’m sure this results in a FreeCORE update blog in the not too distant future. Perhaps one to end 2026?

There’s some blogs about Omarchy too, as I’m picking up the EliteBook 6 G1a (now with 32GB RAM) more than I am my MacBook at the minute, and that’s definitely an OS preference over a hardware one. I’ve drafted this in LibreOffice on Omarchy!

But today has been a good ‘sysadmin’ day. Sad to leave Free/TrueNAS behind but happy to have found FreeCORE as a very familiar replacement.

 

I Loved FreeNAS

This blog is a bit of a con, as I drafted it to post on the iXsystems forum in the hope that I might get some ideas about what to do with my TrueNAS server. I’ve posted it below, but I still haven’t added it to the forum!

 

I loved FreeNAS, and since first installing it over ten years ago, I have learned so much about hosting and maintaining my own micro data centre, much of it from the old forums.

The death of the old forum and the focus on TrueNAS Scale have been quite disheartening, and I don’t know where I go from here. Obviously, everything is still working, so I don’t have to make any immediate decisions, but I feel I should at least be thinking about what the future holds and how I might get there.

I was hoping that if I described how I was currently using TrueNAS Core, I might gain some of the forum’s incredible wisdom to help reinvigorate my love for all things Free to TrueNAS.

For background, I started this journey running an HP N40L Microserver and about 8TB of RAW storage.  That was upgraded to an NL54 and 16TB, and then I built my own server in a Fractal Define R5 case with an Intel Xeon E3 and Supermicro motherboard, ECC RAM and 36TB storage.  See, I was paying attention!  This was upgraded about 4 years ago using the same case, an updated Xeon and Supermicro motherboard, 64GB ECC RAM and 64TB storage, with some SSDs for jails and VMs.  I still have the old one and replicate to it monthly (it used to run 24×7 until the cost of electricity went through the roof), so I have something I could play around with using Scale.

Obviously, my main use is storage.  I have two pools.  This main pool is 8x8TB RAIDZ2 WD HDD which is currently running about 75% (I know, too high really but I was considering upgrading the 8TB drives to 16TB over a few months).  The second pool is 2x1TB Mirror Crucial SSD which is currently running about 65%. The main pool is predominantly long-term storage, and the second pool is for the jails and VMs and associated databases.

I know Scale will do all the storage stuff, so that’s not my worry.  I’m worried about the jails and virtual machine I have running lots of externally facing services accessed through NGINX Proxy Manager, now running on a separate Raspberry Pi (it was running in a jail for a few years)

I have six jails of varying degrees of importance.  My Nextcloud jail is probably the most important and is used by family and a small charity I was a Trustee for.  The files and database are in separate datasets, and I know this could be run on Scale, but moving it and breaking it scares me!  I also have an emby jail used by around 15 family members and friends.  It has the emby_server folder mounted to a separate dataset, so again moving should be possible.  I’m assuming it is possible to mount datasets into containers or VMs similarly to jails, but not whether the structure of these will be the same cross-platform?  I also have a WordPress jail, which used to run my company website, but it’s now just my personal blog.  It’s a similar set-up with a dataset stored outside the jail.  I have three other jails running OpenVPN (so I can connect back to my network remotely), Limesurvey (running small surveys, which could be lost) and Airsonic (an old music streaming service, which has pretty much been replaced by emby).

I then have some VMs.  The first one I created, running Ubuntu 20.4, was to play around with Docker.  I’m still no expert, but I have become quite dependent on this VM as it is running Bitwarden, Teslamate, Calibre-web and a test instance of WordPress (for playing with new plugins and themes). It does have some mount points to a Docker dataset with more permanent storage, but it’s not as consistent as I’d like.  It became a little unwieldy at one point, as I was also trying to run other things, so eventually, some of these got moved to their own VMs.  I have an Ubuntu VM just running ONLYOFFICE Document Server, which is linked to the Nextcloud service for an Office 365-type experience.  I have another Ubuntu VM running Pi-Hole and another running Mattermost, although this is still pretty much a playground and not used in anger, so I wouldn’t be too upset to lose it.  The last one is a ubuntu 22.4 VM running CrashPlan, which backs up any irreplaceable data in the cloud.

Having learnt to set all this up and maintain it for the last 7-8 years gives me hope to learn the necessary skills to do it all over again.  My only worry is this time I have data I don’t want to lose, and services I’d be devastated to have to start again from scratch, which I didn’t have when I started my FreeNAS > TrueNAS journey.  I know ownCloud was rebuilt a few times, as was emby, before iocage allowed jails to be upgraded after the version of FreeBSD was!

 

 

 

Another New Year TrueNAS Update

Although this is going to be similar to my New Year update at the beginning of 2020, this New Year TrueNAS update feels very different!

I’ve been working from home since March 2020 and for at least some of the year had a little more time to play around with things.

Similar to the beginning of 2020, I start 2021 with most things up to date.  Midway through December, I upgraded my FreeNAS machines to the latest version of TrueNAS Core, which I blogged about here.  Although there were a few bumps, everything has been running well since, and I’ve also upgraded all of the jails without too many issues.

FreeNAS0 has performed fantastically during 2020, and while I did have an issue with one of the HDD in the pool. it was replaced with the cold-spare and the RMA process with WD worked flawlessly, which I blogged about briefly here.

I also replaced my UPS during 2020, although this was a simple return process through Amazon and an identical replacement from eBuyer.  The old one had been shutting down randomly, which kind of defeats the object, and Amazon’s solution was to return it for a full refund.  They couldn’t provide a replacement, so I picked one up from eBuyer, saving about £30 and that’s run 24×7 since it was switched on, as you’d expect from a UPS.

 

From a VM perspective, very little has changed.  I still have Ubuntu VMs running ONLYOFFICE, CrashPlan and Docker, although I have a new one configured with Docker just for running Pi-hole.  I also had a VM running NGINX Proxy Manager although that’s been moved across to my Hass.io Raspberry Pi now.

I still have a couple of VMs for playing with Windows and TrueNAS, although they spend most of the time switched off.  Other than a couple of sharing issues, which were easily resolved, the TrueNAS update for the VMs was excellent.

In terms of the iocage jails, here’s an update on those, in ‘jail-name’ alphabetical order:

Airsonic

This is a new one from last year which I built in June and blogged about here.  Given I’ve spent most of 2020 at home, it’s not really been all that useful but I imagine I will use it a little more when things start to return back to the new normal.

emby

emby is still my media server and there’s no going back to Plex. It’s running 4.5.3.0, which is the latest stable release.

MediaWiki

Another new one, which I’ve blogged about a few times in 2020.  I’m not really using it for anything, but hope to find something at some point.

Nextcloud

This is probably the jail I’d be most lost without, although the WordPress one might put up a strong case.  I use this for syncing files across all of my devices (iMac, MacBook, iPhones, etc.) but also for accessing files remotely when I’m not on my own devices.  The integration with ONLYOFFICE just adds the cherry on top!  I’m up to date running the latest 20.0.4, PHP 7.4.13 and can only see my use of this continuing to grow in 2021.

(Lime)Survey

This was my newest jail last year and has been used again this year to run a couple of small surveys for my daughters Out Of School Club again. It’s updated less regularly now as I tend to do it manually rather than paying for the privilege of using the built-in Comfort updater.  It takes a little longer but is saving me £80-90 a year.

(Open)VPN

Still working well for connecting back into my network securely whilst away from home, although again it’s had less use this year given I’ve spent most of my time at home! Having just upgraded all of my jails with TrueNAS, it’s also surprisingly up to date running 2.4.9.

WordPress

As with last year, I’ve still got the same WordPress jail for my company website and this blog, and a Docker container running another version of testing.  They are both up to date running 5.6, although my test instance is a little behind on PHP 7.3.12 while the live system is running 7.4.13.

And that’s it, so a few haven’t made it to the end of 2020! 

I replaced the SSL Proxy jail in June when I was running NGINX Proxy Manager in a small VM.  Around October this move to my Hass.io Raspberry Pi.  It’s a much cleaner tool for managing this, but I’m glad I ran things manually for a little while and have some understanding about what this is doing in the background.  I moved all the old config and deleted the jail, but I can’t imagine ever going back to doing things manually.

The Home Assistant jail was moved to a Raspberry Pi as there’s so much more functionality available running Hass.io and Home Assistant Core than you can via the virtual python environment used to power things on FreeBSD.  I didn’t blog about this at the time, but I perhaps will sometime this year as I really need to move the OS onto an SSD rather than the default micro SD card.

I switched to running Calibe-web in a Docker container almost straight after my 2020 update, which I blogged about here.  It’s so much better and is running the latest 0.6.9 version.  I still have the calibre jail, although it’s switched off and is another one to delete this year.

And lastly, my DNS jail.  This was replaced with a VM running Pi-hole in February and has again proved to be a much nicer solution.

So as with 2020, almost everything is up-to-date and running well, and in many cases better than it was 12 months ago.  I still have over 18TB of storage capacity on FreeNAS0 and the jail SSDs are still running less than 50% capacity.  FreeNAS1 on the other hand is approaching 80%, although given that’s really just a backup target, I don’t think I need to worry about that limit.  So long as I don’t hit 100% everything should be fine, and I can always be a little more selective about what I backup.  At some point, I will replace the 4TB drives with 8TB drives from FreeNAS0, but I shouldn’t need to do that this year. 

Anyway, that was my New Year TrueNAS update!  Have a prosperous New Year and hopefully, 2021 will be a much more normal year.  If you didn’t get around to playing with FreeNAS in 2020, makes 2021 one to check out TrueNAS…

TrueNAS

TrueNAS

A major FreeNAS upgrade to TrueNAS has been on the cards for most of 2020. TrueNAS 12.0 was released in October, but I’ve waited until the first U1 release before attempting my FreeNAS upgrade. How did it go? Well, there’s some good and bad…

(more…)

Lost Blogs

I’d drafted 2 blogs yesterday – one about a portable monitor and another about Firesticks. This morning I decided to be brave and update WordPress to 5.5 (I’d tested two other sites yesterday which both worked perfectly) but it failed and I’ve lost (the) blogs.

(more…)