TrueNAS 13.3

Oct 27, 2024 | IT, Server, TrueNAS | 0 comments

Well, it’s been a busy month!

TrueNAS and FreeNAS (as it was named before that) have been one of, if not the, most blogged-about subjects in the last 250 blogs (yes, I made it to 250 and didn’t even notice) over the past 8 years (or at least it will be in November 2024).

I’ve just upgraded from TrueNAS 13.1-U6 to 13.3, which could be the last major release of TrueNAS Core, the FreeBSD-based version that was FreeNAS 9.2.8 when I started using it in 2013. 

Upgrading TrueNAS is usually pretty uneventful, even though it causes me no end of worry for weeks before and sometimes weeks afterwards.  This is one of these updates that I might still be fighting with in months to come!

I’ve already blogged about some of iXsystems focus changes here, with TrueNAS Scale based on Linux being the future direction of travel. What this means for TrueNAS Core is still a little unclear, but hopefully, there are still a few years before that becomes a major issue for me.

After updating TrueNAS (which is built on FreeBSD 13.3 rather than 13.1), the process of upgrading the jails begins.  This updates them to the base version of FreeBSD.  I successfully upgraded one jail from 13.1 to 13.3 and then updated and upgraded the packages (#pkg update & pkg upgrade), but it was only my Airsonic jail, which isn’t really used in anger anymore. I wouldn’t be upset if it broke and I had to destroy it.  

Lots of my jails are web-based applications running on a FEMP stack, or FreeBSD (as the operating system), Nginx (a web server pronounced Engine-X), MySQL (a database server, in my case Mariadb), and PHP (a programming language to process dynamic PHP content).  I’ve kept the jails running by upgrading the operating system between versions of TrueNAS using commands like #iocage upgrade -r 13.3-RELEASE <jailname> and then within the running jail #pkg update && pkg upgrade.  I needed to update PHP separately, which I blogged about here, but the one thing I neglected was the MySQL element.  My Nextcloud, WordPress, and LimeSurvey jails are all running Mariadb103, which, along with Mariadb104, are no longer supported.  The latest version of MariaDB is 10.11, so I’m pretty far behind.

I asked for some help from the TrueNAS forum, and as ever, I wasn’t let down with some excellent advice.  It turns out I can update the jail base to FreeBSD 13.4 on TrueNAS 13.3, so I’ll be doing that and then trying to update Mariadb103 to Mariadb105 as a first step.  I’ve struggled with this in the past and simply rolled back and found a way to update without touching the database.  In hindsight, that wasn’t a great idea, but at least now I have a problem that needs to be solved, and once I’ve solved it for one, it should be easy enough to apply to the others.  

It turned out to be a little more problematic than I thought, although that’s more down to my lack of System Administration skills than the technology. The first rule of updating databases is to ensure the old database has been shut down correctly and not aborted by trying to jump a step ahead and simply install the new one! I’m sure this will be in System Administration 101, which I think I also skipped.

I also learned about something called tmux, which lets you keep a session running on TrueNAS even when your SSH session is lost. How have I made it 10 years without knowing this? To be fair, they have only just removed the built-in shell console, which I tended to use more than the SSH method.

Anyway, I’m going to list some commands I’ve used to upgrade, which is more for me than anyone reading this:

 

#tmux new -t iocage (starts a tmux session called iocage)

#tmux a (restart the previous tmux session – I’m guessing you have could several running, although I’ve only tried with one)

#iocage fetch -r 13.4-RELEASE (downloads the release)

#iocage upgrade -r 13.4-RELEASE <jailname> (update the jail to the specified release – note, there are various ‘y’ prompts to confirm the process here, which is where tmux comes in handy.  There is also a step which appears to stop with a : prompt, and this is skipped by entering q several times)

#iocage restart <jailname> (restart jail after upgrading)

#iocage console <jailname> (console session into jail)

#pkg info (lists all installed packages and versions)

#service mysql-server stop (clean stop of database server)

#pkg install mariadb105-server (this will remove the old database server and then install the new one – 10.5 – along with any dependencies)

#service mysql-server start (this should start the new database server; check with #service mysql-server status)

#mysql_upgrade -u root -p (to upgrade the databases)

#pkg update && pkg upgrade (this will upgrade all of the other packages in the jail)

 

I’m not going to pretend I understand exactly what changed between mariadb103 and 105, but the main issue I encountered was the UNIX socket connection, which changed from /tmp/mysql.sock to /var/run/mysql/MySQL.sock. These changes were in the /usr/local/etc/mysql folder and ./conf.d folder.

So, while the database started and the upgrades worked, none of my web applications would start, as the configuration to the database was wrong.  So far I’ve fixed Nextcloud (/usr/local/www/nextcloud/config/config.php where you set ‘dbhost’ => ‘127.0.0.1’,) and WordPress (/usr/local/www/wordpress/wp-config.php where you set define(‘DB_HOST’, ‘127.0.0.1’);)

This got everything back up and running but after a reasonably stressful afternoon and several snapshot rollbacks where I’d broken things and had no idea what to do.  Thank goodness for TrueNAS snapshots and the incredibly helpful people who use their forum.  It’s still one of the best online communities I’ve come across, even though I visit less frequently since the forum platform changed.  I need to try and fix that!

 

I Loved FreeNAS

Sep 1, 2024 | FreeNAS, IT, TrueNAS | 0 comments

This blog is a bit of a con, as I drafted it to post on the iXsystems forum in the hope that I might get some ideas about what to do with my TrueNAS server. I’ve posted it below, but I still haven’t added it to the forum!

 

I loved FreeNAS, and since first installing it over ten years ago, I have learned so much about hosting and maintaining my own micro data centre, much of it from the old forums.

The death of the old forum and the focus on TrueNAS Scale have been quite disheartening, and I don’t know where I go from here. Obviously, everything is still working, so I don’t have to make any immediate decisions, but I feel I should at least be thinking about what the future holds and how I might get there.

I was hoping that if I described how I was currently using TrueNAS Core, I might gain some of the forum’s incredible wisdom to help reinvigorate my love for all things Free to TrueNAS.

For background, I started this journey running an HP N40L Microserver and about 8TB of RAW storage.  That was upgraded to an NL54 and 16TB, and then I built my own server in a Fractal Define R5 case with an Intel Xeon E3 and Supermicro motherboard, ECC RAM and 36TB storage.  See, I was paying attention!  This was upgraded about 4 years ago using the same case, an updated Xeon and Supermicro motherboard, 64GB ECC RAM and 64TB storage, with some SSDs for jails and VMs.  I still have the old one and replicate to it monthly (it used to run 24×7 until the cost of electricity went through the roof), so I have something I could play around with using Scale.

Obviously, my main use is storage.  I have two pools.  This main pool is 8x8TB RAIDZ2 WD HDD which is currently running about 75% (I know, too high really but I was considering upgrading the 8TB drives to 16TB over a few months).  The second pool is 2x1TB Mirror Crucial SSD which is currently running about 65%. The main pool is predominantly long-term storage, and the second pool is for the jails and VMs and associated databases.

I know Scale will do all the storage stuff, so that’s not my worry.  I’m worried about the jails and virtual machine I have running lots of externally facing services accessed through NGINX Proxy Manager, now running on a separate Raspberry Pi (it was running in a jail for a few years)

I have six jails of varying degrees of importance.  My Nextcloud jail is probably the most important and is used by family and a small charity I was a Trustee for.  The files and database are in separate datasets, and I know this could be run on Scale, but moving it and breaking it scares me!  I also have an emby jail used by around 15 family members and friends.  It has the emby_server folder mounted to a separate dataset, so again moving should be possible.  I’m assuming it is possible to mount datasets into containers or VMs similarly to jails, but not whether the structure of these will be the same cross-platform?  I also have a WordPress jail, which used to run my company website, but it’s now just my personal blog.  It’s a similar set-up with a dataset stored outside the jail.  I have three other jails running OpenVPN (so I can connect back to my network remotely), Limesurvey (running small surveys, which could be lost) and Airsonic (an old music streaming service, which has pretty much been replaced by emby).

I then have some VMs.  The first one I created, running Ubuntu 20.4, was to play around with Docker.  I’m still no expert, but I have become quite dependent on this VM as it is running Bitwarden, Teslamate, Calibre-web and a test instance of WordPress (for playing with new plugins and themes). It does have some mount points to a Docker dataset with more permanent storage, but it’s not as consistent as I’d like.  It became a little unwieldy at one point, as I was also trying to run other things, so eventually, some of these got moved to their own VMs.  I have an Ubuntu VM just running ONLYOFFICE Document Server, which is linked to the Nextcloud service for an Office 365-type experience.  I have another Ubuntu VM running Pi-Hole and another running Mattermost, although this is still pretty much a playground and not used in anger, so I wouldn’t be too upset to lose it.  The last one is a ubuntu 22.4 VM running CrashPlan, which backs up any irreplaceable data in the cloud.

Having learnt to set all this up and maintain it for the last 7-8 years gives me hope to learn the necessary skills to do it all over again.  My only worry is this time I have data I don’t want to lose, and services I’d be devastated to have to start again from scratch, which I didn’t have when I started my FreeNAS > TrueNAS journey.  I know ownCloud was rebuilt a few times, as was emby, before iocage allowed jails to be upgraded after the version of FreeBSD was!

 

 

 

Long time, no blog

Jan 1, 2024 | IT, TrueNAS | 0 comments

Well, it really is a long time since the last blog, which was at the start of 2022! Life just seemed to get in the way as things started to get back to normal after the COVID-19 pandemic, but that’s really quite a poor excuse. I did start a new permanent job, something I’d last done all the way back in 1995, although I just missed the 20 years working as a Contractor by less than a handful of months.

So why am I back? To be honest, I’ve missed blogging and thought I needed a New Year’s resolution, so what better than to restart this blog? And why not start as we ended, with a TrueNAS 2024 update?

The first thing to comment on is the version of WordPress, which has jumped from 5.8.2 to 6.4.2, and the block editor already feels somewhat alien! It’s going to take some time to get back into things!  The Divi theme has also seen some updates, so you’ll have to excuse some of these early blogs as I get back up to speed!

TrueNAS has received a few updates and is now running 13.0-U6, although feature-wise, it’s very much the same. IXsystems seems to be focusing their development on their Linux-based SCALE product rather than the long-serving FreeNAS-based CORE product, but I’ve not been tempted to switch and still love the flexibility of running both FreeBSD jails and Linux-based virtual machine.

I’m going to keep this first blog reasonably short, so let’s start with a table for the jails:

Jail20222024Comments
Airsonic???10.6.2This might be the only thing that hasn’t changed in this entire blog, but it still works and is up and running.  I still don’t really use it much and tend to listen to music using Apple Music or occasionally via emby.
emby4.7.6.04.7.14.0I’m still running emby as my Home Media Server, although it’s been through some bumpy times with a fairly major security issue and a pretty slow development/release cycle. I wasn’t directly affected and think I have things reasonably well locked down via reverse proxy, but I’m hoping some extra security is coming in the product’s next release. I’ve dropped Plex and no longer have a jail or lifetime account.
Nextcloud22.2.327.1.5It feels like there have been lots of Nextcloud updates, and the version numbering would support that. It’s still one of my most heavily used services, and I’d be lost without it. The PHP stack has also been updated and is now running 8.2.7, although I’ll look to bump that to 8.3 early this year when version 28 is released.
Limesurvey???6.4.0+I don’t update this as often as I should (I did update it over the holiday though), but I’m also not using it much. It got quite a lot of use over Covid running surveys for both charities my partner and I helped to run, but we’re both less involved now.  It hasn’t run a live survey in well over 12 months.
OpenVPN2.5.32.6.6This is updated regularly, and it just ticks along in the background, letting me access my network remotely whenever required. It’s used less often than it was pre-Covid, but it still comes in handy occasionally.
WordPress5.8.26.4.2It’s another one that is updated regularly but hasn’t been used. I ran my company site and blog from this, but my company was wound up when I took the permanent job, and as you know, this blog has been dormant. It’s running PHP 8.2.11.

And another one for the VMs:

NameOS 2022OS 2024Application(s)
cplanubuntu-desktop 18.4.6ubuntu-desktop 22.4.3This runs Crashplan exclusively, and I kind of forget it’s there, other than the weekly e-mail to remind me it’s backing up my TrueNAS server and the monthly invoice for $11.99!  It’s still the cheapest I can find for offsite backup, which is especially important as my onsite backup no longer runs 24×7, given the rising electricity costs and a remodelled home office with no space for two large workstations/servers. It does get updated every few months, but it’s nice to know that Crashplan has a much more up-to-date backup.
mattermostubuntu server 20.4.2ubuntu server 20.4.2I should upgrade the OS for this one, but it is still supported until 2025 and will be updated whenever I update the LTS release of Mattermost. It was running 6.0.3 and is now running 8.1.0. 9.5 is due next month, so perhaps I will upgrade both simultaneously.
oedocsubuntu server 18.4.5ubuntu server 22.4.3I should have called this oodocs, as it’s running ONLYOFFICE Document Server, which I use with Nextcloud, but it just works, so I have left things alone. I’m getting wiser in my old age! It’s updated every few months and is currently running 7.5.0-125.
piholeubuntu server 18.4.5ubuntu server 22.4.1Another one I update regularly, and it’s worked brilliantly to hide most advertising when web browsing, but also speed up DNS searches using unbound.
portainerubuntu server 18.4.5ubuntu server 20.4.4It’s another with the wrong name, as it’s running all my Docker containers, although one is Portainer. It is hard to cover them all in detail here. Still, the main ones are Bitwarden, Calibre-web, Teslamate/Grafana, Uptime Kuma, WordPress (a development version), and other web tools (LEMP stack, phpmyadmin, etc.)

I have a few dormant VMs running ONLYOFFICE Suite, a test version of TrueNAS Scale and Windows 2016 Server, but none are running, and I can’t remember the last time I started them up.

I’m still running a Raspberry Pi 4 from an SSD running a pretty up-to-date version of the Home Assistant Operating System (11.2) and an equally up-to-date version of Home Assistant Core (2023.12.3). As well as providing all the home automation for my Philips Hue lighting and TP-Link switches, I have various integrations for my Tesla and Octopus Energy. Still, the most important is probably the Addon for NGINX Proxy Manager, which controls all access to my network and the Certbot certificate renewals every three months. This has worked brilliantly since I switched from doing this semi-manually in a jail, which I think I blogged about here!

So that’s about it. My TrueNAS system main pool uses 70% of its capacity but still has over 12TB of storage, and the SSD pool is 62% with ~340GB of storage. Both should be good for another few years, and the server performance is still fine even though it’s now four years since I built it and blogged about it a lot here.

I will try to blog once a month, so this covers me for at least 50 days! I’ve got much more to say about Whisky than I did in 2022, and I have built up a small collection of about ~75 bottles in the loft! A Raspberry Pi 5 arrived last month, too, so there might be some blogs about that. And the Tesla I’ve been driving since this blog is due to return in June, so there might be something to say about that. And perhaps something about my new Office, which I mentioned earlier, and some of the new things displayed? I’ll let you know before the end of February….

HAPPY NEW YEAR 

TrueNAS 2022

Jan 1, 2022 | IT, TrueNAS | 0 comments

This has now become a regular annual update, but I really didn’t think after the crazy year of 2020 (blogged about here), that we’d be in a very similar situation at the end of 2021.

I’ve spent the whole of 2021 working from home, working for the same customer, which might become more permanent in 2022, but that’s for another blog.  Before starting to write this blog, my impression is that not much had changed, but let’s see…

As with my 2021 update, I start 2022 with most things up to date.  There was a TrueNAS Core update in December which I still haven’t applied yet, as once again it’s broken some of the reporting functionality on the dashboard.  Nothing major, but frustrating nonetheless, especially as this has been an ongoing issue with quality control for much of 2021.  I’m still running 12.0-U6.1, so not too far behind.

There hasn’t been a major update to TrueNAS during 2021, so no need to upgrade the version of FreeBSD running in any of the jails, although I’ve tried to keep them reasonably up to date.

I’ve had no hardware issues with my main FreeNAS machine, and only replaced one of the case fans on my backup machine which was causing the others to run too quickly and noisily!

Given the increasing cost of electricity, I’ve switched to running my backup server once a week (rather than 24×7) and only backup at the weekend, which I’m sure is saving me some money, although you wouldn’t tell from my ever-increasing utility bill!  The UPS replaced in 2020 has continued to work perfectly, and helps with keeping an eye on the energy used.

 

At the beginning of 2021, I had one VM running Docker that had several services running as containers within that.  I’ve split some of those out, so have a few more VMs running fewer things, but with pretty much the same services:

  • Ubuntu Server 18.04.6 LTS running ONLYOFFICE Docs Community Edition (updated around the middle of the year) for integration with Nextcloud
  • Ubuntu Server 20.04.2 LTS running Mattermost Team Edition 6.0.3, so pretty up to date!
  • Ubuntu Server 18.04.5 LTS running Docker/Portainer which is mainly used for Bitwarden and Calibe-web, although does have a few other containers for testing things (this was the main one at the start of 2021)
  • Ubuntu Server 18.04.5 LTS running Pi-hole which I update every few months
  • Ubuntu Desktop 18.04.6 LTS running Crashplan

I still have a Raspberry Pi running Hass.io for all my home automation, which also runs NGINX Proxy Manager, and that’s been upgraded to run from an SSD rather than an mSD card.

And I still have a couple of VMs to run Windows Server 2016 and test different versions of TrueNAS, although if they’ve been started more than a couple of times this year I’ll be surprised.

I should probably think about updating some of the 18.04 systems in 2022 given standard support finishes for these in 2023, but I’ve not had much luck with upgrades in the past, so might end up rebuilding them at some point with a 22.04 LTS version when it lands. 

In terms of the iocage jails, this list is identical to last year, with just jail and application updates:

Airsonic

It’s rarely used or updated, as I don’t think Airsonic is well supported anymore, or at least not the Java version I’m running.  I used iTunes at home for listening and cataloguing music until a few months ago, but I am now tending to listen using Spotify, Apple or Amazon Music.  I can also access my library through emby, so this might not see the end of 2022.

 

emby

 

emby is still my media server of choice and something I would be lost without. It’s running 4.7.6.0 (from 4.5.3.0) which is the latest stable release.

 

 

Nextcloud

 

It’s a close call between this and emby as the jail I’d be most lost without, although the WordPress one might put up a strong case too.  I use this for syncing files across all of my devices but also for accessing files remotely when I’m not on my own devices.  The integration with ONLYOFFICE just adds the cherry on top!  I’m almost up to date running the latest 22.2.3, PHP 7.4.22 and am now using this for a charity I help out with IT.

 

(Lime)Survey

 

I’m still keeping this up to date manually, and although it’s not used heavily, it’s handy to have an online survey tool.  My partner ran a survey for the charity she helps out with and I’m sure we need to run a few more for my daughter Out Of School Club in 2022.

 

(Open)VPN

 

Working from home means this gets used less frequently, but it’s still really useful to have when I’m away from home.   I think it’s a version behind on 2.5.3, but that’s not bad for this one which I tend to avoid updating unless I really need to!

 

WordPress

As with last year, I’ve still got the same WordPress jail for my company website and this blog, and a Docker container running another version of testing.  They are both up to date running 5.8.2, although my test instance is a little behind on PHP 7.3.12 while the live system is running 7.4.22.

 

I have jails for Bitwarden, Calibre, MediaWiki, Minecraft, Photoprism and Plex, but none are running.  Bitwarden and Calibre are covered by VMs.   Minecraft and Photoprism were just to test what was possible, but are too resource-intensive to run 24×7.  I’m not sure why I keep my old Plex jail running as there’s no going back from emby, and it’s not been updated at all this year.

 

So I was right at the start.  Very little had changed, other than keeping most things up to date!

And as with 2021, almost everything is up-to-date and running well.  I still have around 17TB of storage capacity on FreeNAS0 and the jail SSDs are still only running around 60% capacity, although probably a little higher than I’d really like.  FreeNAS1 on the other hand is approaching 90%, although given that’s really just a backup target, I don’t think I need to worry about that limit.  So long as I don’t hit 100% everything should be fine, and I can always be a little more selective about what I backup.  I had planned to replace the 4TB drives with the 8TB drives in FreeNAS0 when I upgrade them, but I suspect I’m more likely to retire it all together and rely on Crashplan.

Anyway, that was my New Year TrueNAS update!  Have a prosperous New Year and hopefully, 2022 will see the world return to ‘normal’ and start like that when the end of the year comes around again.

Painless Upgrades

Aug 19, 2021 | IT, Server, TrueNAS, WordPress | 0 comments

Well, almost! It’s been a little while since I blogged about upgrading PHP here, and while I think I’ve done one update (from 7.4.4 to .13) I haven’t kept things all that up to date. I thought it was about time to change that, and overall they were pretty painless upgrades.

(more…)

Restarting TrueNAS and Portainer

Mar 24, 2021 | Docker, IT, TrueNAS | 0 comments

Restarting TrueNAS and Portainer probably isn’t a great title for this blog, but it will pop-up as a reminder for me to simply restart the NFS service after I’ve done this and hopefully avoid a few wasted hours trying to work out the problem!

(more…)