Or should that be FreeNAS and AirPort issues? Given my UPS died less than a couple of weeks ago (I blogged about the new one here), the saying ‘bad things come in threes’ seems pretty relevant for me at the minute!
Saturday
It started on Saturday evening, which at least gave me some time on Sunday to try and make sense of what was happening, and although things are mostly resolved, I still have some outstanding issues and might not make the best job of explaining what happened or what I did, and still need to do.
Sunday am
I woke on Sunday to some e-mails from my WordPress server (running in a jail on my FreeNAS box) to say it had gone offline. My initial thought was the new UPS, but when I checked the FreeNAS box it was still powered on, but completely unresponsive and none of the other services was running. After some head-scratching, I decided to hard boot it and take the dog for a walk!
I came home to a rebooted server that I could access from the Web User Interface (WUI) – check, all the jail services appeared to be up and running – check, and the Ubuntu VM up and running – Uh-Uh!
From what I could tell, the VM was running, but I couldn’t access it via a VNC connection, not any of the services I have running from it. I tried another reboot, and a few posts to support forums, but I was running out of ideas fast. Fearing the worst, I thought about spinning up a new Ubuntu VM, as I was starting to think I’d need to rebuild things. Other than the time, that wasn’t necessarily a bad thing as I could start afresh with Ubuntu 18.04 LTS (my current VM is 16.04)
But after downloading the 18.04 ISO file, I couldn’t get the installation to connect to the network, as it wasn’t able to get a DHCP connection. This led me to check a few other things on my network, and the first few devices I checked were all connecting fine via WiFi, but when I check a wired device without a static IP address that wasn’t connected either.
Sunday pm
I’m still not even sure that’s possible, but it steered me to accessing my VM through the FreeNAS console, which worked fine. So the VM was running, just not connected to the network. A little googling and I’d configured a static IP address for the VM and viola, accessible and everything running again.
I then started to look at what other devices across my network weren’t connected. More than I imagined! Everything with a static IP address was working (hence all the FreeNAS jails starting up) but anything wired with a fixed IP assigned from the MAC address, or anything wired requesting DHCP was offline. I spent some time working through these, assigning a static address wherever possible, but a few devices defeated me, namely my Virgin Media TiVo box and HDHomerun (which only work with DHCP) and my AirPort Express router (which I should be able to assign a static IP, but I can’t seem to access).
A fairly stressful day, but one which ended with most things working as they had the day before, and the ones that didn’t I could live without.
Monday
I woke the next day to the same dreaded message from my WordPress jail, and found a completely unresponsive FreeNAS box. Another hard reboot and dog walk, and this time everything that was working before I went to bed, was working again after the walk. Not distracted by the missing DHCP devices, I started to look at what might be causing FreeNAS to hang.
I found the solution to my problem fairly quickly in /var/log/messages:
Nov 3 23:59:35 freenas1 swap_pager_getswapspace(32): failed
Lots and lots and lots of times! I’d come across this before when running of resources in a VM, whether that be space or memory, so this was my starting position. My FreeNAS box has 32GB RAM installed (the maximum for the motherboard) which should be enough for what I was running, or at least it had been for quite some time and little had changed.
I knew I was filling up the 9X4TB RAIDZ2 and was running at around 75% of the usable capacity, taking into account the 2 disks for parity. I also knew that 80% was recommended as an upper limit as FreeNAS would seriously slow down and could start to misbehave above that threshold. Well it looked like for my configuration, that limit might be slightly lower!
I had about 5.85TB of space available, but after spending a few hours pruning I got things back up to a more respectable 7.15TB and back to 70%. I mainly deleted some Time Machine backups that I don’t expect to need, removed some old snapshots and changed some of the frequencies so I’m not keeping as many. There is still some scope to cleanse more things if required in a couple of months.
Wednesday
So where does that leave me? Well, yesterday morning there were no WordPress e-mails and everything was up and running, and that was the same today! I still have a broken AirPort Extreme, that won’t assign IP addresses to wired devices, but have started to look at a DHCP jail on FreeNAS as a short term solution (a separate blog on that to follow).
Replacing my FreeNAS box has been on the cards for several months, but the system I’m looking to build is going to cost about £3000, so it’s taking a little while to get the funds together. This has hopefully kick-started that and I’m hoping to build it early next year.
I’ve also been meaning to update my AirPort Extreme (which is probably around 6 years old and Apple no longer make them) but I wanted to wait until a few more WiFi 6 devices were available, as the current choice is limited and seems to be a little focused on gaming. If I can get a DHCP server running on FreeNAS, a new network can wait until later in 2020!
A bit of a long blog, but hopefully that’s my ‘three bad things’ out of the way for some time…
Update
So perhaps my AirPort Extreme isn’t broken, and I actually had some configuration issue on my Netgear 8-port managed switch. At least that’s my diagnosis after trying another router here.
Trackbacks/Pingbacks