Networking has long been my Achilles heel. I know the very basics, but the more complex areas of networking have been a bit puzzling to me. By the time I figured out how IPv4 works, I found IPv6 and that my ISP supports it.

Back to square one.

That didn’t stop me from learning some bits, and after 8+ years of self-hosting as a hobby, I’ve settled on a setup that works for me and overcomes common residential internet connection nuances, such as dynamic IPv4 addresses and changing IPv6 prefixes. I’m sharing these tips and tricks with the goal of helping out other hobbyists out there that happen to share a similar stack.

Background

My ISP is polite enough to provide a public IPv4 address, and allowing incoming traffic is a toggle in their online self-service. Not perfect, but at least you can do it. However, they charge about 6 EUR a month for the static IP address service, which I am not willing to pay out of principle.

They also support IPv6, which is great, and they provide you a whole /56 slice of it to play with using IPv6 prefix delegation. Unfortunately they have configured the lease time for the prefix to be incredibly short: 26 minutes!

A router reboot or short power outage usually results in the IPv4 address and IPv6 prefix changing, which is really annoying as my services become unavailable for a short time.

Dynamic DNS

A common way to overcome the dynamic IP address limitation is to sign up with a provider to set up a DNS record that changes whenever your home IP address changes. My domain registrar does not have this as a feature, and I’m not interested in using a different provider, so I went in a different direction and built a home-grown script that does the same thing.

Initially, this script relied on a public service that tells you what your IP address is, and based on that I could check if things have changed and I need to update my DNS record. This approach has one glaring catastrophic failure mode though: that provider could lie to you one day and now you’ve pointed your DNS records at the attackers’ servers. :)

I ignored that failure mode for a while, but once I learned about the effectiveness of LLM-based tooling, I decided to give it a go and to build a better solution that takes into account my setup and requirements, while at the same time saving me from the frustration of troubleshooting and debugging this in a late evening. I’m still very limited on available free time, so optimizing for that is a priority for me.

My main networking gear runs OpenWRT, and it supports running shell scripts periodically in a crontab. The router has two WAN interfaces, one for IPv4 and one for IPv6. It already knows what IP address and prefix have been assigned to it, so I don’t have to rely on an external service provider for finding this out.

Handling IPv4 addresses is simple: check the IPv4 address of the WAN interface. Query your existing DNS records, diff it, and if it has changed, push an update in a separate API call. Super simple!

With IPv6, the approach is slightly different. Instead of the WAN interface, I have to get the IPv6 address of the target machine, and make sure that it’s routable over the public internet. When you’ve checked your network settings in an IPv6 network, you may have noticed a lot of different IP addresses there, with lots of letters thrown into the mix.

Here’s an example from the machine that is serving you this blog (likely out of date though!):

    inet6 fdb3:6dad:6dce::f41/128 scope global dynamic noprefixroute 
    inet6 fdb3:6dad:6dce:0:2e0:4cff:fe0c:9ddb/64 scope global noprefixroute 
    inet6 2001:7d0:856c:4000::f41/128 scope global dynamic noprefixroute 
    inet6 2001:7d0:856c:4000:2e0:4cff:fe0c:9ddb/64 scope global dynamic noprefixroute 
    inet6 fe80::2e0:4cff:fe0c:9ddb/64 scope link noprefixroute 

The two relevant ones are the ones that start with 2001:, others are link-local or accessible over the local network only. The shorter one consists of the IPv6 prefix part, and then the unique bit at the end is a predictable suffix that the host gets. The other one also works, but is as far as I understand randomly generated and more difficult to predict when we get around to next sections.

I know that there is probably a better way to do this, but I wanted to keep things simple enough so that I can troubleshoot them if needed. It may be possible to trigger this updater script on events that WAN and WAN6 interfaces send, but I have not validated this theory.

There are many different ways to find the IPv6 address of a particular host, so the script I have just tries multiple approaches to find the one that we’re looking for.

Here’s the script in case you’re interested in setting up something similar. It reads credentials from an .env file and is built around the Zone API. On OpenWRT, the only dependency that you need to install is curl, which to my surprise was not part of the default packages list, probably to save on space.

One lesson I learned from a previous iteration of the script: if you trigger DNS record updates every minute, then Zone will actually reach out to you via e-mail telling you to cut that shit out, politely. It was just one missing if statement, and yet it caused some frustration to engineers far away. Sorry!

Predictable IP addresses

It’s a good idea to set up static IP addresses for your hosts, both for IPv4 addresses and IPv6 prefix delegation via DUID-s.

The OpenWRT GUI LuCI makes it quite simple, just set the addresses as static on the landing page for the hosts that you are interested in forwarding traffic to, and you’re done!

My recommendation here is to also set a predictable IPv6 suffix, otherwise all your IPv6 traffic rules may break once again due to this nuance.

I like to make that host number the same for both IPv4 and IPv6, quick example:

• 192.168.1.2
• 2001:7d0:854f:8e00::2

Here’s a configuration snippet example from /etc/config/dhcp, look for the hostid option:

config host
        option name 'mycoolserver'
        option ip '192.168.1.69'
        list mac '12:34:56:78:90:AB'
        option duid 'yourduidgoeshere'
        option hostid '69'

Apply with service dnsmasq restart.

In LuCI, as of OpenWRT 25.12, look for “IPv6 token”.

Note that due to a bug, it doesn’t seem to be possible to set a numeric IPv6 token via GUI, which is why you will need to add it manually in CLI using the above approach.

Port forwards, traffic rules, potato, potahtoh

Whenever you want to make a local machine accessible on the internet for IPv4, the solution is simple: set up a port forward to that particular machine, and you’re done! It’s a common enough flow for people who’ve set up game servers and the like, and well understood by more novice users.

With IPv6, port forwards don’t help. You’ll have to check one tab over at “Traffic rules” in OpenWRT GUI.

It’s a common misconception that by using IPv6 you are exposing everything to the world as each device gets its own IPv6 address, but turns out that this is not the case in most common setups. By default, OpenWRT forwards only a few types of traffic to IPv6 hosts, such as ICMP packets that make ping work between devices over IPv6 across the public internet. If you are interested in allowing IPv6 clients to access services on your local server that has an IPv6 address, you will have to explicitly allow it by adding a new traffic rule.

There’s one issue with this approach that a lot of users seem to run into: if the IPv6 prefix changes, then all my traffic rules that are pointing to a particular host are automatically broken!

Luckily there is a clever workaround implemented on OpenWRT that bypasses this issue. Assuming that you followed the previous step and set yourself up with a predictable IPv6 suffix, when setting up a traffic rule, set the target device up as ::69/-64, just replace 69 with your actual suffix. The IPv6 prefix can now change, but the ports that you’ve made accessible on this specific host will remain working.

At this point, you should be all set with a reasonably well working setup where you’ve handled the issues with dynamic IPv4 and IPv6 prefix, and you can access your services over the public internet even when things happen.

Limitations

One issue that this setup has is the fact that DNS change propagation takes time. Usually clients will pick up the new records within 5 minutes, but in my professional career I’ve seen some clients take up to 24 hours or longer to finally start sending traffic to the new DNS record. Whenever your IP address changes, there will be a mini-outage. Not catastrophic if you’re just hosting hobby projects and personal services at home, but I wouldn’t host anything mission-critical in such a setup.

When your OpenWRT device is as underpowered as mine, then you may notice that the TLS encryption overhead when curl -ing around can be significant. I have set my dynamic DNS script to run every 5 minutes, and it shows up on the CPU usage graphs on my router.

Wireguard all the things!

I know that Tailscale is a popular method of connecting up your devices and making your personal services privately accessible over that, which significantly reduces your attack surface. Being behind a few updates or not being vigilant enough is less of an issue compared to exposing your services over the public internet.

You don’t necessarily need Tailscale for that though! If you just need a way to access your services over a private and secure network, then setting up a dedicated mini PC or single-board computer is a very good starting point. Let it be the server, allow traffic to move between the clients over the Wireguard interface, and you’re all set!

Alternatively, if you have an OpenWRT router, then you can do it right there, but I found the GUI management setup to be a bit clunky compared to deploying the plain configuration files to clients. When I did do that test, I discovered quickly that my router and its single ARM CPU core with no cryptography extensions is too slow for managing my Wireguard network, with speeds topping out at 20 Mbit/s. 20. The LattePanda IOTA can easily saturate its gigabit link, as does the ThinkPad T430, and even devices like the Orange Pi Zero can handle a theoretical maximum of about 240 Mbit/s over Wireguard measured using wg-bench.

My current Wireguard host is the LattePanda V1, the most unstable computer in my fleet. With a USB adapter, it can push almost half a gigabit of traffic over Wireguard.

If you’re like me, and you like hosting your services over Docker or Podman, then instead of listening on ports for all interfaces on your containers (default behaviour when setting up port forwards), I recommend listening only on the Wireguard interface. This makes the service only accessible over Wireguard, meaning that you only need to set up one port forward and traffic rule to connect to the Wireguard network, and then you have access to all of your services. The attack surface is significantly reduced, the whole Wireguard solution is stable and very small, and unless you leak your private key, you are reasonably secure!

Here’s a snippet from a compose file showcasing how to set this up for IPv4 and IPv6:

ports:
  - 10.69.69.12:2283:2283
  - "[fded:abba:acca::12]:2283:2283"

Want to make the service available over Wireguard and over the local network directly? Just add those to the list! Note that if your local address changes and you don’t update it in the compose file, your container will refuse to start up as it cannot listen to the interface any longer, but you can mitigate that with the static IP addresses step.

When you are going with this route, it is unlikely but still possible that by the time the container starts up, the Wireguard interface is not yet up. To resolve this, you can use systemd to set Wireguard up as a dependency that you will have to wait for before the container starts up.

I manage my Wireguard connection with wg-quick@interfacename service. You can set up a systemd override for Docker, or if you manage your Docker/Podman services via systemd, then you can set it up per-service using this pattern:

# /etc/systemd/system/myimmichserver.service.d/override.conf
[Unit]
Requires=wg-quick@interfacename.service
After=wg-quick@interfacename.service

By the way, systemd overrides are also really useful for ensuring that your storage that your containers rely on is properly mounted. If my service requires the path /immich to be available and mounted, add something like this:

BindsTo=immich.mount
After=immich.mount

If you unmount the mount point, it will also properly bring down the service. The service won’t start if the mount point is missing. I’ve had the issue with containers seeing blank mount points more times than I’d like to admit, and this has eliminated this issue for me.

systemd has received a lot of hate online, and I don’t think it’s fair. The ease with which you can set up dependencies on your system, set up resource limits, make services more restricted to improve the security posture is great and allows me to and avoid all sorts of failure modes. Production services that I’m responsible for make use of these systemd features, with great results.

For services that need to be public, such as Nextcloud and its public shareable links, this approach won’t work, obviously, but for things that only you and your family members use, this is a viable approach.

Conclusion

This setup works well enough for me to confidently host my blog and self-hosted services off of it. I’ve hit a lot of paper cuts and frustrations along the way, but after following this guide, you don’t have to do the same.

Yes, VLAN-s are intentionally missing from this guide. I’ll get to them eventually, maybe by Q4 2037 given my lack of free time.

And no, IPv6 isn’t complicated, it’s just different from what everyone is used to. If we started out with IPv6 right from the get-go, we wouldn’t be having dumb arguments online.

Do you have a need for speed really fast disk performance?

Are you unwilling or unable to buy/rent a machine with fast, NVMe-based storage?

Are you OK with potential data corruption?

Then cache=unsafe is the solution to all your problems!

We had an interesting conundrum at work recently. Our platform does not use a lot of resources, but there are bursts of activity that require a lot of CPU and performant disk IO from our infrastructure. This was previously handled by manually starting some expensive AWS EC2 instances to cope with the load, but this manual process was error-prone due to the human factor (which did end up causing an actual production outage once), and AWS is stupidly expensive for what you get in return.

Around this time I also learned about a Proxmox server that we were underutilizing. My goal was to investigate the resources that we had available and to ensure that we didn’t have to think about taking any manual actions while at the same time not relying on AWS and its expensive resources.

I set up a few VM-s on the Proxmox machine, and did some testing. CPU, memory, that was all fine, but the IO-bound workloads that we had to run during those bursty periods would still be relatively slow. Not much slower than the main infrastructure provider that we were using, but slow enough for a beefy machine to not be able to handle more than a few parallel IO-heavy workloads running at the same time.

We exhausted a few other wild-ass ideas during the investigation:

• Docker on a RAM-backed storage drive
  • online resources did not inspire confidence in this working well, so we didn’t try this
• optimizing the workload to not be IO-heavy
  • unsuccessful after spending a few hours on it, the high IO was a consequence of making an intentional trade-off to reduce CPU load, and the IO requirement was much more manageable
• putting certain folders in the container itself on RAM-backed storage
  • highly container specific, and did not yield the desired results

Then one day I was browsing around Proxmox and noticed an interesting option on the virtual storage drives: setting the cache mode to unsafe. With this one trick, your VM will see really fast disk speeds up to a certain point, and it’s invisible from the perspective of your workloads, no customization needed. In a way, this is like one of the RAM-backed storage options, but for the whole VM.

The major trade-off is that an unexpected shutdown of the VM or the VM host will likely result in data corruption. This is because you’re writing everything to memory first, and eventually the writes will end up on persistent storage, whenever the disks catch up with you. Something happens while changes are in memory, and they are lost.

In our case, the data corruption risk is completely OK, as the workloads are ephemeral, the results of the work are sent to another machine immediately after completion, and the configuration of the machine is largely automated with Ansible.

One instance of our workload would usually result in writing 50 MB to disk, and we observed about 300-500 IOPS of performance from HDD-backed storage. The disks were not able to handle more than one at a time if we cared about execution time.

With the unsafe trick, and on some relatively old hardware (assume DDR3 memory), we saw numbers as high as 15K IOPS and disk throughput of 500+ MB/s. This was more than enough to handle peak loads, and the resources were always on and available on a rented server with a stable price that compared extremely well to AWS.

Cloud service providers have their benefits, sure, but when all you need is raw speed and configurability to make it happen, then owning a physical Linux server (or a few of them for redundancy) is a no-brainer, slam-dunk decision, as long as you have someone in your team that knows how to manage one. Since you’re working with Linux VM-s already in the cloud, then you already have that person in your team, don’t you? :)

Remember that failed experiment where I ran Jellyfin off of a LattePanda V1?

Do you recall all the parts where I said what this single board computer cannot do?

Yeah, I remember.

Then I took it and put the two of the most critical services running on it: the blog you’re reading right now, and my Wireguard setup.

Trust me, it makes more sense with some context. The board is incapable of doing anything else other than serving content from the eMMC module, and it has a functioning network port. It doesn’t seem to crash in these scenarios.

When I try anything else with this board, especially things that include USB connectivity, things break. This makes the board ideal for a light workload that needs to be up 24/7.

The biggest threat to my uptime is not internet connectivity or loss of power (although that did happen for the first time in a year recently), it’s me getting new ideas to try out on my setup, which results in downtime. This board is so unreliable for trying those ideas out that it removes any and all temptation to do that, resulting in a computer that has the highest chance of actually being up and running for a very long time.

To play things safe, I used an IKEA SJÖSS 20W USB-C power adapter that I got for 3 EUR, with a cheap USB-C to USB-A adapter thrown into the mix. It looks janky, but the adapter outputs 5V 3A, which makes it the beefiest power adapter that I have in my fleet for plain USB-A powered devices.

I then hit the board with some stress commands, including hitting the 2 GB of memory. It ran really well for days, no issues at all.

I also improved the cooling situation. I am now a proud owner of an assortment of M2, M2.5 and M3 screws and bits, and equipped with a Makita cordless drill, I made some mounting holes into an old aluminium server heat sink. The drilling was a complete hack job, everything was misaligned, but it was good enough. Certainly better than holding the board and heat sink together with thin velcro strips. The cooling performance is completely adequate, the board hits a maximum of 65°C with the heat sink facing down. This is well below the point at which the board starts to throttle its CPU.

The theoretical maximum Wireguard throughput on this board is about 340 Mbps, measured using the fantastic wg-bench solution.

Remember the part about the USB ports being flaky? Yeah. That didn’t stop me from getting a USB Gigabit Ethernet adapter to remove one of the main limitations of the LattePanda V1. Based off of vibe-recommendations by Claude, I got a TP-Link UE300 for its alleged low power usage and its availability at a local computer store in Estonia. It seems to work well enough, you can push gigabit speeds through it measured by iperf3, and the actual Wireguard performance that I could push through it with an actual workload was at about 420 Mbit/s, higher than indicated by the benchmark, and plenty fast for most workloads, especially in external networks that are usually slower than that.

A few hours after making that change, a HN post put some mild load on the LattePanda V1, what good timing.

As of publishing this post, the blog has been running mostly off of the LattePanda V1 for over a month now, with that gap in it being caused by contemplating getting that USB Ethernet adapter and temporarily running the blog and Wireguard off of another mini PC during that time.

Did you notice?

2026 July update

I was running out of room and network ports on my router, so I decided to stop this experiment. During all this time, there were no issues with the LattePanda V1, except when I accidentally bumped into its power cable and made it go offline. Oops.

The apartment that I live in right now has two small rooms with doors on them.

One of them is like a normal closet and that’s where the ISP passive fiber optic line comes in.

The other one is a similarly sized room with a bunch of shelves and ventilation to keep things cool. A pantry, basically.

I’ve used the closet as a networking and home server spot for a while now, with a small dedicated shelf holding everything. That shelf was a bit limiting, it could not accommodate all of my hardware that I wanted to play around with, and the closet did not have very good ventilation, which resulted in the top half of the closet getting notably warm during summer months. Not too warm, but warm enough to feel it when opening the closet.

One day I decided to realize an idea that had been in my mind ever since I started renovating this apartment: turn the pantry into a server room.

Meet the server pantry.

It’s cool.

It’s the furthest point in the apartment from the bed and my home office setup, so I can put incredibly noisy computers in it.

It only has a few spiders.

Due to the pantry being cool and dry, it makes for a good spot for hosting food and stuff. Adding <100W of heat output at the top of the room should not be an issue as hot air tends to rise, and if temperatures end up being an issue, I can open one of the two vents that are directly connected to the outside to get some fresh and cool air in. The temperature delta between the bottom and top of the pantry seems to be about 10°C, ranging from 6-16°C.

The impact of running things in a slightly chillier room is noticeable on the temperature sensors of the hardware that I put in there. A PC with overkill CPU cooler on an Intel i5-10500 did report 16°C CPU core temperatures at one point when idling, compared to 20-24°C when running in a cold corner in the living room. The LattePanda IOTA hit 30°C on the CPU cores. Hard drives hover around 30-37°C, with a maximum of 40-44°C.

It’s pretty cool.

2026 June update

I’ve now moved away from this concept due to Summer being a thing. Beer got too warm in there with 60-110 W of compute equipment being in there.

It has happened.

The cloud just stopped scaling.

Hetzner’s cloud, for now.

At this rate, my home server will actually have to become production at work, and my gaming PC has to be converted to a server because it has a whopping 32 GB of RAM and 6 good CPU cores.

With Forza Horizon 6 on the horizon, it is time for some difficult decisions…

2026-03-02 update

Shortly after publishing this post, AWS had a different type of issue with availability zones going down due to… Iranian missile strikes. I’ve wanted decentralized hosting to be more popular, but not like this.

LattePanda V1. It’s been a while, huh?

I had it as a backup server (which I blew up).

Then it got promoted to… a backup server. But then its PSU blew up.

Then it was waiting for some cool ideas at a potato enthusiast.

Now it’s back in my hands.

Unlike its modern counterpart, the LattePanda V1 is a flawed single board computer.

It’s slow.

It’s unstable if you connect power hungry USB devices to it.

It needs some cooling, but I’m too cheap to buy a properly designed heat sink for it.

It has display quirks.

It only has a slow 100 Mbps Ethernet link.

It only seems to work reliably on one side. Not even joking, the “heat sink down” configuration is the only one that works for me.

LibreELEC failed to play back any videos on it properly. The videos would play for 10 seconds, and then it would hang, no matter the encoding or hardware acceleration settings.

With hardware prices being wonky again, I decided to give this board a last chance of being useful. If new hardware is absurdly expensive, then it makes perfect sense to use what you have, no matter how slow or crummy it might be. Reduce, reuse, and only then recycle.

That’s when I decided to run a Jellyfin server off of it. Transcoding is out of the question, but serving media files over the network should still be quick enough, right?

For this experiment, I used one of the 18 TB hard drives that I’ve covered earlier. Just the single one, no redundancy here. This one drive is probably about 10x the cost of the LattePanda V1 itself, making it a perfectly reasonable choice.

For ease of troubleshooting, debugging and guaranteed eventual reinstallation, I put Fedora Server on an 128 GB USB 3.0 flash drive by Samsung. Risky move, I know, but as you might have noticed, this whole build is everything but reasonable. That left the eMMC storage as the perfect candidate for storing cache and service related files.

Cooling is solved by a random server heat sink slapped on the bottom of the LattePanda, with a few critical components like the CPU making contact with thick and soft thermal pads. The whole thing is fastened using velcro strips that I cut to a thin size in the middle so that I can route it between the components on the PCB and within the heat sink fins. The edges of the heat sink are covered with some painters’ tape that I had around to avoid shorting anything out on the board, because those parts on the board contain all sorts of metal bits and pins that have power going through them. I’m actually quite happy with that mount!

I’ve been experimenting with Podman again after Docker kept doing weird things with the v29 release, and I’ve been happy with the results, so that’s what I used on the LattePanda V1 as well. I slapped Jellyfin on it, threw in some test files and gave it a go.

Navigating the UI feels a bit slow at times, but it’s not really noticeable on an LG smart TV with a really laggy user interface.

Actual video playback that requires no transcoding works quite well, at least for smaller media sizes.

Technically, you can do transcoding on this and even utilize the tiny little integrated GPU in it, but the results are not usable. With smaller files it might be usable, but in one test I saw 7-8 FPS transcoding speeds and the server struggling to keep up, with CPU usage locked at 100%.

If we ignore all the downsides, then the LattePanda V1 can actually be a usable media server. Serving files off of a big drive does not require a lot of resources, and for that the LattePanda V1 is a solid choice. It also uses only a few watts of power on its own, so you can keep it on 24/7 guilt-free. In this build, the hard drive itself is actually the most power hungry component by a long shot (about 2/3 of the total power budget).

The 2 GB of memory is juuuuuuuust enough for this setup.

As of writing this post, it has been running for about a week, and it’s been fine. I intend to keep it running in a drawer for as long as possible just to see what will die first.

Will it be the eMMC storage?

The USB flash drive holding the operating system?

The expensive hard drive?

The power supply?

The thin velcro strips holding the cooling together?

My patience?

Don’t worry, I have backups. Well, backups of the important bits. I’m ready to lose data again.

2026-01-22 update

Okay, it was too bad of an idea.

The LattePanda V1 would occasionally just… stop.

From what I gathered by accident, it’s likely that the USB port containing the operating system would flake out and result in the system not being able to run any tools that are not in memory. I tried to move the installation to the eMMC drive, but after failing multiple times due to the display not working, or the system deciding to shut down randomly, I gave up on it. For now.

Guess I’ll have to use it in a solar-powered website project. ¯\_(ツ)_/¯

Disclosure: the review sample was provided by DFRobot, the makers of LattePanda. I am allowed to keep the review sample indefinitely, no money exchanged hands, and as always, this post covers my own thoughts and views on the product.¹

In 2023, I happened to find a LattePanda V1 for sale at a good price. Given the then-poor availability of affordable Raspberry Pi units, I got one for testing and finding potential use cases for it in my setup. However, it was just a little bit too weak for any practical uses in 2023, with its CPU and USB connectivity being just slow enough to be of less use, and the networking being capped at 100 Mbit/s.

In 2025, we have the spiritual successor to it: the LattePanda IOTA. It keeps the same form factor, but the connectivity and raw power have all received a significant jump, with the CPU performance rivalling my current home server, the trusty ThinkPad T430.

The marketing materials list all sorts of sensible use cases for it. I’m sure that it works fine for those, but I’m only interested in one thing: how close does this board get to being the perfect home server?

The perfect home server uses very little power, offers plenty of affordable storage and provides a lot of performance when it’s actually being relied upon.

In my case, low power means less than 5 W while idling, 10+ TB of redundant storage for data resilience and integrity concerns, and performance means about 4 modern CPU cores’ worth (low-to-midrange desktop CPU performance).

The model and accessories

The model I’m reviewing is the 8GB RAM/64GB eMMC one, with a Windows 11 installation on it (not activated).

Along with the review unit itself, I got sent the following accessories:

• active cooler
• M.2 M-key expansion board
• 51W PoE expansion board
• M.2 4G LTE expansion board
• UPS expansion board

The board was tested with a Lenovo 65W USB-C power adapter, because that’s what I had available. Given the specs of the board and the accessories, that should be plenty. As far as I know, USB power delivery seems to work fine and it’s not just a weird USB-C connector that requires specific voltages to work.

The M.2 NVMe SSD used in this review is a 512 GB Samsung PM9A1. I got that one from another PC that really didn’t need a boot drive that large.

Most of the testing was done with a fresh Fedora Server 43 installation, kernel version 6.17.7.

The hardware

I suggest looking at the spec sheet if you’re interested in all the fine details and available configurations.

In short:

• CPU: Intel N150, 4 cores, 4 threads, up to 3.6 GHz
• RAM: 8/16 GB (depending on model)
• Onboard storage: 64/128GB eMMC (depending on model)
• Networking: gigabit Ethernet port
• Real-time clock: yes!

The overall connectivity has been improved with the new version of this board compared to the old board. The USB ports are all fast 10 Gbit/s ones, and we have actual PCIe connectivity to play with, although the available bandwidth is quite limited with a PCIe 3.0 x1 lane available on the port that both the M.2 M-key and PoE adapter connect to.

What caught my eye was the CPU performance I’ve been proudly running an old ThinkPad T430 as a server for a while now, with some failed attempts to find a more low-power and efficient solution.

The Intel N150 is now offering similar levels of performance, but in a much smaller power envelope.

When it comes to more specialized functionalities, such as GPIO and the RP2040 microcontroller, I don’t currently have a solid use case for them, so they won’t be covered in this review. I might fancy giving them a go in the future though, it would be nice to get some environmental sensors on it to monitor the temperature and humidity of the server room (which is a closet).

Since I also don’t have an 4G LTE modem available, I did not test the associated adapter.

M.2 M-key adapter

The way you can add expansion boards to the LattePanda IOTA is quite similar to how Raspberry Pi 5 and other similar single board computers do it: you simply run a flexible cable to an adapter board, and bam, you have extra connectivity!

With the M.2 M-key adapter kit, you get the adapter itself, some mounting screws and brass stand-offs, and a tiny little flexible cable for the PCIe signal.

The link speed is PCIe 3.0, with one lane available. In theory, this means a maximum of 1 GB/s of throughput. In practice and with this board and SSD combination, I got a maximum of ~810 MB/s. I expect some levels of losses with these types of setups, so in my view this seems normal. For the test, I just did a dd if=/dev/nvme0n1 of=/dev/null bs=8M status=progress. The SSD itself supports up to 4 lanes of PCIe connectivity so that should not be a limiting factor here.

The lovely part about M.2 NVMe ports is that you can use it for a lot of off-label use cases.

Fancy some SATA ports? There’s an adapter for that.²

Or a network card? Some fancy AI accelator thingy? Or a full-sized GPU? Anything will work (probably), as long as the cables and adapters are high quality, and you provide extra power to the device through other means.

PoE expansion board

The only device on my network that is connected over PoE is currently an Ubiquiti Wi-Fi access point, and that is unlikely to change in the near future because that would require a full replacement of my networking gear.³

However, I still gave this board a quick go, and I’m happy to report that it also works as an additional standalone Ethernet port. The Ethernet controller seems to be similar or the same as on the main board, and it shows up as a separate networking device. Both are Realtek NIC-s (Realtek Semiconductor Co., Ltd. RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller), and they work with the r8169 driver. Realtek has a spotty compatibility story overall on Linux from what I’ve read, but this one seems to work fine on Fedora Server 43.

I was very close to pulling the trigger and turning it into a beefy router so that I can finally move my Wireguard networks on the router as my current one cannot do more than 20 Mbit/s of Wireguard traffic, but I didn’t end up going through with that idea because of how well the SBC did in other areas.

UPS expansion board

As some of you might know, I’m a fan of playing with fir- 18650 Li-ion battery cells, and I’m hoping to one day build a solar-powered server of my own (of which there are many examples).

I took some spare 18650 cells that came from an old ThinkPad battery, made sure that the voltages are more-or-less the same, and threw them on the board.

Connecting the UPS board with the standoffs was fine, but the cable connecting it with the SBC was finicky. I triple-checked that the connector was the right way, but had to still use an uncomfortable amount of force to connect it all up. The battery cells themselves sit snugly on the board, and unless you drop the board, they should not fall out on their own. You’d still want to build a case around it if you’re going to actually put it to use in rough environments.

The manual for the UPS board emphasizes that it only works on Windows 10/11, and sadly that seems to be the case, the UPS does not seem to show up as an USB-listed device, and tools like NUT did not find anything to monitor with a quick 5-minute investigation.

The UPS board also has an interesting selection of switches that you can use to adjust the behaviour of the board, like automatically turning the board on when power comes back on, and setting an 80% battery charge limit. The first one was not really necessary to use, the board would follow whatever setting you have enabled on the SBC itself. I configured mine via UEFI settings to automatically turn on with a power adapter connected, and that worked here as well.

The run time of your LattePanda IOTA with the UPS expansion board will heavily depend on your workloads and quality of your battery cells. Mine were used cells, and then I hit the board with stress -c 4 to create some load on it. It ran for over an hour like that, and then I got bored and wanted to proceed with testing other accessories. The marketing materials mention up to 8 hours of runtime, and I suspect that with good Li-ion cells and workloads where you idle most of the time, it will likely be achievable.

The board seems to trigger a hard shutdown on Linux because the host OS is not aware of a battery being connected. Not that catastrophic for most modern filesystems and database engines, but something to consider in your own workloads in case they are Linux-based.

The UPS board seems to handle power connection and disconnection events well enough, it did not do anything weird when repeatedly plugging and unplugging the USB-C cable.⁴

Based on the readings from a wall outlet energy meter, the board uses up to 20W when charging the cells. It’s possible for the board to pull more than that with a maximum CPU load and connected peripherals, so I wonder if that may be an issue with more intense usage scenarios.

During charging and discharging cycles, even under heavy loads, the battery cells did not get hot and were at best warm to touch.

Ethernet

It’s gigabit.

Fine for my use case given that I still live in 2006 and only have devices that support gigabit Ethernet speeds at best (excluding the Ubiquiti Wi-Fi AP), but certainly less than some competing products.

eMMC and USB storage

Compared to the LattePanda V1, the USB port performance is actually decent for my use case. I can connect up to three USB-connected storage devices to the board, so that’s exactly what I did.

I set up three different USB-connected devices:

• USB hard drive (Seagate Basic)
• USB SATA SSD (Samsung QVO 4TB in ICY BOX USB-SATA adapter)
• USB NVMe SSD (512 GB Samsung PM9A1 with some random cheap USB to M.2 NVMe adapter)

For each device (including on-board eMMC device), I ran a while true; do dd if=/dev/X of=/dev/null bs=8M status=progress; sleep 5; done, which puts a sequential read workload on all the drives in an infinite loop.

After about 72 TB of data read in less than 24 hours, I checked the kernel logs and there were no stability issues whatsoever. The NVMe SSD started throttling due to heat, which was expected with that cheap adapter.

Assuming no issues with any cables and adapters, the USB ports seem to be solid enough for running storage devices off of. Yes, it can be a horrible idea in some use cases, but at the same time my ThinkPad T430 has been excellent with USB-based storage, and that’s with one of the USB ports being coffee-stained!

The eMMC chip is also more performant compared to the previous iteration, with sequential read speeds averaging around 316 MB/s, writes around 175 MB/s, and average read latency being around 0.15 ms. Certainly good enough for a boot drive.

Display connectivity

The LattePanda V1 struggled with larger displays, and when I gave it a go during this review, it would not properly display an image on my 3440x1440p monitor.

The LattePanda IOTA just did it, at 60 Hz. On Fedora Workstation and GNOME, the experience was smooth. Once you start doing things in the browser, like video playback, the situation is less optimal, but as a makeshift desktop PC it is alright for most low/mid-range activities.

Windows compatibility

The board came with a Windows 11 installation (not activated). As is tradition with Windows, the initial impressions are horrible, update processes running in the background made the active cooler go wild and the device felt sluggish.

But after that process is done, the experience is not bad at all if you look past the OS being Windows.

I did not do a thorough investigation and I suggest formatting the device boot drive either way when receiving it, but the Windows 11 installation looked clean enough, with no obvious bloatware.

Linux compatibility

The LattePanda V1 had some quirks. The performance was iffy, and you had to specify a Linux kernel parameter on first boot so that Fedora Linux does not confuse the optional display interface to be an always-connected primary display.

The previous version also didn’t include a real-time clock (RTC) by default, which meant that it was impossible to schedule some systemd timers as the time would always jump on boot years ahead on distros like Fedora Server. I got stuck in a reboot loop with a scheduled reboot job that way, was not fun to recover from.

With the LattePanda IOTA, I have not observed any weird oddities and quirks with it. Even the kernel logs don’t show anything that’s problematic, and the RTC is handy to have around as that helps avoid the issue mentioned above.

Active cooler

With the LattePanda V1, the cooler was not strictly required, but strongly recommended if you were going to use the board with moderate to high sustained loads. My solution was to slap an inappropriately sized heat sink to it with a thermal pad and zip ties and/or velcro strips, which looked horrible, because it was.

With the LattePanda IOTA, the cooler is now a mandatory part of the assembly. It can be fitted with either a passive cooler, a case with passive cooling, or an active cooler.

The active cooler does a good job of keeping the board cool, but it does get super loud at higher loads. The default fan curve is very primitive, with the fan changing it speeds in big and sudden increments. Bursty workloads certainly feel bursty with this fan. You will not want to be in the same room with this active cooler.

The sound profile is very similar to a thin and light laptop, and the fan has a very strong high-pitched whine to it.

Here’s an audio recording of the noise under heavy load if you’re interested (MP3 file). Recorded using a Google Pixel 8a.

You can mitigate the active cooler noise issue by reducing the CPU clock speed by setting a lower power limit in UEFI settings, or on Linux, setting a lower CPU performance ceiling using intel_pstate driver max_perf_pct option once on boot. This comes at the obvious cost of some raw performance, but given that CPU power scales non-linearly, you may not even notice it that much.

If you are sensitive to fan noise, then do get the passive cooler and slap a Noctua fan on it, it will likely be a much better experience with both the cooling performance and noise levels.

Oh, and fun fact: I got so carried away with testing that I actually forgot to remove the plastic film on the larger thermal pad that cools supporting components.

And then I did about 24 hours of stress testing with that arrangement.

I can confirm that the design of the board is idiot-proof, as I did not actually notice any severe throttling or thermal issues with that mistake. You can actually see the plastic film being present in a few photos of the board in this review. I still can’t believe that after all these years I ended up making that one mistake that you usually see online in tech support gore posts.

Power consumption

The idle power consumption of the LattePanda IOTA seems to be around 4.0W, which is more than the Raspberry Pi 5 8GB with its power consumption being around 3.2W. Slightly higher compared to that, but lower than most x86 mini PC-s with idle power consumption typically in the range of 6-14W.

During the disk read speed stress test, I saw a maximum of 24.4W pulled from the wall.

With the disk read stress test and a full CPU stress test, I saw a peak of 36.3W, with it quickly dropping down as the CPU settled down at a lower clock speed.

Home server

This board came surprisingly close to my perfect home server criteria that I had outlined earlier this year.

Less than 5W when idling? Check.

10+ TB of redundant storage? Check.

4 modern cores’ worth of CPU performance? Check.

Enough performance during bursty workloads? So far, yes.

I then installed a fresh copy of Fedora Server 43 and moved all my home server workloads to it. The eMMC storage is used as a boot drive, journald writes are disabled, workloads requiring good latency and speed are on the 512GB NVMe SSD, and bulk storage is connected via two existing USB-SATA adapters taken from one of those WD Elements/MyBook external hard drive enclosures.

Then it just worked. No issues.⁵

The drop in the overall power consumption of my whole home server and networking stack was also immediately noticeable.

Here are my observations of the CPU performance and behaviour after hitting it with an all-core CPU load:

• 2.9 GHz for a short time period (10-15 seconds), with CPU hovering around 80°C
• 2.2-2.3 GHz after that, with the CPU dropping to around 70°C

I have seen the CPU hit around 3.6 GHz with a single core load while there is nothing running in the background, but during my normal home server operations the cores are doing enough work across all 4 cores, so that doesn’t happen all that often, and 2.9 GHz is the ceiling for single core performance.

The only limiting factor so far has been the 8 GB of memory on my review unit, but on the bright side that limitation forced me to review the memory usage of some of the jobs that I run on my home server, which ended up with me finding a few resource hogs and then fixing them all up. Now I can run about 30 Docker containers of various resource consumption on a single board computer, and with less than 4GB of RAM used. I set up an 8GB swap file on the SSD, just in case.

Thanks to the relatively small boot drive, I also learned that even if you move the Docker data-root folder to another location, containerd will still clutter up your boot drive, so you’ll have to change that path in its /etc/containerd/config.toml file root setting.

I’m genuinely impressed with how well the LattePanda IOTA runs as a home server. The board isn’t really designed with that use case in mind, and I suspect that the Intel N150 might be doing most of the heavy lifting here, but still, very impressive!

Is it the perfect home server? No, but it’s pretty damn close to my definition of it.

UEFI settings tour

For those interested in what options are available on the board via its UEFI settings, here are some screenshots of the settings.⁶

• Main view
• Advanced
• Advanced -> ACPI
• Advanced -> CPU configuration
• Advanced -> Super IO configuration
• Advanced -> Serial port 1 configuration
• Advanced -> SMART Fan Control
• Advanced -> Trusted Computing
• Advanced -> NVMe configuration (no device connected at time of screenshot, oops)
• Advanced -> Power configuration
• Advanced -> USB configuration
• Advanced -> Serial Port console redirection
• Advanced -> SDIO configuration
• Advanced -> Realtek PCIe Ethernet controller
• Chipset
• Chipset -> System Agent (SA) configuration
• Chipset -> Device configuration
• Security
• Security -> Secure Boot
• Boot
• Save & Exit

Conclusion

If the LattePanda IOTA with its adapters fits your project requirements, you’re aware of its limitations, and the price is right, then I believe it’s a solid choice for your next project. My testing didn’t immediately break it, even when I forgot to remove the plastic film on one of the thermal pads.

The current pricing of it and its accessories seem to be roughly in the ballpark of the Raspberry Pi 5 8GB (based on prices in Estonia). Boards like the Zimaboard 2 (have not tested it myself) are more expensive, but they’re also catering to a slightly different audience and have better specs, like 2.5G Ethernet ports and SATA ports with power delivery suitable for running two 3.5" hard drives straight from the board.

It’s hard to beat the bargain that you can get from a used mini PC or NAS, but it won’t come with the charm, low power consumption and bragging rights that a single board computer gets you, especially if you’re using it for an off-label use case like I am.⁷

In the meantime, I’ll keep rocking it as a home server. In case something noteworthy happens, I’ll update this post, which is brought to you by the very same LattePanda IOTA at the time of publishing.

2025-12-12 update

About a month after switching my home server workloads to it, the LattePanda IOTA is still rock solid. No issues with USB HDD stability as well.

The only complaint that carries over from the review is the fact that the active cooler is still way too loud. I’m seriously considering getting the passive cooler for it to shut it up. I can hear it from the bedroom while the IOTA is in a closet on the other side of the apartment!

I also played around with Intel RAPL settings, which allow you to control the power budget of the CPU on the fly.

The power limit is 10W by default, and short term power limit is 20W. You can control it via these toggles:

/sys/class/powercap/intel-rapl/intel-rapl:0/constraint_0_power_limit_uw
/sys/class/powercap/intel-rapl/intel-rapl:0/constraint_1_power_limit_uw

Get current values:

cat /sys/class/powercap/intel-rapl/intel-rapl:0/constraint_0_power_limit_uw
cat /sys/class/powercap/intel-rapl/intel-rapl:0/constraint_1_power_limit_uw

Set a new limit of 8W to both:

echo 8000000 > /sys/class/powercap/intel-rapl/intel-rapl:0/constraint_0_power_limit_uw
echo 8000000 > /sys/class/powercap/intel-rapl/intel-rapl:0/constraint_1_power_limit_uw

These reset every boot, so you should probably set a one-shot systemd unit to call these early in the boot cycle.

The LattePanda IOTA is the most exciting yet boring addition to my setup. It just works.

I take pride in hosting my blog on a 13-year old ThinkPad acting as a home server, but sometimes it’s kind of a pain. It’s only fair that I cover the downsides of this setup in contrast to all the positives.

Yesterday, I happened to notice that a connection to a backup endpoint was gone. Okay, happens sometimes. Then I went into the router and noticed that hey, that’s odd, there’s no WAN6 connection showing up. All gone. Just as if I had gone back to a crappy ISP that only provides IPv4!

Restarting the interface did not work, but a full router restart worked.

Since the IPv4 address and IPv6 prefix are all dynamic, that meant that my DNS entries had just gone stale. I do have a custom DNS auto-updater script for my DNS provider, but DNS propagation takes time. Luckily not a lot of time, my uptime checker only reported downtime of 5-15 minutes, depending on the domain.

Here’s what it looked like on OpenWRT.

Impact to my blog? Not really noticeable, since IPv4 kept trucking along. Perhaps a few IPv6-only readers may have noticed this.¹

I can always move to a cheap VPS or the cloud at a moments’ notice, but where’s the fun in that? I can produce AWS levels of uptime at home, thankyouverymuch!

I think I’ll now need to figure out some safeguards, even if it means scheduling a weekly router reboot if the WAN6 interface is not up for X amount of time.

That, and better monitoring.

Nextcloud. I really want to like it, but it’s making it really difficult.

I like what Nextcloud offers with its feature set and how easily it replaces a bunch of services under one roof (files, calendar, contacts, notes, to-do lists, photos etc.), but no matter how hard I try and how much I optimize its resources on my home server, it feels slow to use, even on hardware that is ranging from decent to good. Then I opened developer tools and found the culprit.

It’s the Javascript.

On a clean page load, you will be downloading about 15-20 MB of Javascript, which does compress down to about 4-5 MB in transit, but that is still a huge amount of Javascript. For context, I consider 1 MB of Javascript to be on the heavy side for a web page/app.

Yes, that Javascript will be cached in the browser for a while, but you will still be executing all of that on each visit to your Nextcloud instance, and that will take a long time due to the sheer amount of code your browser now has to execute on the page.

A significant contributor to this heft seems to be the core-common.js bundle, which based on its name seems to provide some common functionality that’s shared across different Nextcloud apps that one can install. It’s coming in at 4.71 MB at the time of writing.

Then you want notifications, right? NotificationsApp.chunk.mjs is here to cover you, at 1.06 MB.

Then there are the app-specific views. The Calendar app is taking up 5.94 MB to show a basic calendar view.

Files app includes a bunch of individual scripts, such as EditorOutline (1.77 MB), previewUtils (1.17 MB), index (1.09 MB), emoji-picker (0.9 MB which I’ve never used!) and many smaller ones.

Notes app with its basic bare-bones editor? 4.36 MB for the notes-main.js!

This means that even on an iPhone 13 mini, opening the Tasks app (to-do list), will take a ridiculously long time. Imagine opening your shopping list at the store and having to wait 5-10 seconds before you see anything, even with a solid 5G connection. Sounds extremely annoying, right?

I suspect that a lot of this is due to how Nextcloud is architected. There’s bound to be some hefty common libraries and tools that allow app developers to provide a unified experience, but even then there is something seriously wrong with the end result, the functionality to bundle size ratio is way off.

As a result, I’ve started branching out some things from Nextcloud, such as replacing the Tasks app with using a private Vikunja instance, and Photos to a private Immich instance. Vikunja is not perfect, but its 1.5 MB of Javascript is an order of magnitude smaller compared to Nextcloud, making it feel incredibly fast in comparison.

However, with other functionality I have to admit that the convenience of Nextcloud is enough to dissuade me from replacing it elsewhere, due to the available feature set comparing well to alternatives.

For now.

I’m sure that there are some legitimate reasons behind the current state, and overworked development teams and volunteers are unfortunately the norm in the industry, but it doesn’t take away the fact that the user experience and accessibility suffers as a result.

I’d like to thank Alex Russell for writing about web performance and why it matters, with supporting evidence and actionable advice, it has changed how I view websites and web apps and has pushed me to be better in my own work. I highly suggest reading his content, starting with the performance inequality gap series. It’s educational, insightful and incredibly irritating once you learn how crap most things are and how careless a lot of development teams are towards performance and accessibility.

This post is NOT sponsored, the products were bought with my hard-earned money.

I’ve been running a full SSD storage setup for a few years in my home server and I’ve been happy with it, except for the storage anxiety that I get with running small pools of fast storage, which is why I started looking at how the hard drive market is doing.

Half of tech YouTube has been sponsored by companies like ServerPartDeals, so they were one of the first places I looked at, but they seem to only operate within the US and the shipping+taxes destroy any price advantages from ordering there to Estonia (which is in Europe).

At some point I stumbled upon datablocks.dev, which seems to operate within a similar niche, but in Europe and on a much smaller scale. What caught my eye were their white label hard drive offerings. Their website has a good explanation on the differences between recertified and white label hard drives. In short: white label drives have no branding, have no or very low number of power-on hours, may have small scratches or dents, but are in all other aspects completely functional and usable.

White label drives also have a price advantage compared to branded recertified drives. Here’s one example with 18 TB drives, the recertified one is 16.7% more expensive compared to the white label one, and the only obvious difference seems to be the sticker on the drive. I highly suspect that the white label one is also manufactured by Seagate based on the physical similarities.

I took some time to think things over and compared the pricing of various drives. The drives were all competitively priced between each other, with the price per terabyte hovering around 13 EUR/TB, so it didn’t matter much which drive size you picked, you’d still get a pretty solid deal. It was also a better deal compared to using an WD Elements/My Book drive of the same size.

I decided to go with two 18 TB hard drives. I considered buying the 20 TB or 22 TB capacities, but decided to go with 18 TB because it’s the largest single hard drive that I can easily and quickly buy a replacement for in the form of a WD Elements/My Book drive.

The stock on datablocks.dev is quite volatile, the drives are in stock when new batches arrive, but they can also quickly go out of stock. I saw this live with the 22 TB hard drives, one day there are 35 left, the next day there can be 7 left, and then only one lone drive.

At the time of writing, the 18 TB model that I bought is out of stock, so my choice to go with a slightly smaller but more easily replaceable one is validated.

For those that have followed my blog for a while will know that I’m a huge fan of all-SSD server builds, especially this one by Jeff Geerling that I still consider building from time to time. If I dislike noise, higher power usage and slower performance, then why did I get the hard drives? It’s simple, really: I now have an actual closet that I can stash my home server in, meaning that noise isn’t that big of a worry, and as long as my home server takes about the same amount of power as my refrigerator or dishwasher, then that’s fine. SSD prices still haven’t gone down as much as I’ve hoped over the years, so the all-SSD build ideas that I have are way outside my budget.

The drives arrived in a reasonable time window. The packaging was adequate, although I was slightly concerned with the cardboard box showing signs of something hitting it hard. The drives were packaged within sealed antistatic bags, and with ample bubble wrap surrounding them.

Just as described, the drives did have slight scratches and very minor dents in them, but in all other aspects they looked like new.

Before putting them to use, I formatted the drives using badblocks. It took a full 24 hours to do a full drive write. The write performance peaked at 275 MB/s and slowed down to 123 MB/s at the end, which is expected.¹

I also had to choose a larger block size for badblocks because otherwise it could not handle the drive, resulting in the command being badblocks -wsv -b 8192 /dev/sdX.

I unfortunately did not save the SMART data from the time I received the drives, but the contents were as expected, there were no more than a few power on hours and other metrics were OK. Keep in mind that it’s also possible to reset SMART data on a drive so this information cannot be taken at face value.

The drives are noisy, as expected. They run at 7200 RPM and do the usual clicks and clacks that a normal hard drive does. If this bothers you, use foam to fix it. The soft side of a sponge can work just as well.

With these drives I’ve now followed my own advice and tiered my storage: two 1 TB SSD-s for the things that benefit from good speed and latency (databases, containers), and 18 TB hard drives for bulk storage, backups and less frequently used data. Coming from an all-SSD build, I expected the performance to drop in day-to-day operations, but in most cases I cannot tell a difference. My family photos load just fine, media plays back well, and backups take slightly longer, which isn’t noticeable due to them running during the night. Only when I look at the Prometheus node exporter graphs do I notice that sometimes the server is waiting behind the disks a bit more due to higher iowait.

The power usage did shoot up as a result, roughly 10-20 W. Not ideal, but my whole networking and home server setup is idling at below 45 W, and I’ve had less efficient home servers in the past, so it’s not that big of a deal.

In this configuration, the drives run quite cool. During formatting on a hot day, I saw them go up to a maximum of 51°C, but in general use they sit at around 38-42°C.

Overall, I’m reasonably happy with the drives. I expect these to last me at least 5 years, and I’m probably going to switch one of the drives out a bit sooner to reduce the risk of a full drive pool failure. They’ve made it the first 50 days, so that’s good!

Oh, and here’s the smartctl -ax output for the disks after running them for about two months:

• disk 1
• disk 2

2026-02-13 update

6 months later, and my drives are still doing well. smartctl -ax output:

• disk 1
• disk 2

In the meantime, what did die on me was a Seagate Portable 4TB drive that I used mainly for backups. :)

A Latvian blogger that I follow also had a pleasant experience with datablocks.dev, and they wrote about it here.

I use Uptime Kuma to check the availability of a few services that I run, with the most important one being my blog. It’s really nice.

Today I wanted to set it up on a different machine to help troubleshoot and confirm some latency issues that I’ve observed, and for that purpose I picked the cheapest ARM-based Hetzner Cloud VM hosted in Helsinki, Finland.

Hetzner provides a public IPv6 address for free, but you have to pay extra for an IPv4 address. I didn’t want to do that out of principle, so I went ahead and copied my Docker Compose definition over to the new server.

For some reason, Uptime Kuma would start up on the new IPv6-only VM, but it was unsuccessful in making requests to my services, which support both IPv4 and IPv6. The requests would time out and show up as “Pending” in the UI, and the service logs complained about not being able to deliver e-mails about the failures.

I confirmed IPv6 connectivity within the container by running docker exec -it uptime-kuma bash and running a few curl and ping commands with IPv6 flags, had no issues with those.

When I added a public IPv4 address to the container, everything started working again.

I fixed the issue by explicitly disabling the IPv4 network in the Docker Compose service definition, and that did the trick, Uptime Kuma made successful requests towards my services. It seems that the service defaults to IPv4 due to the internal Docker network giving it an IPv4 network to work with, and that causes issues when your machine doesn’t have any IPv4 network or public IPv4 address associated with it.

Here’s an example Docker Compose file:

name: uptime-kuma
services:
  uptime-kuma:
    container_name: uptime-kuma
    networks:
      - uptime-kuma
    ports:
      - 3001:3001
    volumes:
      - /path/to/your/storage:/app/data
    image: docker.io/louislam/uptime-kuma
    restart: always
networks:
  uptime-kuma:
    enable_ipv6: true
    enable_ipv4: false

That’s it!

If you’re interested in different ways to set up IPv6 networking in Docker, check out this overview that I wrote a while ago.

My evenings of absent-minded local auction site scrolling¹ paid off: I now own a Lenovo ThinkCentre M900 Tiny.

It’s relatively old, being manufactured in 2016², but it’s tiny and has a lot of useful life left in it.

It’s also featured in the TinyMiniMicro series by ServeTheHome.

I managed to get it for 60 EUR plus about 4 EUR shipping, and it comes with solid specifications:

• CPU: Intel i5-6500T
• RAM: 16GB DDR4
• Storage: 256GB SSD
• Power adapter included

The price is good compared to similar auctions, but was it worth it?

Yes, yes it was.

I have been running a ThinkPad T430 as a server for a while now, since October 2024. It served me well in that role and would’ve served me for even longer if I wanted to, but I had an itch for a project that didn’t involve renovating an apartment.³

Power usage

One of my main curiosities was around the power usage. Will this machine beat the laptop in terms of efficiency while idling and running normal home server workloads? Yes, yes it does.

While booting into Windows 11 and calming down a bit, the lowest idle power numbers I saw were around 8 W. This concludes the testing on Windows.

On Linux (Fedora Server 42), the idle power usage was around 6.5 W to 7 W. After running powertop --auto-tune, I ended up getting that down to 6.1 W - 6.5 W. This is much lower compared to the numbers that ServeTheHome got, which were around 11-13 W (120V circuit). My measurements are made in Europe, Estonia, where we have 240V circuits.

You may be able to find machines where the power usage is even lower. Louwrentius mada an idle power comparison on an HP EliteDesk Mini G3 800 where they measured it at 4 W. That might also be due to other factors in play, or differences in measurement tooling.

During normal home server operation with 5 SATA SSD-s connected (4 of them with USB-SATA adapters), I have observed power consumption being around 11-15 W, with peaks around 40 W. On a pure CPU load with stress -c 8, I saw power consumption being around 32 W. Formatting the internal SATA SSD added 5 W to that figure.

USB storage, are you crazy?

Yes. But hear me out.

Back in 2021, I wrote about USB storage being a very bad idea, especially on BTRFS.

I’ve learned a lot over the years, and BTRFS has received continuous improvements as well. In my ThinkPad T430 home server setup, I had two USB-connected SSD-s running in RAID0 for over half a year, and it was completely fine unless you accidentally bumped into the SSD-s.

USB-connected storage is fine under the right circumstances:

• the cables are not damaged
• the cables are not at a weird angle or twisted
  • I actually had issues with this point, my very cool and nice cable management resulted in one disk having connectivity issues, which I fixed by relieving stress on the cables and routing them differently
• the connected PC does not have chronic overheating issues
• the whole setup is out of the reach of cats, dogs, children and clumsy sysadmin cosplayers
• the USB-SATA adapters pass through the device ID and S.M.A.R.T information to the host
  • the device ID part especially is key to avoiding issues with various filesystems (especially ZFS) and storage pool setups
  • the ICY BOX IB-223U3a-B is a good option that I have personally been very happy with, and it’s what I’m using in this server build
  • a lot of adapters (mine included) don’t support running SSD TRIM commands to the drives, which might be a concern
    • has not been an issue for over half a year with those ICY BOX adapters, but it’s something to keep in mind
• you are not using an SBC as the home server
  • even a Raspberry Pi 4 can barely handle one USB-powered SSD
    • not an issue if you use an externally powered drive, or an USB DAS

After a full BTRFS scrub and a few days of running, it seems fine. Plus it looks sick as hell with the identical drives stacked on top. All that’s missing are labels specifying which drive is which, but I’m sure that I’ll get to that someday, hopefully before a drive failure happens.

In a way, this type of setup best represents what a novice home server enthusiast may end up with: a tiny, power-efficient PC with a bunch of affordable drives connected.

Less insane storage ideas for a tiny PC

There are alternative options for handling storage on a tiny 1 liter PC, but they have some downsides that I don’t want to be dealing with right now.

An USB DAS allows you to handle many drives with ease, but they are also damn expensive. If you pick wrong, you might also end up with one where the USB-SATA chip craps out under high load, which will momentarily drop all the drives, leaving you with a massive headache to deal with. Cheaper USB-SATA docks are more prone to this, but I cannot confirm or deny if more expensive options have the same issue. Running individual drives sidesteps this issue and moves any potential issues to the host USB controller level.

There is also a distinct lack of solutions that are designed around 2.5" drives only. Most of them are designed around massive and power-hungry 3.5" drives. I just want to run my 4 existing SATA SSD-s until they crap out completely.

An additional box that does stuff generally adds to the overall power consumption of the setup as well, which I am not a big fan of. Lowering the power consumption of the setup was the whole point!

I can’t rule out testing USB DAS solutions in the future as they do seem handy for adding storage to tiny PC-s and laptops with ease, but for now I prefer going the individually connected drives route, especially because I don’t feel like replacing my existing drives, they still have about 94% SSD health in them after 3-4 years of use, and new drives are expensive.

Or you could go full jank and use that one free NVMe slot in the tiny PC to add more SATA ports or break out to other devices, such as a PCIe HBA, and introduce a lot of clutter to the setup with an additional power supply, cables and drives.

Or use 3.5" external hard drives with separate power adapters. It’s what I actually tried out back in 2021, but I had some major annoyances with the noise.

Miscellaneous notes

Here are some notes on everything else that I’ve noticed about this machine.

The PC is quite efficient as demonstrated by the power consumption numbers, and as a result it runs very cool, idling around 30-35 °C in a ~22-24 °C environment. Under a heavy load, the CPU temperatures creep up to 65-70 °C, which is perfectly acceptable. The fan does come on at higher load and it’s definitely audible, but in my case it runs in a ventilated closet, so I don’t worry about that at all.

The CPU (Intel i5-6500T) is plenty fast for all sorts of home server workloads with its 4 CPU cores and clock speeds of 2.7-2.8 GHz under load.

The UEFI settings offered a few interesting options that I decided to change, the rest are set to default.

There is an option to enable an additional C-state for even better power savings.

For home server workloads, it was nice to see the setting to allow you to boot the PC without a keyboard being attached, found under “Keyboardless operation” setting. I guess that in some corporate environments disconnected keyboards are such a common helpdesk issue that it necessitates having this option around.

Closing thoughts

I just like these tiny PC boxes a lot. They are tiny, fast and have a very solid construction, which makes them feel very premium in your hands. They are also perfectly usable, extensible and can be an absolute bargain at the right price.

With solid power consumption figures that are only a few watts off of a Raspberry Pi 5, it might make more sense to get a TinyMiniMicro machine for your next home server. I’m definitely very happy with mine.

2025-06-08 update

The USB storage idea was a bit too insane. I saw sporadic errors on random drives, suggesting that there is an issue with the PC USB ports under long-term use. It wasn’t noticeable at all until I explicitly looked at the kernel logs, just to be safe. Good call on my part.

As a result, I have switched back to my trusty ThinkPad T430.

Some logs that demonstrate the issues, collected using journalctl -o short-precise -k -p 4:

Jun 08 01:33:32.318310 lilnas kernel: usb 2-5: cmd cmplt err -2
Jun 08 01:33:32.318603 lilnas kernel: usb 2-5: cmd cmplt err -2
Jun 08 01:33:32.320109 lilnas kernel: I/O error, dev sdd, sector 1825796880 op 0x0:(READ) flags 0x80700 >
Jun 08 01:33:32.320775 lilnas kernel: I/O error, dev sdd, sector 1825795856 op 0x0:(READ) flags 0x84700 >
Jun 08 01:33:32.321444 lilnas kernel: I/O error, dev sdd, sector 1825794832 op 0x0:(READ) flags 0x84700 >
Jun 08 01:33:32.322078 lilnas kernel: I/O error, dev sdd, sector 1825793808 op 0x0:(READ) flags 0x84700 >
Jun 08 01:33:32.322746 lilnas kernel: I/O error, dev sdd, sector 1825792784 op 0x0:(READ) flags 0x84700 >
Jun 08 01:33:32.323409 lilnas kernel: I/O error, dev sdd, sector 1825791760 op 0x0:(READ) flags 0x84700 >
Jun 08 01:33:32.324049 lilnas kernel: I/O error, dev sdd, sector 1825790736 op 0x0:(READ) flags 0x84700 >
Jun 08 01:33:32.324706 lilnas kernel: I/O error, dev sdd, sector 1825789712 op 0x0:(READ) flags 0x84700 >
Jun 08 01:33:32.324736 lilnas kernel: btrfs_repair_io_failure: 982 callbacks suppressed
Jun 08 01:41:38.201307 lilnas kernel: usb 2-5: cmd cmplt err -2
Jun 08 01:41:38.201614 lilnas kernel: usb 2-5: cmd cmplt err -2
Jun 08 01:41:38.201892 lilnas kernel: usb 2-5: cmd cmplt err -2
Jun 08 01:41:38.202158 lilnas kernel: usb 2-5: cmd cmplt err -2
Jun 08 01:41:38.202529 lilnas kernel: usb 2-5: cmd cmplt err -2
Jun 08 01:41:38.202814 lilnas kernel: usb 2-5: cmd cmplt err -2
Jun 08 08:33:15.572984 lilnas kernel: usb 2-4: cmd cmplt err -2

I moved recently, and so did my home server. You might have noticed it due to the downtime.

This time I have built a dedicated shelf for it, which allows for more flexibility and room for additional expensive ideas.

The internet connection is a fiber line, which is fantastic for a place that’s generally considered to be in the countryside. I had to hire a guy at the last place in Tallinn (capital of Estonia) to pull a fiber line from the basement to the apartment, with my own money, so I’m very happy that I don’t have to do it here.

And yes, the ThinkPad T430 is still a solid home server. I had an issue with my battery calibration script resulting in the machine being turned off, but I fixed it by disabling it, at the cost of the battery probably dying soon. Seems like a tlp and/or Linux kernel issue that has surfaced recently, as it also happened on a different ThinkPad laptop when I last tried it. I can’t really remove the battery, because the “power on with AC attach” setting only works when the battery is connected and charged.

The server/wardrobe/closet room is slightly chillier compared to the rest of the environment, meaning that the temperatures are also slightly lower. I also have an option to do some crazy ventilation experiments in the winter, but that will have to wait for a bit, mainly because it’s spring.

I’m genuinely surprised that the Wi-Fi 5 signal is coming through the closet quite adequately, with the whole apartment being covered with at least 50 Mbit/s speeds, and over 300 Mbit/s when near the closet, which is about the maximum speed that I can achieve from the access point in ideal conditions.

I’ve changed my home server setup a lot over the past decade, mainly because I keep changing the goals all the time.

I’ve now realized why that keeps happening.

I want the perfect home server.

What is the perfect home server? I’d phrase it like this:

The perfect home server uses very little power, offers plenty of affordable storage and provides a lot of performance when it’s actually being relied upon.

In my case, low power means less than 5 W while idling, 10+ TB of redundant storage for data resilience and integrity concerns, and performance means about 4 modern CPU cores’ worth (low-to-midrange desktop CPU performance). I seem to only ever get one or two at most.

Low power usage? Your performance will likely suffer, and you can’t run too many storage drives. You can run SSD-s, but they are not affordable if you need higher capacities.

Lots of storage? Well, there goes the low power consumption goal, especially if you run 3.5" hard drives.

Lots of performance? Lots of power consumed!

There’s just something that annoys me whenever I do things on my home server and I have to wait longer than I should, and yet I’m bothered when my monitoring tells me that my home server is using 50+ watts.¹

I keep an eye out for developments in the self-hosting and home server spaces with the hopes that I’ll one day stumble upon the holy grail, that one server that fits all my needs. I’ve gotten close, but no matter what setup I have, there’s always something that keeps bothering me.

I’ve seen a few attempts at the perfect home server, covered by various tech reviewers, but they always have at least one critical flaw. Sometimes the whole package is actually great, the functionality rocks, and then you find that the hardware contains prototype-level solutions that result in the power consumption ballooning to over 30 W. Or the price is over 1000 USD/EUR, not including the drives. Or it’s only available in certain markets and the shipping and import duties destroy its value proposition.

There is no affordable platform out there that provides great performance, flexibility and storage space, all while being quiet and using very little power.²

Desktop PC-s repurposed as home servers can provide room for a lot of storage, and they are by design very flexible, but the trade-off is the higher power consumption of the setup.

Single board computers use very little power, but they can’t provide a lot of performance and connecting storage to them gets tricky and is overall limited. They can also get surprisingly expensive.

NAS boxes provide a lot of storage space and are generally low power if you exclude the power consumption of hard drives, but the cheaper ones are not that performant, and the performant ones cost almost as much as a high-end PC.

Laptops can be used as home servers, they are quite efficient and performant, but they lack the flexibility and storage options of desktop PC-s and NAS boxes. You can slap a USB-based DAS to it to add storage, but I’ve had poor experiences with these under high load, meaning that these approaches can’t be relied on if you care about your data and server stability.

Then there’s the option of buying used versions of all of the above. Great bang for buck, but you’re likely taking a hit on the power efficiency part due to the simple fact that technology keeps evolving and getting more efficient.

I’m still hopeful that one day a device exists that ticks all the boxes while also being priced affordably, but I’m afraid that it’s just a pipe dream. There are builds out there that fill in almost every need, but the parts list is very specific and the bulk of the power consumption wins come from using SSD-s instead of hard drives, which makes it less affordable.

In the meantime I guess I’ll keep rocking my ThinkPad-as-a-server approach and praying that the USB-attached storage does not cause major issues.

In November 2024, my blog was down for over 24 hours.

Here’s what I learned from this absolute clusterfuck of an incident.

Lead-up to the incident

I was browsing through photos on my Nextcloud instance. Everything was fine, until Nextcloud started generating preview images for older photos. This process is quite resource intensive, but generally manageable. However, this time the images were high quality photos in the 10-20 MB size range.

Nextcloud crunched through those, but ended up spawning so many processes that it ended up using all the available memory on my home server.

And thus, the server was down.

This could have been solved by a forced reboot. Things were complicated by the simple fact that I was 120 kilometers away from my server, and I had no IPMI-like device set up.

So I waited.

50 minutes later, I successfully logged in to my server over SSH again! The load averages were in the three-digit realm, but the system was mostly operational.

I thought that it would be a good idea to restart the server, since who knows what might’ve gone wrong while the server was handling the out-of-memory situation.

I reboot.

The server doesn’t seem to come back up. Fuck.

The downtime

The worst part of the downtime was that I was simply unable to immediately fix it due to being 120 kilometers away from the server.

My VPN connection back home was also hosted right there on the server, using this Docker image.

I eventually got around to fixing this issue the next day when I could finally get hands-on with the server, my trusty ThinkPad T430. I open the lid and am greeted with the console login screen. This means that the machine did boot.

I log in to the server over SSH and quickly open htop. My htop configuration shows metrics like systemd state, and it was showing 20+ failed services. This is very unusual.

lsblk and mount show that the storage is there. What was the issue?

Well, apparently the Docker daemon was not starting. I was searching for the error messages and ended up on this GitHub issue. I tried the fix, which involved deleting the Docker folder with all the containers and configuration, and restarted the daemon and containers. Everything is operational once again.

I then rebooted the server.

Everything is down again, with the same issue.

And thus began a 8+ hours long troubleshooting session that ran late into the night. 04:00-ish late, on a Monday.

I tried everything that I could come up with:

• used the btrfs Docker storage driver instead of the default overlay one
  • Docker is still broken after a reboot
• replaced everything with podman
  • I could not get podman to play well with my containers and IPv6 networking
• considered switching careers
  • tractors are surprisingly expensive!

I’m unable to put into words how frustrating this troubleshooting session was. The sleep deprivation, the lack of helpful information, the failed attempts at finding solutions. I’m usually quite calm and very rarely feel anger, but during these hours I felt enraged.

The root cause

The root cause will make more sense after you understand the storage setup I had at the time.

The storage on my server consisted of four 4 TB SSD-s, two were mounted inside the laptop, and the remaining two were connected via USB-SATA adapters. The filesystem in use was btrfs, both on the OS drive and the 4x 4TB storage pool. To avoid hitting the OS boot drive with unnecessary writes, I moved the Docker data root to a separate btrfs subvolume on the main storage pool.

What was the issue?

Apparently the Docker daemon on Fedora Server is able to start up before every filesystem was mounted. In this case, Docker daemon started up before the subvolume containing all the Docker images, containers and networks was mounted.

I tested out this theory by moving the Docker storage back to /var/lib/docker, which lives on the root filesystem, and after a reboot everything remained functional.

In the past, I ran a similar setup, but with the Docker storage on the SATA SSD-s that are mounted inside the laptop over a native SATA connection. With the addition of two USB-connected SSD-s, the mounting process took longer for the whole pool, which resulted in a race condition between the Docker daemon startup and the storage being mounted.

Fixing the root cause

The fix for Docker starting up before all of your storage is mounted is actually quite elegant.

The Docker service definition is contained in /etc/systemd/system/docker.service. You can override this configuration by creating a new directory at /etc/systemd/system/docker.service.d and dropping a file with the name override.conf in there with the following contents:

[Unit]
RequiresMountsFor=/containerstorage

The rest of the service definition remains the same and your customized configuration won’t be overwritten with a Docker version update. The RequiresMountsFor setting prevents the Docker service from starting up before that particular mount exists.

You can specify multiple mount points on the same line, separated by spaces.

[Unit]
RequiresMountsFor=/containerstorage /otherstorage /some/other/mountpoint

You can also specify the mount points over multiple lines if you prefer.

[Unit]
RequiresMountsFor=/containerstorage 
RequiresMountsFor=/otherstorage 
RequiresMountsFor=/some/other/mountpoint

If you’re using systemd unit files for controlling containers, then you can use the same systemd setting to prevent your containers from starting up before the storage that the container depends on is mounted.

Avoiding the out of memory incident

Nextcloud taking down my home server for 50 minutes was not the root cause, it only highlighted an issue that had been there for days at that point. That doesn’t mean that this area can’t be improved.

After this incident, every Docker Compose file that I use includes resource limits on all containers.

When defining the limits, I started with very conservative limits based on the average resource usage as observed from docker stats output.

Over the past few months I’ve had to continuously tweak the limits, especially the memory ones, due to the containers themselves running out of memory when the limits were set too low. Apparently software is getting increasingly more resource hungry.

An example Docker Compose file with resource limits looks like this:

name: nextcloud
services:
  nextcloud:
    container_name: nextcloud
    volumes:
      - /path/to/nextcloud/stuff:/data
    deploy:
      resources:
        limits:
          cpus: "4"
          memory: 2gb
    image: docker.io/nextcloud:latest
    restart: always
  nextcloud-db:
    container_name: nextcloud-db
    volumes:
      - /path/to/database:/var/lib/postgresql/data
    deploy:
      resources:
        limits:
          cpus: "4"
          memory: 2gb
    image: docker.io/postgres:16
    restart: always

In this example, each container is able to use up to 4 CPU cores and a maximum of 2 GB of memory. And just like that, Nextcloud is unable to take down my server by eating up all the available memory.

Yes, I’m aware of the Preview Generator Nextcloud app. I have it, but over multiple years of running Nextcloud, I have not found it to be very effective against the resource-hungry preview image generation happening during user interactions.

Decoupling my VPN solution from Docker

With this incident, it was also clear that running your gateway to your home network inside a container was a really stupid idea.

I’ve mitigated this issue by taking the WireGuard configuration generated by the container and moving it to the host. I also used this as an opportunity to get a to-do list item done and used this guide to add IPv6 support inside the virtual WireGuard network. I can now access IPv6 networks everywhere I go!

I briefly considered setting WireGuard up on my openWRT-powered router, but I decided against it as I’d like to own one computer that I don’t screw up with my configuration changes.

Closing thoughts

I have not yet faced an incident this severe, even at work. The impact wasn’t that big, I guess a hundred people were not able to read my blog, but the stress levels were off the charts for me during the troubleshooting process.

I’ve long advocated for self-hosting and running basic and boring solutions, with the main benefits being ease of maintenance, troubleshooting and low cost. This incident is a good reminder that even the most basic setups can have complicated issues associated with them.

At least I got it fixed and learned about a new systemd unit setting, which is nice.

Still better than handling Kubernetes issues.

Good news, everyone! Doing IPv6 networking stuff on Docker is actually good now!

I’ve recently started reworking my home server setup to be more IPv6 compatible, and as part of that I learned that during the summer of 2024 Docker shipped an update that eliminated a lot of the configuration and tweaking previously necessary to support IPv6.

There is no need to change the daemon configuration any longer, it just works on Docker Engine v27 and later.

Examples

If your host has a working IPv6 setup and you want to listen to port 80 on both IPv4 and IPv6, then you don’t have to do anything special. However, the container will only have an IPv4 address internally. You can verify it by listing all the Docker networks via sudo docker network ls and running sudo docker network inspect network-name-here for the one associated with your container.

For services like nginx that log the source IP address, this is problematic, as every incoming IPv6 request will be logged with the Docker network gateway IP address, such as 10.88.0.1.

name: nginx
services:
  nginx:
    container_name: nginx
    ports:
      - 80:80
    image: docker.io/library/nginx
    restart: always

If you want the container to have an IPv4 and an IPv6 address within the Docker network, you can create a new network and enable IPv6 in it.

name: nginx
services:
  nginx:
    container_name: nginx
    networks:
      - nginx-network
    ports:
      - 80:80
    image: docker.io/library/nginx
    restart: always
networks:
  nginx-network:
    enable_ipv6: true

There are situations where it’s handy to have a static IP address for a container within the Docker network. If you need help coming up with an unique local IPv6 address range, you can use this tool.

name: nginx
services:
  nginx:
    container_name: nginx
    networks:
      nginx-network
        ipv4_address: 10.69.42.5
        ipv6_address: fdec:cc68:5178::abba
    ports:
      - 80:80
    image: docker.io/library/nginx
    restart: always
networks:
  nginx-network:
    enable_ipv6: true
    ipam:
      driver: default
      config:
        - subnet: "10.69.42.0/24"
        - subnet: "fdec:cc68:5178::/64"

If you choose the host network driver, your container will operate within the same networking space as your container host. If the host handles both IPv4 and IPv6 networking, then your container will happily operate with both. However, due to reduced network isolation, this has some security implications that you must take into account.

name: nginx
services:
  nginx:
    container_name: nginx
    network_mode: host
    # ports are not relevant with host network mode
    image: docker.io/library/nginx
    restart: always

If you want your container to only accept connections on select interfaces, such as a Wireguard connection, then you will need to specify the IP addresses in the ports section. Here’s one example with both IPv4 and IPv6.

name: nginx
services:
  nginx:
    container_name: nginx
    networks:
      - nginx-network
    ports:
      - 10.69.42.5:80:80
      - "[fdec:cc68:5178::beef]:80:80"
    image: docker.io/library/nginx
    restart: always
networks:
  nginx-network:
    enable_ipv6: true

What about Podman?

I’ve given up on Podman. Before doing things the IPv6 way, Podman was functional for the most part, requiring a few tweaks to get things working.

I have not managed to get Podman to play fair with IPv6. No matter what I did, I could not get it to listen to certain ports and access my services, the ports would always be filtered out.

Conclusion

I’m genuinely happy to see that the IPv6 support has gotten better with Docker, and I hope that this short introduction helps those out there looking to do things the IPv6 way with containers.

Just like most people out there, I have some files that are irreplaceable, such as cat pictures.

At one point I had a few single-board computers sitting idle, namely the Orange Pi Zero and the LattePanda V1, and a few 1TB SSD-s.

I hate idle hardware, so I did the most sensible thing and assembled a fleet of networked offsite backups for backing up the most important data.

My setup is based on various flavors of Linux, but the ideas will likely translate well onto other operating systems and solutions.

Networking

The most important part is the networking. The offsite backup endpoints connect together to my home server over a WireGuard network. The home server is, well, the server, and backup endpoints are clients.

I like this WireGuard Docker image a lot because it generates the server and client configurations automatically, but you can use plain WireGuard or a completely different networking solution to connect all the devices together. Some use Tailscale to make the setup process easier, but I like to keep things as self-hosted as possible.

I’m not a networking expert, but here’s how I’ve set up my network. For this example, the WireGuard network operates in the 10.13.69.0/24 range.¹

To only allow traffic between the devices and avoid tunneling everything through the home server, set the AllowedIPs setting to AllowedIPs = 10.13.69.0/24,10.13.69.1. We want to be able to access the backup endpoints, and nothing more.

All the devices have a static IP address in that network, such as 10.13.69.1 for the home server, 10.13.69.2 for a backup endpoint and so on.

The PersistentKeepalive = 25 option is present in the client configurations so that I don’t lose the ability to access the backup endpoints. With it, all the backup endpoints call back to the home server from time to time. The aforementioned Docker image automatically adds it to the generated configuration using the PERSISTENTKEEPALIVE_PEERS=all option. This setting is crucial. Without it, I sometimes ran into problems trying to connect from my home server to the backup endpoint, and that’s something you can’t easily alleviate without having physical access to the backup endpoints, which are offsite.

Remove the DNS configuration from generated WireGuard client configurations, as you don’t need it for this purpose.

Optionally, edit the /etc/hosts file for the home server and backup endpoints so that you can access your backup endpoints using simple hostnames, like orangepizero. Example row can look like this: 10.13.69.6 orangepizero.

if your WireGuard server operates in a network with a dynamic external IP address, as is common with many home internet connections, I recommend getting yourself a domain name that you can update whenever your IP address changes and using that in your WireGuard client configurations. Without this, an IP address change will result in your backup endpoints being inaccessible.

You’ll also likely need to set up port forwarding and/or traffic rules for your backup endpoints to be able to connect back to your WireGuard server.

Once you have the WireGuard connection set up and SSH running on the backup endpoints, you should be able to drop the backup endpoints into any network that you have permission for. Ask your friends and family, and sweeten the deal by offering free technical support or help in some other area in return. The cost of running a single-board computer 24/7 is minuscule with the typical power consumption being 1-3W, so that won’t be much of a concern.

Making backups

For making the actual backups themselves, you have all sorts of options.

I rely on rsync to copy the data over. It’s simple and it works, that’s all I expect from it. Example command: rsync -aAXvz /folder/to/back/up/ backupuser@backupendpoint:/backup/ --delete. The files will be compressed during transit with the -z option, and with --delete you’ll ensure that the target folder has all the files from the source, and nothing else.

The backup storage is specified in /etc/fstab with the nofail option present. This ensures that in case the disk dies, the backup endpoint will still boot properly, allowing me to access the machine to troubleshoot the issue and/or force a desperate reboot to try to fix things. A good alternative approach is to mount/unmount the remote disk manually as part of the backup script.

The backup storage uses the btrfs filesystem, and I use btrbk to take snapshots of the contents. If I accidentally delete all the files on the backup endpoint, then I can still recover from that situation because the data is still present in snapshots. 30 days is a good retention period: enough time to save the data in case of an accidental deletion, but short enough to avoid the backup disk getting full.

If you don’t want to use filesystem-level snapshots, then tools like restic are a good alternative. It can also operate over SSH and you can configure snapshot retention policies in your backup script. Just make sure to not lose the encryption password, and verify the backups once in a while.

Deployment

I manage my backup endpoints using some cobbled-together Ansible roles. I’ve perfected it to the point where the only manual steps are flashing the OS and setting up the storage, the rest is handled via Ansible.

I’d like to share my work here, but it will make Jeff Geerling cry. Maybe one day I’ll take the time to improve things…

Maintenance

All the backup endpoints update and reboot themselves regularly. It’s just the sensible thing to do.

Every 6-12 months I also do major OS version updates. It’s risky because of the whole offsite aspect of the solution, but so far I haven’t been burned yet.

Monitoring

Monitoring is an area where I have some room for improvement. So far, I’ve set up Prometheus node-exporter to all of the backup endpoints, and my home server keeps track of how the backup endpoints are doing.

This allows me to check once in a while if any of the backup endpoints has fallen off the network, or if the backup disk is getting full.

Issues I’ve faced

I’ve had this system running for a few years now, and it’s mostly stable! There have been some issues I’ve faced as well, though. Some are very specific to certain hardware, but I think there’s value in mentioning them.

I once blew up a backup server because of an Ansible configuration issue. That meant that I had to physically go pick up the server to re-image it.

The Orange Pi Zero was running quite hot, resulting in stability issues, so I put together a really janky cooling solution.

Hell, I did the same for the LattePanda as well.

It might look horrific but the extra cooling has fixed all the stability problems on both boards.

The lack of a real-time clock on the LattePanda has required me to make its backup script a bit special. I can’t rely on a systemd timer that automatically reboots the machine once in a while, so instead that part is present in the backup script. The issue is that the LattePanda boots up with the time being set in the past, and once it gets the actual time from the network, it will run all sorts of tasks because enough time has passed! This included the reboot timer as well.

At one point, the power supply on the LattePanda just died, and it was very visible on my graphs. That required a replacement.

Conclusion

That’s how I back up the most important data. I hope that this has given you inspiration to take your own backup approach to the next level!

For a few months, I had issues with my Wi-Fi network. The 2.4 GHz network would be fine, but the 5 GHz one would suddenly stop working and completely disappear from the available Wi-Fi networks. OpenWRT upgrades also didn’t improve the situation. This was very annoying.

After some discussions with a friend, I learned about Dynamic Frequency Selection (DFS). Apparently some channels on the 5 GHz Wi-Fi spectrum are also used by weather and military radars, and those take priority. If such interference is detected, your Wi-Fi access point should switch to a different channel.

It turns out that some implementations are buggy and mine is one of them.

One quick but permanent fix is to manually select a Wi-Fi channel to operate in.

With OpenWRT, you can get a list of all the available Wi-Fi channels using this command:

iw list | grep dBm

Example output:

* 2412 MHz [1] (20.0 dBm)
* 2417 MHz [2] (20.0 dBm)
* 2422 MHz [3] (20.0 dBm)
* 2427 MHz [4] (20.0 dBm)
* 2432 MHz [5] (20.0 dBm)
* 2437 MHz [6] (20.0 dBm)
* 2442 MHz [7] (20.0 dBm)
* 2447 MHz [8] (20.0 dBm)
* 2452 MHz [9] (20.0 dBm)
* 2457 MHz [10] (20.0 dBm)
* 2462 MHz [11] (20.0 dBm)
* 2467 MHz [12] (20.0 dBm)
* 2472 MHz [13] (20.0 dBm)
* 5180 MHz [36] (23.0 dBm)
* 5200 MHz [40] (23.0 dBm)
* 5220 MHz [44] (23.0 dBm)
* 5240 MHz [48] (23.0 dBm)
* 5260 MHz [52] (20.0 dBm) (radar detection)
* 5280 MHz [56] (20.0 dBm) (radar detection)
* 5300 MHz [60] (20.0 dBm) (radar detection)
* 5320 MHz [64] (20.0 dBm) (radar detection)
* 5500 MHz [100] (26.0 dBm) (radar detection)
* 5520 MHz [104] (26.0 dBm) (radar detection)
* 5540 MHz [108] (26.0 dBm) (radar detection)
* 5560 MHz [112] (26.0 dBm) (radar detection)
* 5580 MHz [116] (26.0 dBm) (radar detection)
* 5600 MHz [120] (26.0 dBm) (radar detection)
* 5620 MHz [124] (26.0 dBm) (radar detection)
* 5640 MHz [128] (26.0 dBm) (radar detection)
* 5660 MHz [132] (26.0 dBm) (radar detection)
* 5680 MHz [136] (26.0 dBm) (radar detection)
* 5700 MHz [140] (26.0 dBm) (radar detection)
* 5720 MHz [144] (13.0 dBm) (radar detection)
* 5745 MHz [149] (13.0 dBm)
* 5765 MHz [153] (13.0 dBm)
* 5785 MHz [157] (13.0 dBm)
* 5805 MHz [161] (13.0 dBm)
* 5825 MHz [165] (13.0 dBm)
* 5845 MHz [169] (13.0 dBm)
* 5865 MHz [173] (13.0 dBm)

Notice the ones with (radar detection) at the end? Those are the potentially problematic channels. We’re going to avoid them from now on by picking a specific channel to use.

When it comes to the choice of the channels themselves, you’ll also have to consider the channel width. If you pick a channel next to one of the radar detection ones and with a large channel width, you might still run into issues.

Choosing a specific channel also comes with bandwidth and range trade-offs. If you don’t care much for those, go for the lowest one and with 40 MHz width. Picking the optimal Wi-Fi channel and channel width configuration for your specific needs is better explained by other resources.

I recommend getting a Wi-Fi spectrum analysis app that shows you the channels that are least populated by neighboring Wi-Fi access points.

It is also possible to define the list of channels that the Wi-Fi AP can automatically choose from using the channels option for the wireless interface. We can use this setting to avoid the radar detection channels completely. This setting doesn’t seem to be configurable via the graphical interface (LuCI), but you can change it in /etc/config/wireless using the command line and vi, over SSH.

More information about this option and others can be found in OpenWRT documentation.

Based on our example, a configuration that avoids radar detection frequencies can look something like this:

config wifi-device 'radio0'
        option type 'mac80211'
        option path 'pci0000:00/0000:00:00.0'
        option channel 'auto'
        option channels '36 40 44 48 149 153 157 161 165 169 173'
        option band '5g'     
        option htmode 'VHT40'
        option country 'EE'    
        option cell_density '0'

Using a manually specified channel has resulted in no Wi-Fi related issues for over half a year.

I consider this a permanent fix.

If you’re using a PC and don’t want to mess with Wi-Fi issues ever again, then just run some Ethernet cables. It’s worth it.

This is a follow-up to my two previous attempts on this topic:

• Can a laptop from 2012 be a viable home server?
• ThinkPad as a server: the follow-up

Since then, I’ve had quite a few changes to my home server setup:

• I put my home server stuff on an IKEA pegboard
• I tried the Zimaboard
• I switched back to the ASRock DeskMini
• I got fiber again!

Zimaboard was great, until the CPU became just a little bit too slow for my needs.

ASRock Deskmini X300 was great, until I learned that it’s actually relatively bad at idle power usage due to lack of lower sleep states.

Combined with my recent tech cleanup, I am now proudly running all of my services on a single home server, powered by my trusty ThinkPad T430.

The promise of USB-C everything was too good to pass up, so I stopped resisting.

Doesn’t mean that I won’t run hardware into the ground.

Compared to last two attempts, I’ve made a couple of important changes to the setup.

There is a weekly tlp recalibrate BAT0 job scheduled that completely drains the battery and charges it up again, which should make sure that the laptop battery survives for a longer time. The battery charge threshold is set to 80%.

I’ve used a knock-off Honeywell PTM7950 thermal pad on the CPU. The cooling performance is comparable to using thermal paste, but hopefully it doesn’t drip out.

The whole setup is mounted to an IKEA pegboard using a Nillkin ProDesk laptop stand and a generous amount of zip ties. The combination of zip ties and the rubber feet on the stand result in the stand barely moving once mounted. The laptop stand itself is very sturdy, and you’ll have to use a lot of force to change its angle.

The CPU uses less power compared to the previous one. The one in the server is the Intel i7-3632QM, a quad-core CPU with a 35W TDP.

I still have some scripts running to limit the CPU temperatures to 85°C, just to be on the safe side and avoid overheating at all costs.

The power consumption is improved on this setup:

• mean: 27 W
• min: 23 W
• max: 70 W

Results on the DeskMini on a typical day:

• mean: 34.8 W
• min: 28 W
• max: 89 W

All measurements include the power consumption of the UPS, fiber PON and the router, which is around 10 W.

And the performance is good enough.

I’m happy with it.

It’s also handy that I now have that one laptop that I can sticker bomb without feeling guilty about removing them later on.

2025 June update

I have been running the ThinkPad T430 as a home server for a long time now. In fact, it’s one of the longest stints I’ve had with a specific home server setup. The battery recalibration was temporarily disabled because of it interfering with UPower configuration that I had in place for shutting down the laptop at a higher battery percentage (40%), but in all other aspects it’s doing fine, and after removing UPower, the battery recalibration works again.

There was a short week-long experiment with a ThinkCentre M900 Tiny, but that got put on pause because the USB-connected drives were a bit flaky. The ThinkPad T430 has proven itself to be a more stable server platform as a result.

I had the opportunity to test the resiliency of my home server setup due to a scheduled power outage on 2024-09-13.

It was also Friday the 13th. I’m not superstitious, but I’m a little stitious.

My setup usually consists of the home server, a Wifi AP/router combo box, a converter box for the fiber line, and a CyberPower UT850EG UPS.

The planned power outage was communicated a week in advance and was supposed to take up to two hours. It ended up taking about 1 hour 20 minutes.

The CyberPower UPS is good for temporary loss of power and can probably run my whole networking and home server setup for about 10-15 minutes, which is not enough. Luckily I have one of those Jackery power banks that can provide 230V output for a much longer time. During a trial run, I found that this Jackery box can sustain my homelab for about 4 hours without any issues.

At 11:02, the power went out and UPS did its thing and kept the setup alive. I then turned on the Jackery power bank, connected the UPS to the 230 V power line and everything remained operational. I continued working on my laptop as usual.

My blog and all the other services I run on my home server kept working.

I hope to one day do something similar but on a bigger scale, involving solar panels and big batteries. Would be wicked cool to keep a whole building running off of batteries during outages.

This blog is running on a home server (again).

I have once again gained access to a competent internet connection¹, and I think I have figured out the IPv6 setup as well², leading to this change.

The IP address is dynamic, there are occasional power outages and I might just mess up my configuration and bring it all down, but I get to brag about this setup so it all balances out.

My current ISP provides an internet connection over a copper wire. To use it, I have a crappy modem (Technicolor CGA2121, DOCSIS 3.0). It’s running in bridge mode, meaning that all it does is convert the signal running over the coax cable into plain old Ethernet.

My main networking device is a TP-Link Archer C7 v5. It runs OpenWRT. This router/Wi-Fi AP box connects to the modem and handles everything, including getting a public IPv4 address from the ISP.¹

After a power outage or my ISP doing maintenance, the public IP address has usually changed. This wouldn’t be a problem if I just stuck to the ISP-approved modem.²

With my setup, there was a problem. The OpenWRT box would try to operate with the IPv4 address that it was given since the DHCP lease had not yet expired. However, this meant that there was no internet connectivity. A reboot of the OpenWRT box would resolve the issue.

This manual workaround wasn’t good enough for me. It would be quite problematic if this issue happened while I was away from home because I’d still like to access my home server.

After traversing OpenWRT forums and consulting the Slack workspace of my local hackerspace, I found that bringing up the WAN interface again would result in the OpenWRT box getting a new public IPv4 address. Problem solved!

To automate this workaround, I created a single crontab entry in the OpenWRT box. This is also configurable in a graphical user interface as long as you have LuCI installed.

The crontab entry looks like this:

*/5 * * * * /bin/ash -c '/bin/ping -c 3 8.8.8.8 > /dev/null || /sbin/ifup wan'

Every 5 minutes, the router pings Google’s DNS server. If that command succeeds, then the internet connection works and that’s it. If the ping fails, then the other half of the shell command is executed, which brings up the wan interface on my router.

Feel free to use a different IP address to test with. Your WAN network interface might also have a different name.

The downside of this solution is that if the server you’re using to verify your internet connection is down or refuses pings, then you’ll be causing interruptions in your home network every 5 minutes.

Talking to the ISP about this issue was something I considered as well. Then I remembered that it took me 1.5 months of fighting chatbots and repeating the same information to different customer care agents to use my own modem that’s identical to the one the ISP uses. That’s a hell no from me.

This post isn’t a detailed line-by-line tutorial on how to set up each individual piece of the setup as those types of guides tend to get out of date really easily, but if you know your way around Linux and the command line, then you can definitely replicate this setup on your own.

Over the past few years I’ve been interested in learning about how much energy my computing setup and home appliances use. I’ve used a simple digital energy meter before to get instantaneous readings, but that was not ideal for monitoring how an electrical appliance consumes power over a longer time period.

My friend bought a few smart plugs from athom.tech. After getting confirmation that they don’t suck, I went ahead and ordered some myself.

The smart plugs I bought were the EU style plug V3 variant. The shipping times in EU were reasonably fast, shipping in 9 days (and that included Christmas!).

The plugs ship with Tasmota pre-flashed. The plugs come with a small paper strip explaining the steps to take to connect the plug to your Wi-Fi network, and after that you can manage the plug in your browser.

Updating the firmware to the latest version is easy and doable in the web GUI with just a few clicks.

The software stack

My initial goal with these plugs was to visually monitor the power consumption of a few devices, such as my home server, router, workstation setup and the electric water heater.

The power meter data is collected to an existing Prometheus instance on my home server. If you’re not sure what Prometheus is, then think of it as a tool that periodically reads metrics from different sources, saves them to disk and allows you to later query and manipulate that information.

I run multiple instances of this tasmota-power-exporter solution on my server, one per plug, which get scraped once per second by the Prometheus instance. It is possible to also make the plugs themselves export these metrics, but I didn’t fancy building Tasmota firmware myself, yet.¹

I already had a Grafana instance running on my home server, so I reused that to show a few basic graphs for the power meter setup.

Free tech tip: make sure to change the min step setting to 1 second to get the most detailed data points on your graphs.

Observations and findings

So, what have I learned after running this setup for almost 4 months?

Water heater

It shouldn’t come as a surprise that the electric water heater uses up the most power. The one we have is a 30L one, just enough for a quick shower or two or for washing a large load of dishes.

Typical power usage: 4.51 kWh/day.

Minimum observed: 0.56 kWh/day, happens usually when nobody is at home

Maximum observed: 11.1 kWh/day, a lot of washing and showering happened on that day

Home server setup

I run all my home server workloads off of a Zimaboard. One of its big selling points for me was its super low power consumption. When idling, the Zimaboard can use about 2 W, typical power usage with all my services running was about 7 W and the maximum power consumption was around 15 W.

The Zimaboard was actually using less power than my internet modem/router box (which is hot garbage by the way). The ISP-provided box used 12-14 W at all times, regardless if it was operating in router or bridge mode.

At one point I temporarily switched my home server setup back to the ASRock Deskmini X300, mainly because I added some latency-sensitive workloads onto my home server setup and the Zimaboard was a bit too weak for those.

One thing that became really obvious from the power meter is that the Deskmini idle power consumption is horrible in comparison to the Zimaboard, coming in at around 15-20 W. This might be in part because my Deskmini doesn’t seem to expose any lower CPU power states than C3 while the Zimaboard exposed C-states all the way down to C10. I could not find a way to expose lower C-states in UEFI settings or the Linux kernel. Based on a quick remark in this video I believe that it should be possible to drop ASRock a message and receive a custom BIOS that enables some extra features, but that seems even more unlikely for a consumer-grade device such as this one.

I’ve added a panel to my usual Prometheus node exporter Grafana view that shows the server setup power consumption alongside other metrics. Higher CPU activity is clearly visible on the power consumption graphs.

Voltage

The Tasmota plugs also report current voltage values. I graphed them just for fun, but noticed that there is a sort of seasonality to the values. During the usual peak power consumption hours the voltage drops across the board.

This seems like a candidate for testing anomaly detection, as shown in this great GitLab article. I’ve used this concept at work and it works reasonably well for some metrics.

In other cases the voltage drops were caused by running appliances that use a lot of power, such as the water heater, electric kettle, electric stove or the microwave.

During cooking with an electric stove, the on-off cycles were also noticeable on the voltage reading on all of the plugs.

Workstation

I have one plug that reports numbers for everything that’s connected to my home office setup: monitor, USB-C dock, monitor light bar and anything that might be charging on the table.

The power consumption of this setup varies a lot. Sometimes I do a longer home office stint. Sometimes I recharge various devices.

The typical power consumption of the whole setup while doing something on my computer is around 45-60 W, with peaks near 90-110 W. For an ultrawide monitor, USB-C dock and my laptop I think this is a pretty good result. I’ve had desktop PC-s that have used just as much power at idle, and in the olden days there were incandescent light bulbs that used up more electricity than this!

Typical power usage: 0.95 kWh/day.

Minimum observed: 0.07 kWh/day

Maximum observed: 1.52 kWh/day

Charging

These plugs are also great for observing the charging patterns of various devices.

A laptop or a power bank charges faster at the beginning, but the speed drops off as the battery gets more full. At one point trickle charging seems to kick in until the device is fully charged.

This pattern is similar on most devices that I charge. The battery on my e-bike seems to be an exception to the rule, with a slighly increasing power consumption throughout, and a faster drop at the end of the charging cycle.

0.5kWh of power consumed for 60km of range with the “Turbo” preset on the bike doesn’t sound bad at all. Half a cent per kilometer!

Stability

The stability of the smart plugs is usually fine, but there are frequent cases where certain plugs don’t report back in time with the statistics. This may be a Wi-Fi access point issue, but annoying nevertheless.

I’ve had to completely power cycle two plugs a few times because they dropped off from the network completely and would not come back. Again, could be the fault of my Wi-Fi AP.

Future ideas

The plugs also provide a way to turn the devices on and off over different API-s (even over HTTP!), which is something I’d like to utilize either using something like Home Assistant, or a simple script that toggles certain devices on and off based on the current electricity price.

I haven’t gotten around to this part yet, but from my research it seems that the necessary integrations exist for Home Assistant.

Closing thoughts

Overall, I’m very happy with this setup. I can get reliable measurements for all sorts of computing setups from now on, which will make judging the power efficiency of the devices I use much easier. No more guesswork!

If you enjoyed this post, then I highly recommend giving this FOSDEM 2024 talk by Florian Quèze a listen. The talk also goes into details about measuring the power consumption of various devices in the speakers’ home and observations that they’ve made.

I have a bad habit of changing my computing setups all the time. I tend to see new gear, then I get some new ideas, and then I obsessively think about it for weeks and months until I just buy it. And then the cycle repeats.

I’ve had time to think about why that keeps happening and I think I’ve got it.

I keep changing the goals, constantly, and with that I kept optimizing my setup in a different direction every time.

This post also serves as a way to explain my thought process behind my last post about the Zimaboard, since my writing may not have been clear enough on that point, or some of you just can’t read.

The treadmill

It all started with the first step in my self-hosting adventure. I had a 1TB WD external hard drive that was a single copy of all my important files, which is less than ideal if you want to preserve your data. My goal became to have another redundant copy of the storage, so I built a server from old PC parts and used hard drives, all running ZFS, and hosted in a room where loud PC-s were more of a norm.

After that setup moved to my rented apartment, I discovered that having a buzzing tower of 8 hard drives (that was an upgrade I later made) consumes a lot of power and is also unbearable when you’re trying to sleep in the same room. My goal became to reduce the noise and the power consumption, so I upgraded to a Ryzen 2000-series based setup and at least two WD Red 4TB hard drives. It was quieter, faster and more power efficient compared to the previous setup.

Then I discovered that laptops use even less power than that. A ThinkPad X230 I had around used less than 10 watts when idling, and it just so happened that I had one to play around with. My goal became to reduce the power consumption of my home server setup even more. I ended up putting my storage on two 4TB WD Red hard drives connected over USB 3.0 and running my home services off of that ThinkPad.

Soon after that, I discovered that having a cat around USB-connected storage running btrfs is a really bad idea. There was data loss, but luckily nothing I could not replace. My goal became to have a low power setup that was more cat-proof and did not rely on USB-based storage. That’s how I ended up with a TerraMaster F2-221.

Damn, these hard drives sure are loud in that NAS! I also discovered that some external 2.5" hard drive enclosures can be opened up to reveal 2.5" 15mm SATA hard drives in up to 5TB capacities. And you can plop those onto a PCIe card, for some reason. And I really liked the Thermaltake Core V1 and SSD-s for certain types of storage. My goal became to build a really cool and performant server that housed both hard drives and SSD-s, because why not?

Oh, wait, no, that thing uses a lot of power and a ThinkPad T430 became available in my computing fleet. I probably got a good deal on some 8TB hard drives around this time as well. My goal became having a lot of storage around while also using less power, and having it all as cat-proof as possible. That’s how a ThinkPad T430 ended up being a home server.

Oh, crap, that ThinkPad kept overheating because the CPU in it was a bit too much at higher loads! My goal became to find a suitable replacement that did not use a lot of power. Luckily I heard about the concept of TinyMiniMicro and got myself a Lenovo ThinkCentre M73 Tiny.

Damn, those hard drives sure are noisy, apparently due to WD running those at 7200rpm. My goal became to silence those hard drives as much as possible. I got a lot of acoustic foam, and what ended up working was to stack the drives horizontally, and add some foam between the pieces, that got rid of the humming. I also added a fan to keep the drives from overheating.

Hold on, the Raspberry Pi 4 is allegedly usable as a home server now? And it uses even less power when idling? My goal then became to have a server setup that has a single-digit idle power consumption, meaning less than 10 watts.

Unfortunately I could not get the software part running right, the setup would freeze due to unknown reasons. At least I wrote a post about troubleshooting ZFS issues as part of this run, and Google keeps recommending it to people according to Google Search Console data.

Alright, I can’t make the setup more efficient. Hold on, I also have a desktop PC, and that also consumes power when idling. That seems wasteful. My goal became to do my computing all on one machine, filling the roles of a server, workstation and gaming PC all at the same time. And I did it, and wrote a post about it as well!

Spoiler alert: it was in fact not the end-game of my setup, because I soon discovered that having everything on the same PC requires quite a special setup, and a reboot on the host OS affects all the other use cases. If you care about uptime, then it’s probably better to have separate machines for all of that. I didn’t want to give up on the idea of virtualized setups just yet, and thus my goal became to have one PC for my server and gaming needs. Games need storage, lots of it. My server has lots of storage. Makes sense, right?

After using that setup, I discovered that VR and virtualized gaming don’t mix that well. Damnit. Oh, hey, I have all these SSD-s now from that experiment, and if I shrink my data needs a bit, I could fit it all on a mirrored setup of two 4TB SSD-s, and they don’t use much power as well. That opened up a lot of possibilities regarding my next setup because a lot of small PC-s can handle two SATA SSD-s just fine. My goal became to build a silent and efficient home server. This resulted in the ASRock Deskmini X300 setup.

I’m hoping that the current DeskMini-based server configuration is the one that will last me a while.

That aged well.

I had a ThinkPad T430 lying around, and now that my storage needs were served by two 4TB SSD-s, I could finally do that one stupid idea from 2020. I also dislike knowing that I have perfectly good hardware sitting in a box and collecting dust. My goal then became to use my ThinkPad T430 as a low-power, silent and efficient home server. It was successful. Or at least mostly successful.

But the search for that elusive combination of low power consumption, adequate performance, stability, resilience and silence was not over. My blog gained traction from time to time and it would be really embarrassing if an overheated laptop or a longer power outage resulted in an outage. You see, the ThinkPad was good as a server, but it still overheated under specific conditions and once the battery runs out, the “Power on with AC attach” functionality does not work, which is quite bad for a server. My goal became to find that elusive solution.

I tried the LattePanda V1 since I got it for a good price. Didn’t work out for my use case, sadly. The website situation got solved with moving that portion of my self-hosting setup to k-space hosting, leaving me to do further experiments without affecting my lovely website that much.

After keeping my eyes open for various options, the Zimaboard was the one that finally won. Silent. Meets my storage, networking and performance requirements. Small. Super low power usage. x86 based so I know my stuff runs on it well. It’s not perfect, but it did all of those things.

And there you have it, the reason why I have written over half of these blog posts, and why my setup just keeps changing and depleting my wallet at the same time.

Let’s be honest, I’m sure that as soon as I find something exciting in the tech world to play around with, I’m probably going to try it out. Some call it passion, I call it lack of fiscal responsibility.

Free tech tip to those starting out in self-hosting: experimentation is really fun, but it’s easy to lose focus and get carried away with changing your setup all the time. You can do a lot with much less hardware than you think, so try to approach building a minimum viable home server setup first and upgrade only if absolutely necessary. In my experience it’s just as fun and probably much cheaper as well.

I stumbled upon this Hardware Haven video about the Zimaboard recently.

I liked it a lot.

I finally bought one.

In short, Zimaboard is a small single-board computer that is relatively affordable and comes with an interesting selection of ports, which includes an exposed PCI Express port.

Before we get down to the build, here’s a list of aspects that I want to see in my dream home server:

• low power usage (2-15W typical power usage)
• 8GB of RAM or better
• enough performance to run my workloads, most of which are containerized
• 2x SATA or NVMe SSD slots, plus option for a third drive for the OS
• passively cooled and completely silent
• compact size
• gigabit Ethernet or better

You might be thinking, “Wait, that’s your dream setup? No clusters of machines, Threadrippers, 10 Gigabit networking, crazy number of disks?”. Well, yes. After years of trying all sorts of setups and learning about my home server usage patterns, this is the set of requirements that finds a balance between performance, efficiency and silence.

Basics

I recommend checking the product page to see the exact specifications. The configuration I bought was the 832 model: 8GB of RAM, 32GB of eMMC storage and a quad-core Intel Celeron N3450 CPU. It’s not the most powerful setup or even a recent one with the CPU being from 2016, but it’s just powerful enough to fit my needs.

This variant of the board costs 200 USD, but other configurations cost much less than that, I just needed the extra memory to be on the safe side. If you don’t care about the noise and size aspect of home servers, then you can get a better deal on the used market (see the TinyMiniMicro project for inspiration), but as you know by this point, I care about those aspects a lot.

The package

What sets the Zimaboard apart from other single board computers is how polished the product feels. Unlike a board like the Raspberry Pi, this one comes with a case and a cooling setup already attached to it. I suspect that a similarily configured Raspberry Pi 4/5 with all the accessories added on top would result in a price that’s quite similar to the cost of a top-of-the-line Zimaboard.

The heatsink looks great and is practical at the same time. Under the most torturous loads I could only see the CPU being around 72°C and due to it being passively cooled it made absolutely no noise. With the case being present, I do not have to worry about placing the board on my desk and scratching the table or shorting something out.

The board does not seem to have a power button, but by default it’s configured to power on as soon as you connect the power supply, which is great if you’re going to use this as a home server.

The box that the board was shipped survived and overall I’d say that the packaging is good. The board comes with some stickers and a single SATA data+power cable. The included power adapter comes with EU, US and UK plugs all included.

Shipping to Estonia was quite fast, taking just 10 days. The shipping costs were 18 USD.

Storage

The included 32GB eMMC storage is fine for hosting your operating system. The read speeds cap out at around 175 MB/s and the typical write speeds I observed were around 50-100 MB/s. The storage is identified as mmc-BJTD4R_0xc7d04e40 under /dev/disk/by-id/, and searching online suggests that it’s a Samsung chip.

Because this board offers two SATA ports, I also added a SATA Y-cable to my order (4 USD) so that I can take my existing Samsung 870 QVO 4TB SATA drives and move my home server setup to this board.

Allegedly this cable can also be used to drive two 3.5" hard drives powered by the board itself according to the shop page for the Y-cable, but I suspect that at that point you’ll be pushing the limits of the 12V/3A power adapter.

There is also a white drive activity LED placed near the SATA power connector on the board. Perhaps not for everyone, but I like the aesthetic and the sight of the server doing server things.

The performance of the SATA ports is what you would expect. When performing read operations on both SSD-s I saw the maximum total transfer rates hover around 900-950MB/s, which is pretty close to the SATA III transfer speed limit.

There is no native way to mount the two SATA drives to the Zimaboard. The creators of the board do sell a metal bracket, but it doesn’t seem to integrate that well to the board. However, there exists a 3D printable design that houses two 2.5" drives, even 15mm ones, and that’s how I ended up using a 3D printer for the first time in my life. The print was done using a Voron v2 Afterburner printer hosted at k-space and it came out pretty well.

The SSD mounting holes were a bit finicky due to slight printing errors, and during my first installation attempt I forgot to put in the plastic middle layer of the case that’s between the PCB and the stock bottom cover, but other than that the installation was a breeze.

The drives never hit above 64°C, which is absolutely okay for these SSD-s. I’m not sure how actual spinning hard drives might fare in this environment, but I don’t think that it’s going to be that serious because 2.5" hard drives should not generate as much heat in the first place.

Power consumption

What I love about this board is the power usage. At this point my home server is not even the most power hungry component of my setup, that honor now goes to the ISP-provided router/modem combo box that always draws at least 12W, even when it’s in bridge mode.

Here are my power usage measurements (measured with a simple power meter):

• idle: 2.5W
• no drives attached, CPU stress test (stress -c 4): 9.0W
• one Samsung 870 QVO 4TB SSD attached, CPU stress test: 13.2W
• 2 SSD-s attached, typical power draw in my setup (~10-40% CPU usage): ~8-9W
• 2 SSD-s attached, max load on SSD-s and CPU: ~14W

The Zimaboard, ISP modem/router box, my TP-Link router/Wi-Fi AP and my CyberPower UPS all together use at most around 34W, all combined. That’s even less than what my Dell monitor uses at reasonably low brightness levels!

These tests are not scientifically accurate, but they should give you an idea on what power consumption numbers to expect when running this setup.

Performance

I use a simple Prometheus Node Exporter + Grafana setup to view how much resources my various servers use. My home server has lately either been an ASRock Deskmini X300 based setup or an old ThinkPad T430, and something that both had in common was that the CPU usage was generally very low, mostly at or below the 10% mark. There would be bursty loads from time to time and backup processes running that bump that up, but not significantly.

Memory usage of my setup was also quite low, with all my services and containers fitting into 2GB during typical usage. With this information and some CPU performance comparisons done, I knew that the Zimaboard will likely be able to handle my home server tasks.

This board is not very powerful, but if you mainly rely on containerized workloads and can rely on Intel QuickSync to accelerate media transcodes, then you’ll be just fine.

I was actually impressed with how well the GPU-accelerated transcode worked on this machine once you set everything up properly. If you’re running Linux and Jellyfin, run vainfo to get a list of supported codecs and make sure that you have hardware decoding selected for those in Jellyfin settings. Also enable hardware encoding. I checked with intel_gpu_top to see if work was offloaded to the GPU and saw activity there, which means that hardware acceleration for Jellyfin worked out great! Check the Jellyfin hardware acceleration for more details on other requirements that have to be met for all of this to work.

Caveats

The Zimaboard does ship with a Linux-based OS called CasaOS. However, I had no intention of using it as I know what my requirements are and my infrastructure is already decently documented and set up via Ansible, which is why I can’t say how good that experience is. I also did not test Windows 10 or 11.

During testing, I noticed that the USB 3 ports can act a bit weird. The Fedora Server USB stick would not boot properly if connected to one of the USB ports, and when doing other tests with external USB storage I noticed hiccups from time to time. Not sure what might be the cause, but I’m writing it down nevertheless. It’s not a dealbreaker for me luckily since I don’t rely much on the USB ports in typical use anyway.

I did not test the PCI Express port, simply because I don’t need it, yet. It’s a great addition, though, and opens up plenty of modding capabilities in the future.

Fedora Server and btrfs

The Zimaboard gave me an opportunity to start fresh with my home server.

I made a leap: I’ve ditched ZFS and am now running Fedora Server with my storage being on a btrfs RAID1 setup, snapshotted and backed up using btrbk. It was a bit tricky to migrate and set everything up regarding backups and snapshotting, but I got it working.

ZFS is great, but it has always felt like an unwanted guest in the Linux ecosystem. The kernel developers don’t care much for maintaining compatibility with ZFS since it’s not in the kernel due to licensing issues, and Ubuntu has been one of the few distros that actually ships a kernel that includes ZFS built in. I didn’t want to be tied to Ubuntu forever, especially because of how they try to make snap a thing. ZFS DKMS builds are generally okay on distros like Debian, but on others you might find yourself not being able to access your data after a reboot because of a kernel update.

btrfs has had some issues in the past, especially with the RAID5/6 setup, but in my single and dual disk setups it has been solid for years, except for that one time around 2018-2019 when I ran btrfs RAID1 over USB storage. To be fair to btrfs, that was a pretty stupid setup.

Before committing to btrfs, I used two USB sticks to create a RAID1 setup and created real torture test scenarios. Tests looked something like this:

• write a file to the filesystem
• use md5sum to calculate a hash of it for verification purposes
• completely wipe one USB drive with dd
• run md5sum to calculate the hash again (it matched every time)
• run btrfs scrub on the filesystem to fix all errors
• rinse and repeat with variations to this setup

After doing all that, I was quite confident that this was going to work.

One thing to note with btrfs: if you’re running any type of multi-disk and redundant setup like RAID1 or RAID10, then make sure to include the mount option degraded in /etc/fstab so that you can still mount your filesystem if one or more of your drives fail. If you lose too many drives then it’s still probably going to fail to boot.

Here’s my /etc/fstab setup to serve as an example:

LABEL=turbo /turbo      btrfs subvol=turbo,compress-force=zstd:1,ssd,degraded,nofail            0 0

When doing the migration I kept all the paths the same, and each ZFS dataset was recreated using btrfs subvolumes, which is why this filesystem is mounted on a top-level folder.

Alternatives

My notes for the next dream home server setup included a few candidates:

• the Zimaboard

• Asustor Flashstor 6 or Flashstor 12

• a TinyMiniMicro style machine with at least two NVMe SSD slots

The first two fall nicely into my requirements for the dream home server. I picked the Zimaboard mainly because it’s much cheaper and I was already running two SATA SSD-s for my home server storage, so migrating would be really easy and I would not have to buy any new drives.

If large capacity SATA SSD-s start becoming less common and NVMe SSD-s become even cheaper than they already are, then I’ll have to look into something like a Flashstor.

Conclusion

Overall, I’m very happy with this purchase. The performance is just enough for my services to work reasonably fast, the board uses very little power and it’s silent, and it looks good on my wall.

My setup is also quite flexible, so in case I need more resources, I can get more Zimaboards and make them serve different purposes. However, it’s more likely that I’m going to discover a new toy to experiment and play with by the time I run out of resources on this one.

2024-08-20 update

I ran the Zimaboard for over half a year, but decided to switch back to the ASRock Deskmini X300.

I still love the low power consumption and the expandability, but I ended up putting more load on the system than I initially anticipated CPU-wise (running over 20 different Docker containers, some heavy). Should’ve guessed that it would happen eventually.

If it came with an embedded AMD Ryzen APU, then that would probably change things up quite a bit.

Zimaboard: still the closest thing to my dream home server, but it’s that last little bit of performance being missing that gets me. If your needs are moderate or low, then it’s still a fantastic option, if the price is right.

My eyes are now on the Asustor Flashstor Gen 2 machines. Up to 12 NVMe SSD-s plus 4 cores of AMD Ryzen goodness sounds too good to be true. Let’s see what the reviews will say once it’s out.

Steam recently launched a new feature: local network game transfers.

The idea is simple: if you have a game downloaded on another PC and you’re both on the same local network, then Steam can download game data from that PC, avoiding the need to download the game over public internet. Using this method you can reduce your internet usage and enjoy faster download times.

I think this feature is absolutely brilliant. There are many homes out there, even in developed countries, where the internet connection sucks. The speed might be capped at something very slow, such as 10 Mbit/s down, or you might have a bandwidth cap in place. Perhaps you have multiple gaming PC-s and you want to avoid downloading 100+ gigabytes worth of files over the internet to avoid hitting your bandwidth cap. In those situations being able to download your games to a PC that has a bunch of storage in it makes a lot of sense.

This isn’t the only way to achieve the same goal as projects like LanCache exist. However, with Steam the setup is so simple that anyone who knows how to install Steam could achieve a similar result. The only caveat is that you’re limited to games installed through Steam, LanCache supports many other services.

Here are some ideas on how you can set up your own Steam cache.

Using an old laptop

If you have an older laptop around and it has a gigabit Ethernet connection, then that can be a good candidate for setting up a Steam cache.

The hardware I picked for this test is a ThinkPad T430. It has a 4 core CPU in it, 16 GB of RAM and plenty of ways to attach storage to it. With storage I tried two approaches: using an older 1TB 2.5" SATA HDD, and using two Samsung 870 EVO 1TB SATA SSD-s in RAID0.

The hard drive based solution isn’t something I recommend, unless you plan on running two or more in a RAID-like setup. The Steam cache use case seems to have hit the limits of the hard drive and that limited the transfer speeds significantly.

The SSD-based setup was great! You might want to run something other than a RAID0 setup if you want to avoid a drive failure taking your whole Steam library with you, I ran with this setup mainly to see where the limits are. The Steam library is also something you can easily redownload, should things go really wrong.

One benefit of setting up an older laptop as a Steam cache is that you can take it with you to a place that has great internet connectivity, download your games there and bring it back home with you. That’s exactly what I did when performing testing, I went to my local hackerspace, started the downloads, and came back half a day later to find that over a terabyte worth of games had been downloaded. Doing the same on a slower home network would have taken days. I guess the idea of IP over avian carriers, or over humans, isn’t dead yet.

When downloading the games, I noticed that Steam was hitting the CPU hard. Even on a decent 4-core Intel i7-3820QM the CPU was often running at 100%. This meant that download speeds were usually around 500-700 Mbit/s, which is still good, but nowhere near saturating the gigabit link that the laptop had.

Now that I had all the games downloaded, it was time to test the performance. The maximum transfer speeds were around 400-500 Mbit/s over my local network. I used my Steam Deck over an USB-C dock to start the downloads and picked the internal NVMe SSD as the download target to avoid any storage speed bottlenecks. I would have liked to see speeds close to 1 Gbit/s, but these speeds are still a massive improvement if your network speeds are something like 50 Mbit/s down, that’s a 10x improvement!

If you’re worried about the fact that Windows 10 is going to lose support in October 2025 and your Steam cache PC is not supported by Windows 11, then don’t worry, you can do the same thing over Linux.

Using a virtual machine

If you already have a NAS or a home server running, then you can create a new VM, add a bunch of storage to it, setup the OS and Steam and you’re good to go!

I did most of the testing on a Windows 10 VM and a 1TB HDD attached over USB 3.0. I setup the VM using a bog-standard QEMU/KVM setup and set everything up through virt-manager in Linux. The only noteworthy part is that I set up the network interface as a macvtap device, resulting in the VM showing up on my local home network as if it was a separate PC. That step was taken to avoid any issues with the Steam instances on different PC-s not being able to transfer data directly between each other.

My server runs on the ASRock Deskmini X300. It has plenty of CPU performance, but during testing I ran into performance issues with the hard drive again. When I added SSD-based storage to the VM, I found that the performance was great and quite similar to what I saw on the laptop.

The downside with the Windows 10 VM setup was the CPU usage. While the VM was idling, my server saw a noticeable increase in idle CPU usage, utilizing about 10-20% of my CPU at all times. During transfers the VM was using most of my CPU cores.

I did testing on a Linux VM to see if those are more efficient compared to a Windows 10 VM. The answer is obviously “yes”, but there is one issue: I could not get Steam to work. At the time of testing, Steam had recently released a bigger visual overhaul and I suspect that it might have something to do with those issues. The VM has no GPU acceleration and opening Steam would result in “Loading user data” popup showing up, but nothing else happening. I tried Ubuntu Desktop 22.04, Fedora Linux 38, and even Flatpak installations of Steam. All of them ran into the same exact issue. There are probably ways to set Steam to not require GPU acceleration without opening its settings view graphically, but I did not bother with that.

If you like to think in virtual machines, then this is a good option. However, it’s not something I stuck with due to the inefficient resource usage that a Windows 10 VM exhibited.

Using containers

This is the solution that I am currently running.

I recently found the LinuxServer.io webtop Docker images. These containers allow you to run various Linux desktop environments and distros on the same machine. You can connect to them using the integrated KasmVNC solution over a browser. You can even use your GPU on the container host machine to improve the video rendering performance in the container!

I picked the fedora-xfce option because Fedora is great, and the XFCE desktop environment is not that resource intensive. I also utilized the linuxserver.io package install mod to install Steam on container startup. For networking, I created a separate Docker network using the macvlan driver to achieve something similar to the VM setup, resulting in the container showing up as a separate machine on the local network. Inside the container I set up Steam to start on startup using standard GUI tools that XFCE provides for this purpose.

I really like this setup. I can easily point the Steam downloads to a larger storage pool and make use of all the CPU power that Steam needs to perform the transfers.

During testing the limitation seems to have been my Steam Deck. I could still not saturate the 1 Gbit/s link.

The KasmVNC solution is quite nice and performant. The fact that I could use it in the browser makes it very convenient to use as well.

The CPU usage during idle is very small, except for when you accidentally leave the Steam window open with animated content being displayed. During file transfers the CPU gets a beating, but it’s nothing the AMD Ryzen 7 5700G can’t handle.

For storage I eventually ended up using a 1TB Lexar NVMe SSD. That’s not enough to download all the games I have in my Steam library, but at least I can download those games that take 20+GB of space and not have to worry about downloading those over a slower internet connection.

I have not ended up using this setup too much yet, but I imagine that with games growing in size this is going to become much more relevant for my Steam Deck. And if I were to move to the countryside where the only internet connection available is a mobile 4G connection capped at 10 Mbit/s down, then this setup would be fantastic to have.

Tech tips

If you intend to run a similar setup, I recommend enabling “Scheduled updates” to run during the night. I set mine up to auto-update games between 00:00 and 06:00, which is where I’m likely going to sleep and not notice that game downloads are taking place. Steam is notorious for hogging all of the bandwidth on the network and this nifty feature helps avoid disturbing other users on the network.

If you’re performing the initial mass download of your games but would like for it to take place during scheduled times, start downloading all of the games and then restart your machine. When Steam starts up again, you’ll notice that all your games are queued up nicely to download during your scheduled updates time slot.

Caveats

This setup is not ideal and the Steam support page highlights it well. For example, this setup does not work in fully offline situations as internet connectivity is required for the initial setup of the local network game transfer. I recommend giving that page a read to understand the requirements and technical specifics of this setup.

I recently learned about the IKEA SKÅDIS series, which is a pegboard that supports a variety of extras. During my self-hosting journey having to figure out the best place for putting all my compute stuff to has always been at the back of my mind, especially due to limited floor space at my home. This pegboard gave me an idea.

Note that this idea also applies to other types of pegboards that you can find, I just went with the IKEA one for aesthetics (and the wife approval factor).

The layout was improvised during the assembly process, but in the end I’m quite happy with how this turned out. Everything is held in place with zip ties of varying sizes. The biggest ones I used are 360x4.5mm, which fit nicely due to the slot width of the pegboard being 5mm. Some of the zip tie ends have been cut off for aesthetic reasons, but most of them are intact in case I want to perform maintenance on this setup.

There are some improvements that I might implement to a revision of this board. The USB cables connecting the external SSD-s seem to be just small enough to be routable behind the pegboard. Other cables that are visible can probably be hidden on the other side as well, giving the setup a much cleaner look.

I also installed a second pegboard to have access to most commonly used cables and other stuff that I usually store on my table in a chaotic fashion. It’s a work in progress, but it looks very promising already.

Maintenance is not going to be very easy on this setup. Luckily the UPS should be maintenance-free for the next 2-4 years or whenever the battery dies, and with the ASRock Deskmini X300 I can unscrew the 4 screws on the back and slide the thing out without having to take off the whole pegboard. I hope that the external SSD-s don’t die on me soon, but if they do, then sliding them out of the zip ties should not be that difficult.

Those more familiar with my content might see echoes of the seedbox on a wall in this post.

A while ago I wrote about how easy it is to download an archive of Wikipedia and host it anywhere you want using Kiwix.

Here are a few things you might want to consider before doing so yourself.

The spam

I have a specific e-mail address set up so that readers of my blog can reach out to me no matter where they see my post. I knew about the risk of spam, but receiving genuine feedback and questions is something I appreciate a lot and I’m willing to mark everything else as spam if needed.

What I did not expect was the amount of spam that originates from my self-hosted copy of Wikipedia, or that they’d use the e-mail that is present on the main ounapuu.ee domain.

The scheme seems to work like this:

• crawl the web
• find links to genuine sources that have broken (HTTP 404 errors)
• spam the owner of the site with a politely worded e-mail template that asks you to fix those broken links with these new ones that link to some marketing garbage or SEO spam
• go to step 0

And the worst part is that this probably works for a lot of smaller sites with non-technical owners.

I’ve deleted most of the spam but have some recent examples (click on the image to see the higher quality image).

Wikipedia has a lot of broken links and since my copy of it will always be a little bit out of date, I will probably keep receiving spam like this until the end of time.

Security

Turns out that the version of Kiwix that you can sudo apt install in Ubuntu Server 22.04 LTS is really out of date. I learned about that when CERT-EE reached out to me about an XSS vulnerability that Kiwix had.

If you’re running a Kiwix version older than 12.0.0, then congratulations, you have this vulnerability!

If your distro ships the latest version (Arch Linux, Fedora Linux etc.), then you’re good to go!

If your distro doesn’t, then feel free to use the Kiwix Docker images to get around that limitation.

A very basic example command that should get you started:

docker run --rm  \
    --name=kiwix \
    -v /path/to/content/:/data \
    ghcr.io/kiwix/kiwix-serve:latest \
    /data/*.zim

2024-09-05 update: LLM bots!

Well, it turns out that Kiwix is horribly inefficient at disk usage when LLM data scrapers are hitting it at 50+ requests per second.

The following is the result of running Kiwix for 4 days. Over 8 TB of disk reads is incredibly wasteful.

CONTAINER ID   NAME                          CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O         PIDS
734d8e1b4385   kiwix                         50.28%    10.36GiB / 27.27GiB   38.00%    7.73GB / 94.7GB   8.45TB / 12.9GB   7

I’ve decided to finally take down my instance of kiwix.ounapuu.ee as a result.

It all started with me getting a Steam Deck.

Background

After getting familiar with the Steam Deck and how the Proton compatibility layer works, I decided to write a backup script that would back up everything in the home folder, excluding the Steam games themselves due to the sheer girth of modern games (how the hell has GTA V ballooned up to 100+GB???).

Among the folders that I backed up was compatdata, a folder that contains files that Proton needs to make Windows games run. If you browse the folder, the contents look like a mini-Windows installation, and among those files you can also find your savegames. It made sense to me to back up this folder, and I was quite happy to know that no matter what happens, my game saves would be safe as long as I made a copy of this folder.

Fast forward a few weeks: I’m doing some changes to my self-hosting infrastructure and decide to redeploy changes to a backup server that I have. Nothing fancy, just Ansible roles that make sure that the backup server has some configuration present and that the backup folder has the right permissions.

The step looks something like this:

- name: Fix permissions
  ansible.builtin.file:
    name: /path/to/backups
    state: directory
    owner: user
    group: user
    recurse: yes

Everything was okay until I saw this.

After trying to recover the backup server and giving up as soon as I found that I cannot log in as root or use sudo, I gave up on doing that over the network and decided to get physical acccess to the server. One reinstall and Ansible run later, the server is okay again.

Investigation

How did we end up here though?

The backups from the Steam Deck are made to my home server using rsync. The Steam Deck is just a Linux machine, after all, and it made the most sense to me as I could make a backup that preserves all the permissions and links. Should I ever screw something up, I can run the same script in reverse and have everything working as it used to. I would hate setting up all my games and customizations again.

The script looked something like this.

rsync -aAXzv --delete-before /home/deck/ backupuser@myserver:/path/to/steamdeck/ \
  --exclude .local/share/Steam/steamapps/common \
  --exclude .local/share/Steam/steamapps/downloading

sudo shutdown now

Free tech tip: you can create a desktop entry for any script, add it to Steam, and run it within the Steam Deck UI when you’re done playing. You’ll have a backup and the system will shut down automatically once it’s done!

Nothing suspicious about the script, right?

There’s one problem with it: rsync also sends over links. These come in various types (symlinks, hard links) and act as a pointer to another file or folder. Turns out that Proton (or Wine) loves using symlinks. Most usages I saw were pointing towards common distributions of Proton, which makes perfect sense since it saves disk space.

There are also links that point to the root folder /. The use case for these seems also makes sense: present the game with a drive like Z:, and point it to / on Linux side, and now the user can easily navigate their whole file system within the context of the game they’re running. Maybe you want to install the game or its add-ons into a different folder, or maybe you want to load a save game that you have somewhere on your Steam Deck.

Just one problem with this: what happens when you have this symlink on another machine, such as my backup server, and you use Ansible to set the permissions for a folder containing this symlink, recursively? And what if the follow setting in Ansible is on by default since Ansible 2.5?

Turns out that the answer is that you’re going to mess up the file permissions on the whole machine, and most things stop working at that point. I could still login to one user on that box, but I could not do anything that would help recover the state of the machine at that point.

The fix (?)

There are many options for avoiding this problem, or at least working around it. I’m not sure what the perfect fix looks like. In case you know one, let me know.

One thing I added to my backup script was --no-links, which instructs rsync to not copy symlinks. That setting has to appear after the first batch of arguments (rsync -aAXzv) because otherwise it will be overridden by the -a parameter.

I also updated my Ansible setup to avoid setting the permissions for the Steam Deck backup folder as a precaution.

The major downside with my tweaks is that I don’t really have a “full” backup of my Steam Deck anymore. I do have backups of my game saves, but recovering from the backup will become a bit of a hassle due to me having to find and copy the game saves manually to the new compatdata folders since the backed up ones are missing all the symlinks to Proton/Wine-managed dependencies.

Alternatively I can consider using a backup tool like restic that should preserve symlinks. I’ll just need to also test recovering from that backup method.

Either way I can’t really blame Valve, Wine or Proton developers for my own fuck-up. They used symlinks in a way that allowed them to save disk space and give the user easier access to their files in-game. It’s just unfortunate that I learned about this setup the hard way.

Completely unrelated to my issue, but I can’t help but remember that one person whose machine got wiped by Steam.

There are two reasons why I haven’t written much lately, and one of them is that I was cooking up a talk about self-hosting, more specifically my own self-hosting adventure (the other one is the Steam Deck, more about that in the future).

The talk took place on May 25th 2023 at k-space, a hackerspace in Tallinn, Estonia.

This talk was livestreamed and you can watch the recording by clicking on this link.

Hard drives are still the default choice for many homelab and data hoarding enthusiasts. They still hold the gigabytes per dollar advantage over SSD-s (for now), and if you buy the big external drives and take the drives out of the enclosures, you can get a pretty good deal.

Spinning rust has one obvious downside: it’s slow, both in maximum transfer speeds and in latency. For most use cases this is fine, but if you run a service that depends on a database or if you have a lot of clients relying on that storage, you’ll find that hard drives just won’t cut it.

At this point you probably don’t want to splurge and build an all-SSD NAS, so you look up caching solutions. Depending on your platform, you’ll have all sorts of options.

If you run ZFS, you’ll likely learn about L2ARC and why you probably don’t want to use it (hint: check your ZFS ARC hit/miss ratio first). Or SLOG, or the special metadata device. And in most cases the top recommendation is to add as much RAM as you can to the system so that you can benefit from a bigger filesystem cache. You’ll see some improvements, but not all workloads will benefit from this and you might up being disappointed after all that work.

I cannot speak for options on other platforms, but in most cases the idea is similar: buy a separate sacrificial SSD and use it as a cache drive.

Cache is (not) king

Take a moment and think about what type of data you’re storing.

Here’s an overview based on my own setup:

• archived YouTube videos (bunch of big files)
• cat pictures (bunch of small and big files)
• services that utilize a database (bunch of smaller files)
• various installation media (bunch of big files)
• backups of physical disks (bunch of big files)
• a copy of my Steam library (bunch of big and small files)
• a web server (bunch of small files)

Most of the data I have is accessed relatively infrequently, and when it is, the performance requirements are not that high. Any hard drive based array will be able to handle streaming video or copying big bulky files over the network.

Data that is more latency sensitive, such as Nextcloud, Jellyfin (and its .sqlite DB) and PostgreSQL, take up a relatively small part of the overall storage.

In this case (and assuming that the setup allows for it physically), the solution is simple: add a smaller flash-based storage pool to the setup and use it for data that benefits from it.

I experimented with a similar setup in the past with two 8 TB hard drives holding all the big files and a pair of 250 GB SSD-s handling Nextcloud and other services. Now I had the best of both worlds.

In short, you don’t need to mess around with a fancy caching solution if your storage usage patterns are simple and predictable. If you do a little bit of investigation once, you can bypass all the logic that is usually built into various caching solutions and get a better end result as no caching solution is 100% perfect.

As with many homelab experiments around 2022/2023, it all started with Raspberry Pi-s being either out of stock or absurdly overpriced.

I once noticed a listing for a LattePanda V1 on a local auction site. The price was reasonable at 45 EUR, and after I looked at the specifications, I had to get one to play around with.

The auction site usually sells used items, but this LattePanda V1 was brand new in box, which surprised me. I got the 2GB RAM/32GB eMMC option.

I put the LattePanda through its paces, and here’s what I learned.

Please note that my use cases are more on the software side, I did not try out the hardware capabilities that SBC-s are also associated with (GPIO, camera/display ports etc).

The hardware

What makes the LattePanda V1 special is that it’s based off of an x86 CPU. Mine came with an Intel Atom x5-Z8350. 4 cores, not very fast ones, but similar or better to what you’d find in a Raspberry Pi 4. Most single-board computers usually come with an ARM-based CPU.

The board behaves like any normal PC. You power it on, you see the LattePanda logo, and the OS just starts up. Smashing the “Delete” key during the boot will direct you to the UEFI settings. Since this board is marketed towards the technical crowd, it has so many settings that you can play with. Definitely more than what I’d know to do with.

The board comes with Windows 10 preinstalled. I booted it up and gave it a go, and it was slow as molasses while it downloaded the latest updates. At one point I reinstalled Windows 10 and ran into issues. It “works”, but I had trouble getting any Intel GPU drivers installed on it, which rendered it almost useless for any media playback use case that I could have used it for.

The port selection is okay. It comes with 100 Mbit/s Ethernet, one USB 3.0 port and two USB 2.0 ports. I could get around 350 MB/s out of the USB 3 port with an external SSD, and the USB 2 ports pulled anywhere between 15-40 MB/s, depending on my luck. If you fancy a gigabit connection, then an USB-to-Ethernet adapter might do the trick.

Adding one external HDD or SSD over USB is okay and works well, but after adding a second one I noticed issues with either one of the disks not receiving enough power. If you intend to use the LattePanda V1 as the brains for a cheap NAS, then use drives that are powered externally.

The eMMC storage on the board is decent. Regarding performance it’s somewhere between a hard drive and a proper SSD, and the sequential read speeds max out around 150 MB/s. My main concern with it is the durability. The eMMC chip is probably hidden behind the metal shielding on the board and replacing it (if that is even possible) might turn out to be tricky. ls -lah /dev/disk/by-id/ lists the eMMC chip as mmc-DA4032, which a little bit of Googling refers to it being a SanDisk chip.

The microSD card port works, but either booting from the microSD card is not possible or the board was a bit temperamental when I gave it a go with Ubuntu Server 22.04.

The board runs hot in its barebones configuration, any decent load over a longer time period will result in the CPU throttling itself. I recommend rigging up a metal heat sink to the bottom of the board to avoid throttling and stability issues under very high load. A thermal pad, a piece of metal, and a bunch of zip ties will do wonders.

The Linux experience

When experimenting with Linux, I found a couple of annoyances. The LattePanda V1 seems to have a display connector on board. However, in Linux it turned out to be an annoyance as the system always presented a 1024x768 display being connected to the system. Booting into a Fedora Linux liveUSB environment will lead you to staring a blank view with only the wallpaper present. Playing with settings in UEFI did not lead to any improvements. What works is adding a kernel parameter: video=DSI-1:d. With Fedora Linux, if you apply this fix in GRUB when first booting into the liveUSB environment, it will persist after the installation is done.

Once I had Fedora Linux booted up, I did not have a good time. The display flickered when I dragged the mouse near the edges and the performance was awful. It didn’t take long for the desktop session to crash.

One of the main reasons I wanted to experiment with this setup was the low power usage of this board. The LattePanda V1 idles at around 3 watts, and under a high CPU load generated with stress -c 4 I could get it to 5-6 watts. This made it a great candidate for running this website and Wireguard VPN 24/7 at home. I ended up installing Ubuntu Server 22.04 on it and actually ran this website off of it for about a week or so.

The reason I stopped using the LattePanda V1 for that purpose was the performance. The 100 Mbit/s Ethernet wasn’t a disaster, but it did set limits to how much traffic I could push through the board. What made me reconsider was the CPU performance. Simply put: SSL encryption speeds were about 5-10x slower than on my ThinkPad T430 that ran as a server before.

My performance testing wasn’t scientific or anything, but I did run simple tests. One of the involved running curl against resources, such as an image on my blog, in an infinite loop and over multiple threads. The ThinkPad T430 peaked at around 650 Mbit/s of network bandwidth while the LattePanda V1 struggled at around 60 Mbit/s.

After looking for some other benchmarking options that I could set up really quickly, I also gave ApacheBenchmark a go. This more-or-less confirmed my findings and the requests per second results are the following:

• ThinkPad T430: 301.72
• LattePanda V1: 55.86

The LattePanda V1 was over 5 times slower in that test. With a slow internet uplink this might not matter much, though.

I ended up putting the T430 back in service after doing more calculations. One idea I had with the LattePanda V1 was to have that run 24/7 due to its low idle power usage and have another beefy server run during certain hours. I liked that idea until I calculated the difference in power savings. Having the main server with around 12 W of idle power turn off for 6 hours and the LattePanda running at the same time would have actually increased the total power consumption of the setup compared to having the main server running 24/7.

The math will make sense if your main server uses much more power. Most desktop PC-s that I have seen idle around 35-50 W and rack-mounted servers use even more than that, so in those situations this might make more sense. Or you can whip up a solution that starts and stops machines in your homelab on-demand.

Oh, and don’t expect to run CPU or GPU based transcoding off of this thing with Jellyfin. It works okay for certain video formats and resolutions, but a 1080p H.265 video transcoded to H.264 is just not fast enough on this machine.

A suitable use case

I might change my mind at one point and give the LattePanda a go again as a low power web server. I hate unused hardware, which is why this LattePanda V1 is now serving as an offsite backup. The USB 3 port makes sense for attaching a bigger storage device and if you run ZFS, you can also send your whole filesystem to it with zfs send/receive or syncoid. I haven’t had much luck with ARM and ZFS, but with the LattePanda and its x86-based CPU I have not had issues.

Conclusion

All-in-all, it’s a neat little board that shows signs of instability from time to time. If you intend to just run Ubuntu Server on it and your workloads are not very performance critical, then it might work out well for you. Think of it like a juiced-up Raspberry Pi 4 to get an idea of what it might be suitable for regarding performance.

It did not fit the use case I intended to use it for, but I’m still happy that I could play around with this board. It was fun!

It’s been a while since I last wrote about my ThinkPad T430 that was tasked with being a home server. After doing some experiments with my setup a couple of times, it is once again the main server for all my self-hosting needs. However, this time I have made some adjustments.

Background

The monstrosity that you’re looking at is what I’d call “the minimum viable ThinkPad as a server”. To be fair, you could go even more minimal by ripping out everything but the motherboard and the drives, but for my use case this is how far I was willing to go.

In my previous post about this setup, I mentioned that the T430 suffered unexpected shutdowns under certain situations, which usually involved running the fan at the lowest speed, moderate to high CPU load, and a lot of storage activity. It wasn’t the CPU as the temperatures were way below the critical point, so I suspected either the motherboard or the storage running hot and tripping some sensor that I wasn’t aware of.

For a while now, I have been thinking about running the T430 as bare as possible. One idea involved taking the bare motherboard and the storage and slapping it on an acrylic sheet with some stand-offs, which should give it the best chance at cooling. Unfortunately I didn’t have enough time to commit to this, and that setup would have had some downsides as well. At some point an idea popped to my mind: I could probably achieve a similar result by stripping everything but the bare necessities from the laptop and improve the cooling as a result. That’s how I ended up with what you just saw.

T430, stripped

I ended up removing the following from the T430:

• palm rest
• smart card reader
• Bluetooth chip
• WiFi chip
• the whole display assembly

This was in part motivated by the fact that those parts are the ones that are not in the best shape on my main ThinkPad T430 that I use as an actual laptop. The other part was this nagging feeling that the most powerful PC that I own (the ASRock DeskMini X300) was woefully underutilized as my home server.

The keyboard had to stay because the power button is part of the keyboard assembly. There are ways to power on the laptop without the keyboard:

• put the laptop in a dock and use the power button that is present on the dock
• short a specific pin on the keyboard connector to ground

I do have some extra ThinkPad docks around that I could have used for this purpose, but it would have made the setup bulkier, which is why I did not go for that, at least not yet.

There is also an option in UEFI settings that can be used to turn the laptop on by connecting the charger (“Power on with AC attach”), but that only seems to work if you have a battery connected and after you have turned the laptop on manually once. The same caveat applies for Wake on LAN functionality. If the laptop overheats or it runs out of battery completely, then you need to manually power it on.

The cooling situation

Removing the palm rest reveals something that might be related to the unexpected shutdowns that I observed with this setup previously.

Under this area is a chip on the motherboard that seems to generate a modest amount of heat. While the machine is running, it gets warm to touch and gets hot if you throw a high load at the system, especially if you include storage in the mix. With the laptop lid closed and the palm rest being present, the cooling situation for this area can be quite bad indeed. Now that the palm rest is removed, this area of the laptop is now exposed, which should help with the cooling for this chip.

Having the keyboard connected in this state isn’t too bad. It does restrict the airflow a bit, yes, but at least it can be fastened using two screws, which means that it won’t move around and cause any other issues.

With this setup, the cooling has improved noticeably. Unfortunately it didn’t come to my mind to make a before/after comparison to better illustrate this, so I will just write about how well it operates in this configuration.

Currently, I have the laptop fan controlled by a modified version of the minimum viable fan control script. The fan only kicks in after the CPU reaches 60°C, ramps up to a moderate speed at over 85°C, and lets the default fan behaviour handle everything if the temperatures reaches over 95°C. This results in the fan being off completely for most of the day as the server is idling at around 5-10% CPU usage and temperatures being anywhere between 45-60°C. When there is a workload on the system that taxes the CPU, the fan will kick on with a small delay.

I’ve also disabled turbo boost on the Intel i7-3632QM that is currently in this T430, which limits the clock speed to 2.2 GHz max. Performance takes a small hit, but the CPU is now running much more efficiently. This is one of the tweaks that allows the laptop to be near silent. At a full 8 thread CPU load using stress -c 8, the CPU core temperatures peak at 70°C at 22°C ambient temperature, with the fan running at around 2600 RPM.

As a result of all of this, the T430 is very quiet while using about 10-12W of power while idling. Under the maximum CPU load generated by stress, the total system power usage is around 34W.

Putting the “paste” in thermal paste

Originally, the T430-as-a-server ran vertically using a basic laptop vertical stand. Back when I concluded the initial test in 2022, I disassembled the machine and noticed something very odd with the thermal paste.

Based on this image and the fact that I ran the machine quite warm most of the time (CPU temperatures around 60-80°C), I assume that the excess thermal paste started slowly dripping down. The part that concerns me the most is that the top of the chip was barely covered by the thermal paste. Either that part of the chip had great contact with the heatsink, or the thermal paste drooped down with time and left it mostly uncovered. It could explain some of the issues, but I’ll avoid making any definitive conclusions from this alone.

To avoid similar issues in the future, the laptop is now on a stand at about 60° angle. Not sure if it will work, but it’s worth a shot.

Closing thoughts

Overall, I’m happy with how this experiment turned out. Cooling is no longer an issue, and the T430 that I initially got for spare parts is actually being useful. The amount of performance this machine offers while using very little power seems to be a good balance for the workloads that I run on my home server.

You have probably heard about the Raspberry Pi. It’s a nice little affordable single-board computer with a huge community using it for all sorts of projects.

I got my first Raspberry Pi, the Model B+, during my first year at university, which was around the winter of 2014/2015. The idea of a super tiny PC that could actually do useful things was just very fascinating to me, and I loved the way that green PCB looked.

Over the years, I’ve used that Pi and newer revisions of them in all sorts of use cases: a simple Kodi media box, a Minecraft server, a web server, retro game emulation box, and much more.

Because the goal of the Raspberry Pi was to be an affordable platform for experimentation and learning for everyone, they had to keep the costs down. This meant that the computing power that the Raspberry Pi 1 packed wasn’t much. With the Model B+, you get one ARM CPU core that you can overclock to 1 GHz, and 512MB of RAM. It wasn’t much even at the time of release, and it definitely isn’t great in 2022.

Due to the very limited performance, the use cases for a Raspberry Pi 1 are quite limited in 2022. Use cases that don’t require much computational power are often better solved by other platforms. Use cases that are actually useful to me and solve a problem I have are too much for the Pi 1. At the same time, I absolutely hate it when I have computing equipment sitting around on a shelf doing nothing. I can’t even sell this Pi, because it is not worth much any more and it has some sentimental value for me.

Taking all that into account, and looking at my parts box, I decided to give one idea a go: let’s build the slowest damn Syncthing backup endpoint imaginable.

The build

Here’s a quick rundown of the parts:

• SBC: Raspberry Pi 1 Model B+
• Storage (microSD): SanDisk 8GB that I had lying around
• Storage (USB): Crucial BX500 1TB SATA 2.5" SSD, in an IcyBox USB-SATA enclosure
• Power: official Raspberry Pi microUSB power adapter
• Networking: TP-Link TL-WN722N USB WiFI adapter
• OS: Raspberry Pi OS 32-bit (lite)
• Acoustic foam: yes.

Use case: networked offsite backup of files that I cannot afford to lose, powered by Syncthing, btrfs as the filesystem, and btrbk as the snapshotting solution.

I set the OS up using the Raspberry Pi imager due to the useful options it offered, such as automatically setting my SSH public key authentication up. Once booted, I logged in and ran sudo apt update -y. I knew what to expect regarding performance, and was still surprised at how slow things have become. Updating the system and deploying the configuration using Ansible took hours.

The 1TB SSD is formatted as a btrfs file system and mounted to /storage. For the Syncthing service, I created a separate user syncthing with the home folder on /storage/home/syncthing. The synced data will live on /storage/syncthing. Keeping the home folder on the SSD is intentional: Syncthing keeps track of application state in a database and storing it on the microSD card will wear it out much faster. The database can also be quite big if you have a lot of files to sync.

I’m not going to go into the btrbk setup in much detail, but if you’re interested in that, then I do have a write-up about btrbk. Long story short: snapshots are cool, and they help prevent data loss in case of accidental deletions.

I set up Syncthing to allow access to the GUI from over the network by changing the ~/.config/syncthing/config.xml file while the service was stopped. Just change the GUI listen address to 0.0.0.0:8384, that will make the GUI accept connections not only from localhost, but from all machines on the same network. The next step was to set a password to the GUI and check the “Use HTTPS for GUI” box, because you probably don’t want any rando messing your machine up.

The choice to go with the WiFi for networking is related to the environment this system will eventually be deployed to. Ethernet isn’t an option there.

Once the system was all set up and files were syncing extremely slowly, I kept an eye on the Pi, especially the red power LED. I had set the CPU to run at 1 GHz using raspi-config, and with both the WiFi adapter and SSD connected, I had concerns that the Pi could not supply enough power. The power limitations of the Pi are a common source for issues and that has come up in the past as well, especially with external storage connected to the Pi. However, with this particular setup this has not been an issue so far. The fact that Pi is so underpowered that it cannot even make full use of the SSD is probably a contributing factor to the overall stability.

The file sync was very slow, but at least it was working. The typical transfer speeds hovered around 1.5 MB/s. Not great, but given the fact that this setup will operate in an environment where the download speeds are typically capped around 15-20 Mbit/s, this will be more than enough.

How well will this setup work over a longer time period? Only time will tell. Just like with other disastrous tests that I have done, I’ll try to update this post whenever anything noteworthy happens.

Does this setup make any sense if you need a performant Syncthing endpoint? Hell no. However, I’m very happy that this little Pi finally has a purpose.

Alternatives

This project was about finding a purpose for an SBC I already had lying around.

If the price and availability of Raspberry Pi-s wasn’t a complete shitshow in 2022, I would go for a Raspberry Pi 4 in a heartbeat. I can’t, and that makes me and most of the Raspberry Pi community very sad. The more performant Pi-s open up so many possibilities.

An Orange Pi Zero running Armbian was something I also considered, since I had one lying around, but it was a bit too competent for the job, and I might find a better use for it eventually.

If you wanted performance, then any x86-based PC will do just fine. My recommendation is to go for an used PC, be it a laptop or a very small and efficient PC. Just keep in mind that the power usage is typically higher on these setups compared to a small ARM-based single-board computer. If a machine runs 24/7, then the difference between 2-3 W and 10-20 W will start to matter, especially when energy prices are quite high in your region.

After some spicy feedback to my post where a ThinkPad T430 acted as a server, I decided to try out one UPS that was recommended on the basis of it being much more power efficient compared to the APC UPS that I previously ran.

That UPS? CyberPower UT650EG.

Why switch out a perfectly good UPS?

The APC Smart-UPS 750 is good, but uses a lot of power compared to the whole setup, making up roughly half of the total power consumption at idle: ~17-18W. It’s also very big compared to what it is actually powering.

One of the main selling points of this particular CyberPower UPS is that it uses very little power itself. Depending on the model, the UPS can use around 1-2W of power, at least according to the tests that they publish on their website.

Once I set up everything with the new UPS, I gave it a go. After the battery was fully charged, I did observe a dramatic drop in power usage. The whole setup now ran around 19-21W when idle, which confirms the power consumption numbers claimed by CyberPower.

Overview

The UPS itself is great for someone who just wants to connect a couple of devices into a standalone UPS. The model I got came with three EU-style sockets. Compared to the APC UPS, this is a much cleaner setup as I do not have to connect an extension cord using an adapter.

The APC Smart-UPS 750 can provide anywhere between 100 and 120 minutes of runtime with my setup. I did a quick runtime test with the CyberPower UPS as well and found that with around 25W of load it ran for 37 minutes. Not the best runtime, but given its size it’s a respectable result.

Uninterruptible power supplies like to be noisy when running off of batteries. This CyberPower unit is no exception. However, by letting it run on batteries and double-clicking the power button I managed to silence the alarm and in subsequent power cut scenarios it has not started beeping. I was unable to find a simple manual explaining how to operate the UPS and what other button combos might be out there. The piece of paper that came with the UPS was also lackluster. I’m not sure if this was intended functionality or not, but I do like that turning the alarm off is an option in the first place.

The status LED-s on the unit are okay. There aren’t many of them and the legend explaining the behaviour of those lights is just enough to understand what’s going on.

The not-so-good parts

This particular unit has an issue. I did a poor job reading the spec sheet at my local retailer and missed the fact that this specific model does not come with USB connectivity. If the UPS runs out of power, it cannot warn my server to shut down in advance, resulting in a dirty shutdown scenario. Given how stable the power at my apartment generally is, this won’t likely be a major issue, but it’s definitely something I miss. The APC UPS supports this, plus you can log everything about how the batteries are doing and even detect input voltage anomalies that you probably weren’t even aware of.

What I’m concerned about is the battery. The specifications say that the battery is not user replaceable. I hope that this is simply something the legal team thought of to avoid any liability when dealing with users who messed this process up. If not, then it will be a major problem down the line. Batteries are consumable items, and according to various threads on the web about UPS batteries, the consensus is that you should replace them every 3-4 years to avoid problems with the batteries swelling up and potentially causing fires.

If the battery dies and I cannot replace it, then what am I supposed to do, throw the otherwise functional UPS straight to the landfill? Not acceptable.

Speaking of concerns, I happened to see some troubling reports about certain CyberPower UPS-es catching on fire. The culprit seems to be a certain type of glue used in the manufacturing of the UPS that ends up conducting electricity after a longer time period. I unplugged my UPS, powered it off and gave it a look, and it seems like I should be fine. At least I hope so.

Based on this view inside the UPS, I feel like changing the battery should at least be possible, assuming that replacement batteries are readily available. Just make sure to not touch the rest of the components as they can probably kill you, and that’s just not a fun thing to do.

Conclusion

If you want a dumb and efficient UPS that protects you from basic power cutouts and input voltage changes, then do consider this UPS. However, if you care about USB connectivity and ease of maintenance, then I recommend looking for other options.

I like having a safety net whenever I’m doing something potentially destructive, which is why I use the btrfs file system for my operating system and my data. Snapshots are one half of my “whoops, there goes all my work” strategy (backups are the other half).

I’ve written about how I use snapshots on btrfs using snapper, but lately I’ve become annoyed with it.

Shortcomings of snapper

snapper is great while you’re on the happy path, but when you wander off of it, it gets a bit frustrating. This is 100% my own personal experience and I cannot rule out any PEBCAK scenarios, but it’s how I felt using the tool.

The snapshots are on the same subvolume, such as /home/.snapshots, which means that I have to specifically exclude the .snapshots folder in every tool and script that I use for making backups. Without that change, the tools ended up scanning the folder and eating up a lot of resources, mainly the CPU and storage.

Backing up snapshots to something like an external backup SSD wasn’t something that snapper supported out of the box as well, meaning that for offline backups I still had to resort to a good old rsync copy. It’s inconvenient and I won’t have a backup that also contained recent snapshots, but it was manageable.

I also reinstalled Fedora Linux on my ThinkPad recently to see how well my Ansible playbooks hold up (quite decently!) and found that setting up snapper is a bit of a pain. I tried to set up snapper again, but it wasn’t as straightforward as I was hoping for, with it not properly picking up the configuration file changes. At the end, it just ran into weird errors when I tried it to prune some snapshots according to the policy I set up. Since I was running out of disk space, I just gave up, uninstalled snapper and destroyed the snapshots manually.

After all that (and a recommendation by a friend) I decided to bite the bullet and try out btrbk.

btrbk

Setting up btrbk on Fedora Linux 36 was a breeze. Install it via dnf install btrbk, copy your configuration to /etc/btrbk/, make sure that the directory used for snapshots exists with mkdir /btrbk_snapshots and you’re good to go! btrbk is just a huge Perl script, so no fancy dependencies are needed.

My btrfs filesystem has three subvolumes: /, /home and /storage. Data stored on these subvolumes has different levels of importance, which is also reflected in the snapshot retention policies.

The configuration on my setup is the following:

timestamp_format        long

volume /
  snapshot_dir /btrbk_snapshots
  subvolume /
    snapshot_preserve_min   12h
    snapshot_preserve       24h
  subvolume /home
    snapshot_preserve_min   48h
    snapshot_preserve       7d
  subvolume /storage
    snapshot_preserve_min   6h
    snapshot_preserve       7d

To take the snapshots, I’ve set up a systemd timer that runs every 5 minutes and runs the following command:

btrbk -c /etc/btrbk/btrbk.conf run --progress

A successful run will have output like this:

--------------------------------------------------------------------------------
Backup Summary (btrbk command line client, version 0.32.1)
    Date:   Sat Jul  9 14:00:00 2022
    Config: /etc/btrbk/btrbk.conf
Legend:
    ===  up-to-date subvolume (source snapshot)
    +++  created subvolume (source snapshot)
    ---  deleted subvolume
    ***  received subvolume (non-incremental)
    >>>  received subvolume (incremental)
--------------------------------------------------------------------------------
/
+++ /btrbk_snapshots/ROOT.20220709T1400
--- /btrbk_snapshots/ROOT.20220708T1300
/home
+++ /btrbk_snapshots/home.20220709T1400
/storage
+++ /btrbk_snapshots/storage.20220709T1400
--- /btrbk_snapshots/storage.20220709T0745
--- /btrbk_snapshots/storage.20220709T0750
--- /btrbk_snapshots/storage.20220709T0755

I love how the output is very concise about what it just did.

To make a backup of my drive to an external SSD that’s also a btrfs filesystem, I created a separate config at /etc/btrbk/backup-to-ext-drive.conf:

timestamp_format        long

target_preserve_min    no
target_preserve        7d

volume /
  snapshot_dir /btrbk_snapshots
  target /mnt/backup/T430
  subvolume /
  subvolume /home
  subvolume /storage

In this example, /mnt/backup/T430 is a btrfs subvolume itself.

Instead of running rsync, my backup script contains the line:

btrbk -c /etc/btrbk/backup-to-ext-drive.conf run --progress

I love how the default behaviour in btrbk is to keep snapshots separate from the subvolumes themselves. Making a full copy of /home or any other filesystem doesn’t require me to always exclude the .snapshots directory, which is nice.

The snapshot names themselves are also human readable. If I accidentally messed up and my changes from an hour ago were lost, I’d just have to go to /btrbk_snapshots, look up a snapshot from an hour ago, and copy whatever I need from the snapshot. This has already saved my butt at least once.

Conclusion

It’s not often that a piece of software gets me that excited, but btrbk has somehow managed to do it. It’s a single script that has all the example use cases documented in its README and the functionality does exactly what I want.

That’s all I really need from software.

Huge thanks to Digital Integrity GmbH for building, maintaining and sharing this tool with the world!

I’m a man of many ideas. It’s a shame that most of them are quite stupid.

This is one of them, but at least it makes for a fun experiment.

Intro

After adding an UPS to accompany my server, my setup was as complete as it could reasonably be. The server is low power, but packs quite a punch with fast storage and plenty of compute power around. The UPS provides a battery backup for both my server and the router.

There was still one annoyance lingering at the back of my head: this setup took 40 watts of power in idle. That’s a lot for something that runs 24/7. Yes, /r/homelab regularly showcases more power hungry setups, but my standards are quite high when it comes to power use. I want to go as low as reasonably possible.

I recently bought a spare ThinkPad T430 so that I can have a backup machine in case something happens to my laptop that doesn’t suck. I’ve also had ideas about a server setup involving this laptop.

That’s when I decided to try running my services off of the ThinkPad T430.

The setup

Here’s a quick rundown of the relevant specifications:

• CPU: Intel i5-3320M, 2 cores 4 threads @ 3.3GHz max
• RAM: 16GB DDR3
• OS disk: 256GB Micron mSATA SSD
• Main storage: 2x 4TB Samsung 870 QVO SSD

One of the main reasons that this setup is viable in the first place is the fact that the ThinkPad T430 supports up to 3 SSD-s if you replace the optical drive with a HDD caddy. Your OS can run off of an mSATA SSD and your mirrored storage can utilize the two 2.5" SATA slots.

You probably don’t want to use a setup like this if you want to store a lot of big files, such as your legally obtained media collection. If your data can fit on a pair of SATA SSD-s, then this setup will be just fine.

Why use a laptop as a home server?

If you don’t have a home server yet and want to start off as cheap as possible while also being mindful of the power usage, then an used laptop with a competent CPU is a fantastic starting point. In an age where 6 CPU cores is considered mid-range, two CPU cores might not sound like much, but for most home server workloads it’s absolutely fine.

Laptop CPU-s are designed to use as little power as possible. After all, they need to run off of a small battery most of the time and have to run cool as well to make do with the limited cooling setup that they have. This makes them ideal for a machine that runs 24/7.

I measured the power usage of my ThinkPad T430 in this configuration and found that it used around 15-20 watts when running with a low load. This is comparable to the ASRock DeskMini X300.

What sets laptops apart from the usual desktop-based home server builds is that they often have a working battery, which can act as a poor man’s UPS. If you momentarily lose power, then the server will happily keep on working until the battery runs out, which might take 1-2 hours (or more if your battery is in a good state). This is comparable to the runtime I saw with the DeskMini server build and the APC Smart-UPS 750.

Most laptops also come with an Ethernet port, and if the laptop is made some time after 2005, then it’s likely running at gigabit speeds. Unless you have a valid use case for 10 gigabits or more, then this will be plenty of network bandwidth for most home server use cases.

One aspect that’s often overlooked with a laptop-based home server setup is the inclusion of an integrated keyboard and mouse. If you accidentally mess up your configuration and your server does not start up properly, then you can just open the lid and start fixing it on the spot. On any other setup, you’d have to find a monitor to hook the server up to and find a spare keyboard to use as well, which can be a hassle.

Performance

A home server that serves the needs of a couple of users at most and runs some scheduled tasks from time to time can run perfectly fine off of a dual core Intel Core series CPU from 2012. You probably don’t want to run intensive computational workloads off of it, such as transcoding video files on a service like Jellyfin. However, you can utilize the integrated GPU to off-load specific workloads to it, and media encode-decode for select video codecs is one of those. Intel QuickSync is an absolute godsend for situations where you need to transcode media on a CPU that isn’t that powerful.

My home server runs a lot of services: Docker containers, this blog, data archival jobs, backup jobs etc. When running them on the T430, the average CPU usage was around 20%. Not ideal, but still leaves plenty of headroom for services.

The overall usability and performance that this machine offers was completely acceptable. It’s not as lightning fast as something that runs a modern CPU, like the AMD Ryzen 5 PRO 4650G in the DeskMini, but it gets the job done.

Caveats: thermals

A home server built around a used laptop brings a lot of benefits. However, it also has some potential downsides that you should be aware of.

Most laptops have fans, and they can be damn loud. The ThinkPad T430 is notoriously loud if you have the cooling fan manufactured by Toshiba. In a home setting this can potentially a deal-breaker, unless you can wedge the laptop into a corner of the room where it’s not audible.

There exist solutions to the fan control problem. You can install a tool like thinkfan or just write your preferred fan level to /proc/acpi/ibm/fan to avoid the cooling fan running too loud (instructions here).

Running the fan at the lowest speed works fine most of the time since your CPU can throttle itself when it’s running too hot. However, with my setup I had an odd issue where the machine would power off suddenly when it was doing a lot of I/O workload, such as scrubbing the ZFS pool on the two 4TB SSD-s. This happened even when the CPU was not even close to throttling, running around 75 C. I do not know what the actual reason for the sudden shutdown is, but I suspect that it may be related to the motherboard overheating or the system treating the hot SSD-s as hard drives that are over the critical temperature limit of around 60 C.

I was able to work around most of the cooling-related issues by mounting the laptop vertically and limiting the maximum CPU clock speed to around 2.4 GHz using the knobs made available with intel_pstate.

The CPU temperatures during normal operation were mostly around 60C, with a high load reaching max 75C with the tweaks applied and the CPU fan running at the slowest possible speed.

Caveats: maintenance

If you have a machine that runs 24/7, maintenance will become an issue. No matter how often you clean your home, dust will accumulate. On desktop PC-s, cleaning out your CPU heatsink is just a matter of taking off the PC case side panel, removing the heatsink fan, cleaning off the dust and putting it all back together. You probably want to replace the thermal paste from time to time as well.

On most laptops, cleaning out the CPU heatsink is much trickier. You will likely have to disassemble most of the machine to even access it, which takes up much more time compared to a desktop PC, which also means more downtime.

Depending on the model, a simple change, such as upgrading your storage or replacing a faulty drive, can also be more complicated. I’ve seen laptops where you actually have to disassemble the whole machine to access the hard drive. The ThinkPad T430 has an easily accessible setup when it comes to storage, but the main storage bay does have an annoying pull-tab setup that probably won’t survive multiple subsequent changes.

Conclusion

Aside from the setbacks that may or may not be due to the specifics of my setup, I find that a laptop can make for a good home server. If you have more specific requirements, then a custom-built server is likely the way to go, but if you’re just starting out, then definitely do consider this option.

After running this setup for a week or two, I decided to go back to my previous setup based on the ASRock DeskMini. The reliability woes weren’t something I was too happy with, and I didn’t really have another use case for the DeskMini either, so it made sense for me.

I changed my setup again.

I’m now the proud owner of an APC Smart-UPS 750.

Why? Because I got it as a gift, and I have never actually had an UPS before, so it makes for a good introduction to this world.

Overview

My setup is quite basic: everything that I have on my table (server, monitor, laptop) is connected through the UPS. Since my setup uses very little power, I’m running way below the output limit of 500 watts.

When I’m not using my laptop, the UPS reports itself at 10% utilization. My server and the UPS itself seems to consume around 35-40 watts, as measured by an energy meter. When the laptop and monitor are also powered on, the load is around 15-25%, depending on the workload.

To make sure that I get the full benefit of the UPS, I connected it to my server with an USB cable and installed apcupsd. By default, the daemon could not detect the UPS, which was then fixed with the help of this StackExchange answer:

I had the same problem with COMMLOST with a USB connection, after a lot of research I found by pure hazard that with a USB connection you should change DEVICE /dev/ttys0 to DEVICE in /etc/apcupsd/apcupsd.conf with nothing after it, this way apcupsd search everywhere on the system to find the UPS and connect correctly, no more COMMLOST.

After that, restart apcupsd with:

$ sudo /etc/init.d/apcupsd restart

With the UPS connected and the service up and running, I could now see more detailed specs. The output for apcaccess looks something like this:

APC      : 001,043,1007
DATE     : 2022-02-04 21:49:08 +0200  
HOSTNAME : deskmini
VERSION  : 3.14.14 (31 May 2016) debian
UPSNAME  : deskmini
CABLE    : USB Cable
DRIVER   : USB UPS Driver
UPSMODE  : Stand Alone
STARTTIME: 2022-02-01 05:45:50 +0200  
MODEL    : Smart-UPS 750 
STATUS   : ONLINE 
LINEV    : 241.9 Volts
LOADPCT  : 9.1 Percent
BCHARGE  : 100.0 Percent
TIMELEFT : 98.0 Minutes
MBATTCHG : 5 Percent
MINTIMEL : 3 Minutes
MAXTIME  : 0 Seconds
OUTPUTV  : 241.9 Volts
SENSE    : High
DWAKE    : -1 Seconds
DSHUTD   : 90 Seconds
LOTRANS  : 208.0 Volts
HITRANS  : 253.0 Volts
RETPCT   : 0.0 Percent
ITEMP    : 33.3 C
ALARMDEL : 30 Seconds
BATTV    : 27.4 Volts
LINEFREQ : 50.0 Hz
LASTXFER : No transfers since turnon
NUMXFERS : 0
TONBATT  : 0 Seconds
CUMONBATT: 0 Seconds
XOFFBATT : N/A
SELFTEST : NO
STESTI   : 14 days
STATFLAG : 0x05000008
MANDATE  : 2007-03-16
SERIALNO : < serial number goes here >
BATTDATE : 2007-03-16
NOMOUTV  : 230 Volts
NOMBATTV : 24.0 Volts
FIRMWARE : 651.13.I USB FW:7.3
END APC  : 2022-02-04 21:49:10 +0200  

I’ve seen some unintentionally funny abbreviations, but CUMONBATT takes the cake.

Battery runtime test

To test out the state of the batteries and understand how the UPS handles power failure scenarios, I decided to pull the plug and see what happens next.

The first thing you’ll notice is the beeping. It’s not quite as loud as a smoke detector, but I couldn’t blame my neighbours for thinking that this was the case. To silence the UPS, just press the power on/test button (the one with the | on it).

While on batteries, the UPS emits a hum that’s not too different from the one present near some power stations and transformers. Other than that, it doesn’t get awfully loud or anything, at least on a small load.

I continued using my laptop and monitor while keeping an eye on the apcupsd daemon and apcaccess output. The estimated runtime was around the 60 minute mark and it ended up being quite close to the actual figure.

Around the 10% capacity mark apcupsd triggered a shutdown of the server. The UPS still provided power at that point. I decided to supply the UPS with power again and turn on my server. To my surprise, the UPS now cut off all output power and started blinking some lights, one of them indicating some sort of an issue with the battery. I turned everything off, then on again and the UPS was just fine.

I’m not sure if the last incident was intended behaviour or my inexperience causing an issue.

Power usage

The goal of moving my setup to an efficient machine was to experiment with how low can I go with the power usage of my setup. Unfortunately with the introduction of the UPS I have wiped out almost all of the gains that I got from the move. The UPS itself seems to constantly consume around 17-18 watts, which is just a bit more than my server itself is using.

However, I hope that the reliability improvements and various protections that come with the use of an UPS make up for that downside.

LAN resilience

With the initial setup that I had, the server and the UPS were both on my desk. In the event of a power outage, the server, my laptop and all the connected peripherals would continue working. The only issue is that the networking won’t work in such a scenario, as both the ISP provided modem-router box and my own router would be knocked offline. What use is a server disconnected from the network?

Since I use my laptop for work, I’m not too worried about it losing power. Laptops have batteries, so in the event of a power outage, it will just continue running off of the internal battery.

I’ve configured my router to resolve my own domains internally, meaning that even when I cannot connect to the Internet, my locally hosted services will keep working in some capacity.

With all of this in mind, I decided to move my setup closer to my main router.

Power goes out, my server and local network will still keep on working for up to 2 hours due to the low combined power usage.

Desktop integration

When testing the UPS on my main workstation (ThinkPad T430 running Fedora 35), I was positively surprised by the out of the box experience that I had with the UPS. With the USB cable connected to the laptop and the apcupsd service running in the background, I could see the remaining battery life of the UPS the same way that the laptop battery shows up.

And once I ran out the batteries, I got a notification saying that the UPS was about to run out of power soon.

I love seeing these relatively niche use cases supported out of the box. Props to the developers behind these quality of life improvements!

Conclusion

Would I have gone out and bought this UPS if I didn’t receive it as a gift? Probably not, because I didn’t see much of a need for one.

Now that I have it, will I keep using it? Definitely, assuming that the inevitable battery replacement cost is reasonable and the process of changing them is easy enough for an enthusiast to handle.

From the manual:

For Professional Business Applications – Not For Consumer Use

Look, ma, I’m a professional now!

The tagline for /r/datahoarder reads: “It’s A Digital Disease!”. I agree.

At some point I realized that the pursuit to hoard all the things will just keep on consuming more and more of my time and money. Storage is cheap up until to a point, once you find yourself tracking hard drive prices via camelcamelcamel and being excited that a 12 TB external hard drive is at it’s lowest price, you may start suspecting that you have a hoarding issue.

I don’t have anything against what people at /r/datahoarder and other archival efforts do. Data archival done by volunteers is what allows us to preserve history and cultural artifacts of our time (YouTube videos, memes etc.). I just had to find a solution to my situation before it got too bad.

Something is better than nothing

Once you find yourself with terabytes of data and decide to scale down, you will need to make some changes. Most of the time this results in deleting a lot of data that you don’t care about that much. This works, but there are some files that will be difficult to part with.

In my experience, most of the storage is taken up by media files. I would still like to have a copy of some videos, but if they are downloaded in 1080p or 4K resolutions, they will take up a lot of space.

That’s when I decided to change my approach with data like that. I would still like to have a copy, but it doesn’t necessarily have to be full quality all the time. When you rewatch YouTube videos that you have archived from 2006-2010s, you’ll notice that the video quality is bad. That doesn’t mean that you enjoy the videos any less. Watching these old videos is just an act of going through your memories and feeling nostalgic, the medium itself is just the spark needed to revive the memories.

With that, I decided to start following this simple rule: something is better than nothing.

I don’t need 4K versions of all the channels I follow. bestvideo[height<=1080] in yt-dlp configuration will limit the resolution of all the videos you download to 1080p, which is still good enough.

Data, value, and you

If you’re someone like me who watches YouTube frequently, then you probably have some channels that you care about a lot.

With channels that I truly care about, I still keep the highest quality copies of all their videos on my server. This has a real cost in terms of storage, but the value I have got out of the videos is much greater than that.

However, there are some exceptions. LinusTechTips is a great channel and has had a huge influence in my life. However, they upload almost daily and do long streams from time to time. Archiving the whole channel is just not feasible at my scale. Even when limiting the resolution to 480p, the full channel archive takes up over 300 GB of space in 2022. Instead, I’ve opted to recognizing the videos that are the most memorable, such as the petabyte project, and have archived those individually.

There’s also a certain type of data that isn’t that common: data created by you and your friends/family. Extra care and focus should be put on this type of data, because it’s very unlikely that someone else has a copy of gigabytes worth of cat pictures you’ve taken over the years. Make plenty of backups and don’t worry too much about hoarding it now, as long as it’s not a big burden for you.

Constraints

Constraints force you to get creative. It also helps with tuning down your data hoarding habit.

Set physical limitations to the data you can store. For example: limit yourself to running only two hard drives at a time. The only way to get more storage is to upgrade vertically to bigger hard drives, which is naturally throttled by the relatively slow increases in hard drive sizes.

For reference, my current setup has some pretty harsh limits when it comes to storage. The only way to get more is to spend a lot of money on bigger SSD-s.

The Internet Archive

If you don’t want to go through all the hassle, but still want to contribute to archival efforts, then you can donate to the Internet Archive instead. Buying and running all those hard drives isn’t free, you know.

And if you’re planning on uploading data to the Internet Archive, then please do not abuse it. Only upload data that you feel is worth preserving.

Around a month ago I shared my blog post on HackerNews. I guess I lucked out with the choice of the topic, because it brought out a lot of enthusiasts who shared their own experiences with older machinery that still works in 2022. I really appreciate the feedback and the experiences shared!

Anyway, what is noteworthy in my opinion is that my blog runs off of a residential connection that has an upload speed limit of 50 Mbps. Once I noticed the post getting traction, I was worried for a moment. It’s not a rare sight to see a post on the front page of HackerNews and see it being down due to all the attention it got.

Somehow, my post managed to weather the storm. Here’s what happened.

Analyzing the logs

My page doesn’t have any sort of first-party or third-party analytics software running. Tracking users across the web is a big no-go for me and I will live by that, especially on my website. I do have nginx logs, though.

After a quick look around, I found goaccess, a tool that can parse nginx logs and put together some basic statistics. I found it to be good enough for my purposes. Here are some notable statistics.

To understand the HackerNews effect, note that the post was published on 2022-01-10 07:42. These logs also include requests towards services that I host myself.

The summary of the week during which one of my posts got popular on HN, as reported by goaccess, looks like this.

During quiet days, nginx logs around 50K requests a day. On 10th of January, there was a 12x increase. The effect was also noticeable the next day, during which I assume people caught up with their 100+ tabs that they usually have open. This may also be down to the post being shared around as well.

Based on this statistic, it seems that the post got around 75K views, either by real people, crawlers or preview generators.

When we look at the number of requests made towards specific assets, such as the images in the post, we can see that real impressions are likely to be around 55K. The first image has been downloaded more than the other two, likely indicating that it was used as the preview image when the article was linked on other sites.

This statistic also highlights a surprisingly large cost of web fonts.

The OS results are likely to be biased, since my self-hosted services go through the same reverse proxy. The “Unknown” section is likely related to crawlers and other bots pinging my server.

No surprise there: Chrome is the most popular browser used to reach my site. Firefox makes up a good chunk of hits as well, but a lot of those are likely requests made from my own machine.

The time distribution of the requests is interesting. You can notice the background noise associated with bots and crawlers that occurs during the night. My post was posted in the morning in my local timezone (UTC +2), which was around the time EU people wake up and get to work. The other spike around 14-15:00 can likely be attributed to our friends over the Atlantic ocean.

16:00 in my local time is a bit special. A lot of things coincide with this time:

• the review embargo for the latest highly sought after CPU/GPU is over, followed by a barrage of videos on this product in YouTube
• the stock market opens and news start rolling in about some big moves
• if Slack is having issues, then it’s around this time, because I can only assume that people get to work and do a production release at the start of the workday.

To my surprise, my server survived this and there aren’t too many issues. The 400 and 500 errors are likely attributed to bots trying to exploit my server and crawlers visiting links that are not valid any more.

There’s likely a bias here as well regarding Europe.

How I built the blog

When I started writing blog posts regularly, I had some principles that I wanted to stick to:

• the goal should be on the writing, not the part where I build the website itself
• the site has to be static: build it once and deploy it, which should result in fewer opportunities for attacks and reduced load on the CPU

With this in mind, I decided to stick with Hugo. Hugo is just a single binary, has plenty of themes to pick from, and it seems like a reasonable-enough choice for a site. It’s not the easiest to use as you’ll be mainly operating in Markdown, and with the theme that I use, I have to copy-paste a <figure> section, change the image name that it points to, and make sure that I didn’t mess anything up. However, because this approach rules out me getting hit by the hottest Wordpress plugin vulnerability of the week, I think it’s a fair trade-off to make.

I picked this theme by Radosław Kozieł and customized it so that it has a landing page of sorts as well. I haven’t updated it yet and it has some flaws, but it gets the job done, and that’s what matters to me the most.

Hosting media, such as images, seems like a no-brainer, just put them on the page and be done with it. Pictures taken with an iPhone SE 2020 are quite big, though (3-4 MB per image), which will result in the page loading very slowly. To avoid images bloating the size of the page too much, I have set up a system where I keep the original images in one folder, copy them to another one, run mogrify -resize 1024x768 -quality 85 *.jpg to keep the images small, but still detailed enough, and then deploy those converted images along with the rest of the blog.

With converted images, a page with three images can fit in less than 1 MB of transferred files without any issues. With original images, the same page would require 10 MB of files to be transferred. The math is simple: with a limited uplink, optimizing the images results in your server being able to serve 10x more requests.

Residential connections and DNS

If you’re like me, then you probably have a crappy router/modem box from your local ISP, and a dynamic IP address that usually changes whenever you reboot said box. This presents a challenge when you try to host anything from your residential connection since the IP address could change at any time.

To resolve this, there are two options I’m aware of: dynamic DNS providers, or setting up a script that talks to your DNS provider over a standard API.

I haven’t personally used any dynamic DNS providers, such as DuckDNS, mainly because my domain registrar has a handy API that I can use to automatically update my IP address with any time it changes. And yes, I did have an issue with the script where I triggered a change every minute, resulting in an angry e-mail being sent to me. Free tech tip: only propagate changes when the IP address actually changes.

There’s one downside with this issue: you can set your domain TTL (time-to-live) low, but no matter how low you set it, there will be a period of time after an IP address change where some DNS servers will point to your old IP address. This is an availability risk you have to consider when setting up a service on a dynamic IP address.

Future steps

At one point I took a look a the assets of the Hugo theme that I use and noticed that it includes a dependency named prism.js, which might be a good thing to include if you want to display code snippets with syntax highlighting. My issue with it was that it was included on every page load and took up a significant chunk of transferred data. I decided to remove it, and just like that, the page loads even faster.

There’s still room for improvement. The page currently also includes some custom fonts. If I decide that a built-in font is good enough, then there’s potential for an additional 0.3 MB of savings.

The web is bloated enough already, but at least I can control what I send to the client machines on my website.

Conclusion

If you don’t want to go through all the hassle and just want a website up and running, just get yourself a cheap virtual machine at a cloud provider, or use something like Github Pages. Those solutions are less likely to hit limits with bandwidth.

If you like tinkering and the decentralized, self-hosted web to exist (not Web 3.0!), then feel free to use this post as inspiration for your very own website. There’s a lot that you can do with limited resources, and it’s fun to push the limit.

And as a quick tech tip: you can use The Wayback Machine as an insurance policy for your website. If you are concerned about it going down, have them take a snapshot of it and link it somewhere in a comment.

I wasn’t happy with my “one machine that does it all” setup, which is why I finally bit the bullet and got myself an ASRock DeskMini X300 bare-bones kit. This is a fantastic little desktop PC kit that is very small, quiet and can still be configured to have ridiculous amounts of power. In the past. I once built a low-power server based off of the predecessor, the DeskMini A300, and instantly fell in love with this form factor. Since then, I’ve tried to come up with a reason to get one, and finally I’ve got one.

Configuration

Because this is a bare-bones kit, you have to supply your own parts.

My configuration:

• CPU: AMD Ryzen 7 5700G, locked to 35 W TDP in UEFI settings.
• Cooling: Noctua NH-L9a
• RAM: 2x 16 GB DDR4-3200 SODIMM
• NVMe storage: 2x Samsung SSD 980 500GB
• SATA storage: 2x Samsung SSD 870 QVO 4TB
• USB storage: 1x Samsung SSD 860 EVO 250GB in an IcyBox USB enclosure

The use case for my kit is simple: it’s my server that hosts all my services and data, including this blog you’re reading.

Why this setup in particular?

• Most of my storage needs can be served by two 4 TB SSD-s.
• 8 cores with SMT are perfect for multithreaded workloads, such as Jellyfin transcoding. This also works well for just running a bunch of services on the same machine.
• Can support up to 64 GB of RAM, which should be plenty in the foreseeable future.
• Supports fast NVMe SSD-s, which are perfect for more IO-sensitive workloads, such as databases.
• Low power usage. Even with my almost maxed out kit, it still uses around 14 W in idle. At most this PC can use around 50-60 watts due to the CPU TDP limit and SSD-s hitting full load.

Why is this form factor so good?

Desktop PC-s are big. Really big. While the small form factor PC movement has become more popular lately, the most commonly available cases and parts still end up taking a lot of space and resources. There are legitimate use cases for these types of setups, especially if you’re running video renders, scientific simulations or a lot of VM-s as part of a lab. Not every workload needs such a big setup, however.

The DeskMini X300 looks like a miniature version of a desktop PC in a tower case. Similar shape and features, but at a fraction of the size and raw resources usage. One might say that it even looks adorable, similar to how Raspberry Pi-based builds look compared to desktops.

This PC takes up just 1.92 liters in volume (excluding the power adapter). For comparison, a micro-ATX case like the Fractal Meshify Mini C, takes up a whopping 33.4 liters. I don’t have the raw numbers regarding material usage or the weight difference, but this comparison should give you an idea of the material usage difference between the two.

Caveats

I love this piece of kit, but I do have to acknowledge that there are some trade-offs that you should be aware of.

Starting with the obvious one first: you’re trading off some expandability. No discrete GPU-s will fit in this thing, not even an external one due to lack of Thunderbolt. On the other hand, if you don’t foresee a need for such additions to your PC, then the DeskMini will be a great choice.

Something that’s specific to the X300 is the lack of USB connectivity. You get two USB 3.0 Type-A ports, one USB 2.0 port and a USB 3.2 Gen 1 Type-C port as well. If you have a USB hub, then this won’t be much of an issue, but this configuration is quite limited compared to most modern desktop motherboards.

The motherboard inside this thing can also be a point of concern. It’s a non-standard form factor, meaning that in case of failure you’re unlikely to easily find a replacement. The most obvious step here would be to either repair it or get another DeskMini kit to replace your existing one.

One aspect of the DeskMini X300 that you may care about is the fact that the power supply is an external 19V 120W power brick. This does mean that replacing the power supply won’t be much of an issue, but you do have to take this into account when planning your build. The power brick is quite big and has to be placed somewhere, after all. In my setup, I’ve opted for sticking it to the underside of my desk with double-sided tape.

GPU-s, APU-s, and the future

For anyone who’s followed the GPU market in 2021/2022, the ridiculously high pricing of GPU-s shouldn’t come as a surprise. As a firm believer in thinking that low power computing is the future, I think that this presents a great opportunity to start thinking differently about our computing architecture.

AMD introduced the concept of APU-s back in 2011. In its current iteration in the form of AMD Ryzen 5000 series APU-s, they provide a compelling package, combining powerful CPU cores with a decent GPU that’s perfectly capable of many tasks, including low-spec gaming.

I’ve previously covered what I think about the state of computer hardware and the resource usage associated with it. It’s possible that we’re simply in a time in history where resource abundance allows us to push the limit of what computer hardware is capable of without thinking much about the cost. At some point we do need to address that issue, and I think APU-s will play a role in it.

We already have examples of what APU-like chips can be capable of in the form of the last two console generations. To get an idea about the power usage numbers, check out the power usage statistics shared by Sony for the PS4 and PS5. The power usage of the PS5 is still almost 200 W under load, which isn’t that great, but that’s likely due to the high performance goals set for the console itself. You’d have to give up some image quality or performance for a more efficient setup, which is something the marketing department isn’t probably very happy about, given that they’re in direct competition with Microsoft with their newest XBOX lineup that is also pushing the limits of the custom APU inside it.

The integrated GPU solutions that AMD and Intel have put out are becoming more and more capable. Even an AMD Ryzen 3 2200G from 2018 with its measly 4 cores and a Vega 8 GPU is still capable enough to run GTA V at 1080p with low-medium settings smoothly (60 FPS or more). You do have to make a conscious trade-off in image quality and performance when relying on APU-s, but if you’re willing to make that trade-off, you can get a decent gaming experience.

Even the Steam Deck is shipping with a custom AMD APU. What makes this one different from AMD-s previous APU-s is the inclusion of a modern GPU core based on the RDNA 2 architecture, which should be a big jump from the relatively old Vega-based GPU-s that desktop APU-s have shipped with.

If it weren’t for the progress in gaming tech, such as high refresh rate monitors and ever-increasing system requirements for newer games, APU-s would be a good default choice for casual gamers, negating the need to get a huge box that you have to somehow also fit on your desk.

I have run on an APU-based setup and was happy with the experience, unfortunately an APU-based machine just doesn’t cut it for virtual reality workloads, yet.

Comparison to TinyMiniMicro PC-s

ServeTheHome’s TinyMiniMicro series gives a good overview of the options available for someone who wants a very small 1 liter PC to do all their work on. Since these machines are also very small, power-efficient and yet still quite powerful, I think a comparison between the two is a reasonable one.

One area where the TinyMiniMicro PC-s win is an obvious one: the size. They are about half the size of the DeskMini X300: 1 liter vs 1.92 liters.

TinyMiniMicro PC-s are also abundant on the used market with various configurations. In most cases, the storage and RAM are upgradeable, and with some older models you’ll find that the CPU is also socketed. Depending on your market, this means that you may be able to get a whole PC for cheaper than the DeskMini + the rest of the components (CPU, RAM, storage, Wi-Fi etc.).

The size and cost benefits do come with some compromises that are not acceptable for my use case. At 1 liter, the only option you have is to include a blower type cooler in your PC. This type of cooler is similar to the ones found in laptops and has in my experience resulted in a lot of annoying noise under heavy load.

The UEFI settings on these machines may also be quite limited. Sure, if you just need the basics, these will be fine, but if you want to configure the fan behaviour, you might find yourself unable to do so. On the DeskMini, you can setup the CPU fan to run just the way you like it. In my case, the fan only turns on when the system is under load, the fan is completely stopped while the machine is idle.

The DeskMini also allows you to change out the CPU cooler with anything that’s AM4 socket compatible. The case of course sets some limits to that, unless you’re a madlad who doesn’t let boundaries bother them. With the Noctua NH-L9a AM4, the setup is very quiet, especially once you limit the CPU TDP and adjust the fan curve. In my experience, the “coil whine” noise that the system produces under load is more audible than the CPU fan itself.

When it comes to storage options, the DeskMini is an obvious winner. In modern TinyMiniMicro machines that I’ve seen, you can at most install two NVMe SSD-s. This is a step up from older machines and you can still do a lot with that, but the DeskMini trumps that with the addition of two SATA ports, which allows you to go crazy with the storage configuration.

Both the DeskMini and TinyMiniMicro-series PC-s are fantastic, but for my use case the DeskMini X300 was the obvious choice.

Tech tip: VESA mounts, for what?

The DeskMini X300 is so light and small that you can easily attach it to a monitor arm using a piece of foam and long velcro strips.

Future plans

I’m hoping that the current DeskMini-based server configuration is the one that will last me a while. The obvious upgrades in the future would be to max out the RAM at 64 GB and upgrade the NVMe and SATA storage to whatever options are available in the future. I’m not expecting the DeskMini X300 to get a CPU upgrade, which isn’t too tragic, given that the Ryzen 7 5700G has 8 fast cores.

My adventures in low power computing may take me to another setup in the future. Until then, I’ll see how the DeskMini does. So far, it has met my expectations in performance, power usage, reliability and noise.

Future of desktop computing?

Predictions are difficult to make, especially in turbulent times that we’ve found ourselves in. I do hope that this form factor takes off and that we reevaluate our current computing workloads to be more suitable for machines that use a lot less power.

Constraints often result in clever tricks and fantastic end results. Hopefully we’ll see something similar in the future.

To get an idea of what you can achieve with very limited hardware, check out the Coding Secrets YouTube channel that showcases all the tricks and thinking behind some of the most impressive effects on game consoles from early 90’s.

Sure, you can argue that not having to think about such limitations can help create solutions faster and that definitely has some value, but at some point we need to start optimizing. Computing resources are cheap right up to the point where every app on your PC thinks the same way.

I’ve been running my all-in-one PC for a while now. It was my desktop, my NAS and my gaming PC. However, during the next couple of months I kept hitting small bumps along the road. Most of these were quite straightforward to fix, but there is one that finally convinced me to go back to a simpler setup.

iSCSI

As mentioned in one of my previous posts, I had set up the storage on my gaming VM over iSCSI. If you don’t know what iSCSI is: think of it like a hard drive that you can hook up to your machine over the network. This setup had a couple of benefits: the amount allocated to the VM was flexible and I would always have a snapshot of my game data in case the Windows VM decided to nuke itself.

What I didn’t take into account was the configuration. tgt allows you to define the IP address of the client machine, which can be seen as an additional layer of security. My gaming VM has two virtual NIC-s, one pointing to the actual physical network, and another one to a virtual one. This resulted in the iSCSI connection being flaky on startup: the disk was not present in the VM on first boot, but was there after a reboot.

This issue was gone once I removed this restriction from the iSCSI target configuration on the Linux VM. My best guess is that the iSCSI stack in Windows might have tried to initiate a connection over the wrong NIC, depending on whichever one was brought up first.

After “solving” this issue, it had been smooth sailing.

GPU audio issues

When the Windows VM had been on for a while, the HDMI audio output would start to glitch out after a while, with the audio sounding slow, robotic and garbled. After doing some research, I found some resources that mentioned enabling MSI interrupts in Windows. The guide for this can be found over at Guru3D forums.

After tweaking the Windows registry settings and reminding myself that I do have snapshots to fall back to in case I mess up, I got the issue fixed.

GPU driver issues

Whenever there’s a new big and popular game released, NVIDIA usually releases an updated version of their GPU drivers. I try to keep my software up to date, so I usually install these updates when they come out. Just one problem: when I upgrade the GPU drivers in the VM, the screen will stay black. After some time passes, I have to force a reboot using virt-manager.

I have not looked into this issue yet as it only happens during driver updates, but it’s definitely an annoying thing to have.

UEFI and CSM

UEFI updates have a bad habit of resetting all the custom settings that you have made, and at least with the motherboards I have used, the settings cannot be restored if the backed up settings are from an older UEFI version. This means that I have to navigate the UEFI quite often to configure all the settings that I need to have in place for VFIO to work.

There is one setting that I didn’t expect to cause that much trouble though: CSM support. CSM (compatibity support module) is generally enabled by default and allows you to boot off of legacy operating systems if needed. I don’t have that requirement, so I went ahead and set my boot options to boot in UEFI mode only.

Reboot, start up Fedora, and boom, anything VFIO related is spewing errors like crazy.

Turns out that enabling UEFI only mode in UEFI settings somehow grabs the GPU and causes problems when you try to then pass the GPU to a VM. While in this mode, I noticed that the GPU was also shown in a separate menu in UEFI settings. Enabling CSM support fixed this issue for me.

VR and USB issues

With the situation in the world being less than optimal, I decided to spice up my gaming setup with the addition of a HTC Vive Cosmos in case in-door activities become much more popular suddenly.

First issue: I can’t get the damn thing working. My gaming VM setup has the inateck KT4006 passed through so that I can plug peripherals to the PC and make them available to the gaming VM with ease. After plugging everything in according to instructions and turning the link box on, I’m met with a barrage of USB connected-disconnected sound effects. 6 or 7 cycles later the sound stops, and with that do the rest of my USB devices.

I took the USB 3.0 card and tried it in an older machine that doesn’t have a fancy VFIO setup on it. Same issue. However, when connected to an USB 2.0 port, the setup shows a warning in VIVE Console, but at least it works! Since my PC literally has no USB 2.0 ports, I had to get a bit creative and plug an USB 2.0 extension cable between the PC and the USB 3.0 cable coming from the link box.

With the help of my friend I got confirmation that the issue is indeed with my USB ports. I also ordered a different USB PCIe card from inateck, this time the inateck KTU3FR-4P. I made the assumption that since the VIVE support page also recommends an inateck card, then they must be fit for purpose.

Nope. The second card apparently has the same chipset and the same issue.

What did end up working was a random VIA USB 3.0 PCIe card, which shows up as VIA Technologies, Inc. VL805/806 xHCI USB 3.0 Controller on my machine. Just one issue: it has PCIe device reset issues. If I shut down the Windows VM and start it up again, I would sometimes get a bunch of IO_PAGE_FAULT errors in my kernel logs. No worries, guess I’ll just avoid rebooting a notoriously reboot-happy OS.

Latency: the straw that broke the camel’s back

If there’s one thing you don’t want to see in a VR gaming setup, it’s latency.

I’ve covered the tweaks I’ve made to my setup in a previous post. Turns out that those might not be enough. The CPU isolation using systemd works well in general, but not with kernel threads.

Whenever I did something IO heavy on an NVMe-based btrfs file system, it would introduce stutter to the gaming VM. Running a scrub operation on the filesystem was enough to put a huge load on the CPU, likely due to the raw speed that the NVMe SSD-s can provide. This results in the gaming VM experiencing stutter that ruins the whole experience.

Here’s an illustration.

Similar issues could be observed whenever I tried to run another VM that shares the same cores.

I looked into alternative solutions as well, and it does seem to be possible to also isolate those CPU cores from the kernel. They will probably work to some extent, but implementing them feels a bit hacky at the moment. I’m hoping that in the future these kinds of tweaks will be exposed as a simple checkbox in virt-manager itself.

At that point I had enough. If I’m not going to be able to use the machine for other purposes at the same time without experiencing all these issues and having to work around them, then I’ll just go back to a simpler setup. I’ll lose some of the benefits, but at least I don’t have to spend time debugging all these issues.

Conclusion

VFIO and virtualization are still interesting topics to learn about and try out, but for my use case they are just too limiting. I’d imagine that an use case that didn’t have such strict latency requirements would still be able to run just fine, unfortunately gaming just isn’t one of them.

This experience has been quite fun in general, but I’m calling quits, at least for now. The one machine that does it all turned out to be a jack of all trades, master of none.

Do you have a need to host Wikipedia on your computer?

Or a StackExchange site, like Super User?

It’s easier than you think!

Step 1: download stuff!

The Kiwix project provides ZIM files for many popular websites, including Wikipedia. These files can be downloaded over at Kiwix wiki.

Take a look at the content packages available and download those that you are interested in.

Step 2: install Kiwix

Kiwix provides all sorts of clients for different operating systems. If you don’t want to go crazy with setting up a server and you just want to browser your offline copy of Wikipedia or any other resource, then go ahead and use that.

However, if you want to make the resources available for other people as well, then you need to install kiwix-serve. On Debian and Ubuntu systems, this is just a sudo apt install kiwix-tools in your terminal.

Once you have it installed, all you have to do is to run kiwix-serve -p 8080 your-file-here.zim and your content package is served to anyone who can connect to your machine with a browser over port 8080.

If you have multiple ZIM files in the same folder, then you can serve all of them at once with something like kiwix-serve -p 8080 *.zim.

Step 3: enjoy!

You should now be up and running with your very own hosted Wikipedia instance! Just keep in mind that your copy might not be 100% up to date. The Kiwix project does repackage these sites quite regularly so keep an eye out for updates.

Tips and tricks

I have set up my own Kiwix instance and the way it works is quite simple.

I use Deluge to download and seed the content packs that I have downloaded.

Then I have also set up a systemd service that automatically starts Kiwix on boot. It is located at /etc/systemd/system/kiwix-serve.service and looks like this:

[Unit]
Description=Serve all the ZIM files loaded on this server

[Service]
Restart=always
RestartSec=15
User=kiwix
ExecStart=/usr/bin/bash -c "/usr/bin/kiwix-serve -p 8080 /path/to/your/files/*.zim"

[Install]
WantedBy=network-online.target

Note that I have set up a separate non-interactive user for this purpose. The path to the ZIM files simply refers to the download folder of the Deluge torrent client.

Making this available to the world is simply a matter of either exposing the selected port to the world, or using nginx as a reverse proxy. There are plenty of guides available on how to achieve the latter so I won’t go into detail about that here.

That’s it!

2024-09-05 update

It’s been a fun experiment, but LLM-bots are overwhelming my Kiwix instance and I can’t be bothered with setting up any protections right now, so the kiwix.ounapuu.ee instance is shut down for now.

As you might have read from my previous post on this topic, I have a pretty neat “cloud” gaming setup running. I have one powerful desktop PC, one virtual machine with a GPU attached to it, and client machines that can be used to stream games from. To keep things simple (and because my current ISP is unbelievably bad), I have so far used streaming only over my local network.

However, there were some pretty annoying issues with this setup:

• The AMD GPU encoder is bad. 720p streaming was the best I could do if I wanted reasonable performance.
• The Windows 10 VM would sometimes not be able to start the host encoder, and the GPU drivers reported issues. This was fixed with a couple of reboots to the VM and I suspect that this may be related to the notorious GPU reset issues that AMD cards are infamous for in the VFIO world, but I wasn’t interested in digging that deep into that topic.
• The stream would sometimes randomly crash or lag. This might have been down to the AMD GPU or Parsec itself.

I read about this topic for a while and found that the overwhelming number of threads related to this topic recommend an NVIDIA GPU and Moonlight for the best streaming experience. And so I went out to get one.

The GPU

It’s 2021. The world is on fire and GPU prices are insane. With that in mind, I tempered my expectations and went for something simple: a NVIDIA GTX 1060 6G by MSI. Simple, relatively modern for my games and the price was around the retail price 5 years ago (260 EUR), which is the best you can do in these lousy times.

But hey, it works! From the testing I did in a previous post, I knew that NVIDIA had finally stopped intentionally throwing error code 43 in their drivers, which made this option viable. Replacing the GPU was simple, I just had to change the PCI device ID-s according to the VFIO setup guide and changed the devices attached to the Windows 10 VM using virt-manager, and that was it!

Parsec: AMD vs NVIDIA

Since I was already using Parsec for my streaming purposes, I decided to do a quick test to see what the difference is between the two cards.

The previous GPU was an AMD Radeon RX 570 4GB by Sapphire. These two GPU-s are in the same performance class, which makes for a good comparison.

Here are some results taken in two quick testing sessions in Dirt Rally. Take these results with a spoonful of salt as these have not been conducted with any kind of standards. I just picked the same graphics preset, resolution and let it run with Parsec statistics view open.

H.264

AMD RX 570: 10ms

NVIDIA GTX 1060: 3ms

H.265

AMD RX 570: 8ms

NVIDIA GTX 1060: 3ms

The difference in the encode performance is huge, especially if you consider that the time between two frames is 16.6ms for a 60 FPS stream. Some of that budget is also taken up by network and decode latency in the client, so the shorter the encode time is, the better.

Parsec is fine, but I have had some trouble getting GPU-accelerated decode working on my Linux clients, and it also lacks an iOS client. While the latter isn’t a deal-breaker, the first one definitely is.

Moonlight: an open-source gem

Moonlight is an alternative to Parsec. It implements the NVIDIA GameStream protocol and has clients for most common platforms. My testing has been performed on Linux clients running Fedora 34, a Windows 10 laptop (ThinkPad X230), iPhone SE 2020 and an old Google Nexus 5. I also made an attempt to run the Moonlight client on a Raspberry Pi 1 B+, but it seems that it is simply too old for the Moonlight client, resulting in an error on startup.

To set up Moonlight, I opened up the client and tried the automatic detection over the network. Unfortunately that did not yield any results, so I input the IP address of the Windows VM manually. That worked well. This initiated a pairing process that I had to continue using Parsec, because you need to input a code on the host machine. After that, I had to quit Parsec and start streaming using Moonlight.

OK, not quite yet. Moonlight is able to detect the games installed on your system, but for a remote desktop experience, you have to do a small tweak for it to be equivalent to the experience you get from Parsec.

One thing that I noticed at the start was that the image quality on the desktop isn’t as good as on Parsec. However, it more than makes up for it in-game. Moonlight just works. There are plenty of knobs you can turn as well, including configuring the image quality by setting the resolution, framerate and network limits.

What makes Moonlight more suitable for me is that the Linux client packaged as a Flatpak is able to use the GPU for hardware-accelerated decode. The experience is great on a Linux machine and it just works, which is not something you can always expect to happen. Even an older XBOX ONE controller connected over Bluetooth works just fine over Moonlight!

The only issues that I had with Moonlight were later determined to be caused by networking issues. This shows up well on the integrated statistics that Moonlight provides using the Ctrl + Alt + Shift + S key combination under the Network jitter field. After stopping some services on my network, the issues went away.

The setup: one month later

After a month of use, I have made some adjustments to the setup. I used to keep the VM running so that I could just jump in and start playing whenever I liked it, but due to a tweak I made in the name of improving gaming performance, half the CPU cores on my system were not available for other purposes, such as suffering through IntelliJ indexing the project again. I decided to only power on the VM if I actually was about to take the time to play it. I haven’t yet figured out the best way to make this more convenient, though. I did make the “Calculator” key on my keyboard start/stop the VM on demand, but then GNOME started ignoring the shortcut I had setup and opened the calculator anyway, so that was a short-lived tweak.

A bigger change that I made to the setup was to store all my games on my ZFS pool that lives in another VM. For that, I setup an iSCSI target using this guide as a starting point . I used the tgt package instead. That has worked out surprisingly well, but the iSCSI target does not want to properly connect in the Windows VM on the first boot, which is mildly annoying. I soon discovered that spinning rust just does not cut it and got more SSD-based storage, otherwise other processes on the NAS could wreak havoc on the gaming experience in games like GTA V. Those issues were manifesting as long loading times and objects popping in too late, resulting in half the roads and building being missing while driving around the map. The iSCSI target solution also comes with the added bonus of having a proper backup of all my games. If Windows decides to blow itself up again and I don’t have a backup of the VM ready, then I could just reinstall it and not worry about downloading and installing all the games again.

Verdict

Overall, I’m very happy about the setup. I don’t use it all too often, but occasionally launching art of rally or BeamNG.drive on any random device I own is great. I do have some concerns about the future of this setup, especially with Windows 11 having more strict system requirements and newer games demanding more performance. For the time being, this works and I intend to keep it that way.

The one box that does it all: so far so good!

This is a follow-up to my previous post where I covered the VFIO setup in general. For many people that would have been good enough, but my goal with this setup was to have a powerful gaming setup that I could access from my living room PC with 20 meters of Ethernet cables between the two.

Cloud gaming?

The concept of cloud gaming has become more popular lately, with Microsoft, Google, NVIDIA and others offering services that allow you to stream games to your device. These setups usually include a powerful server in a datacenter rendering the game and sending the compressed video stream to your device. This works surprisingly well, but due to high cost, poor internet connectivity in many places in the world and licensing issues , this setup might not work well for everyone.

However, if you already own a moderately powerful gaming PC and would like to use it to play games you already own over the network, then there are plenty of solutions that can get you there.

Here are some options:

• Steam remote play: works for games launched via Steam.
• Parsec: able to stream the whole desktop, including games.
• Moonlight: same, but only usable on hosts with NVIDIA GPU-s due to the implementation relying on NVIDIA’s GameStream protocol.

In the past, I’ve had OK results with Steam remote play, but the limiting factor has been the reliance on Steam. I have bought a couple of games from GOG as well, which I could probably also play remotely by launching them via Steam, I just don’t want to go through the hassle.

For this setup I’ve opted to go with Parsec. It’s not perfect, but it’s still good enough for our purposes. The GPU we’re using (AMD Radeon RX 570) also limits our options because it won’t work with Moonlight.

The installation of Parsec is pretty straightforward: just install it on your host and client machines. In my case the client machine is a Lenovo ThinkPad X230 that was brought back from the dead . It’s not powerful, but at least it can do H.264 hardware decoding and it uses only 12W of power when idle, making it a perfect candidate for testing this out. It’s also worth mentioning that both machines are connected on the local network using an Ethernet connection to avoid Wi-Fi becoming a bottleneck.

VFIO, gaming and you

What turned out to be a bigger hurdle was the performance of games inside the VM. After getting everything running with the VFIO setup, I didn’t really spend any time trying to optimize the setup for the smoothest experience. When trying to actually run this setup, I ran into quite a few problems with performance which mainly manifested as stutters and unusually low framerates. Turns out that gaming has stricter latency requirements than other server workloads.

Thanks to the great guide in Arch Wiki, I could get a lot of ideas on what to try out to improve the performance of this gaming virtual machine.

Before doing anything, I ran some benchmarks to get a sense of what performance levels I’m dealing with. To test out each individual change, I used GTA V as the gaming benchmark since it was pretty good at pointing out any performance issues.

Here’s a list of things that I ended up doing:

• setting the CPU model to host-passthrough in virt-manager
• isolating CPU-s dynamically whenever the VM starts and pinning those CPU-s to the VM to avoid the host OS and other VM-s from using those cores
• setting the CPU governor to performance on isolated cores to rule out issues with the CPU not clocking high enough and switching between idle/load power modes: echo performance > /sys/devices/system/cpu/cpu[4-7]/cpufreq/scaling_governor
• enabling static hugepages to rule out issues stemming from poor memory access speeds
• disabling SMT in UEFI settings to get rid of one additional variable
• getting a 64 GB DDR4-3600 memory kit to allocate more RAM to the VM (16 GB) and leave enough for the host machine and other services

The golden rule of troubleshooting is to change one variable at a time and comparing results. Not every change that I made ended up being a positive one. For example, when configuring the VM to use 4 cores and 8 threads instead of plain 4 cores, I saw the framerates in GTA V drop 50%. I assume that the VM treated the “SMT cores” as real ones, causing the Windows scheduler to make incorrect decisions.

After playing around with this setup and accumulating these tweaks, I managed to get rid of most of the issues that bothered me during gaming, resulting in a much smoother experience. It finally felt like the gaming VM behaved like a machine with a 4 core CPU, 16 GB of RAM and an AMD RX 570 inside it.

At this point I’d consider the setup to be fantastic for someone who wants to play games with a VFIO setup. Since I was using this setup over the network with Parsec, I soon ran into more issues, but this time with the GPU.

GPU issues

Parsec and other similar solutions rely on the host GPU to encode the image into a video stream using the encoder present on the GPU die itself. You might see this being referred to as hardware encoding in settings. What I didn’t know before trying this setup is that AMD GPU encoders have a reputation for just being plain bad:

If the host has an AMD card, AMD is usually known for having worse encode than NVIDIA and even Intel. You should be fine at low resolutions. If all guests support H.265 and have it turned on, you may see better performance than with it off

And this shows. When trying to stream at 1080p, the result was a stuttery and inconsistent mess. Imagine trying to play at around 30fps with the frame timings graph resembling a heart rate monitor, that’s what it felt like. At 720p, the experience was so much smoother. Yes, the image quality suffers because of that, but at least it was mostly playable.

I decided to also try out if changing the codec from H.264 to H.265 has a significant impact. I did a quick test in Dirt Rally due to it having a benchmark loop mode and checked the statistics that Parsec shows. With H.264, I saw encode latency of about 10ms. With H.265, this latency was now at 8ms. Still not great, but it is technically a ~20% improvement. The downside of this for my setup is that the client laptop simply does not support hardware accelerated H.265 decode, which is something that Intel integrated graphics received support for in 7th gen CPU-s.

The GPU seems to also cause some trouble for Parsec, as it would fail to connect occasionally and reports a host encoder issue. This is usually overcome by rebooting the VM, which can get quite annoying after a couple of times. At other times, Parsec sometimes just froze during gameplay and caused the client to freeze at 100% CPU usage. Not what I’d call smooth sailing.

Another issue that I noticed by using MSI Afterburner is that the framerate was still not that stable in some games, such as GTA V. After all the fixes and tweaks, there were still small stutters, even with vsync enabled. I decided to look over AMD Radeon settings to see if a driver feature had an unintended side effect. I had picked the Gaming preset because that was what I was using this GPU for, but decided to switch to the Standard mode instead. And just like that, the stutters were gone! I suspect that the Radeon Anti-Lag feature might have been the cause to this, as that was one of the main settings that was disabled after switching to the Standard settings preset.

Is it worth it?

I like a technical challenge and going through all the attempts to get more and more performance out of this setup was interesting for me. However, it might not be the same way for everyone else. If you still feel like you want to go through this and learn something along the way, then feel free to use this as a guide on what you can try. Make sure to also read up on experiences that others have had with these tweaks and always measure your results to see if they had the intended effect.

For those who just want to play games and not worry too much about getting the expected performance out of your machine, I’d still recommend going with building a separate gaming PC.

If I had to build a machine specifically for this type of workload, then I’d make these changes:

• Replace the CPU with a non-APU model, such as the Ryzen 9 5950X. Due to the physical core layout you could assign one CPU core complex to the gaming VM and leave everything else to the host. These CPU-s also have a lot of L3 cache, which should help in workloads that require low latency, such as games.
• If you’re going for a streaming setup, then I’d try it first with an NVIDIA GPU. The error code 43 issues are gone now, making it a viable option with solutions like Moonlight.
• More SATA or M.2 slots on the system. This makes passing through storage devices so much easier.

Future plans

I’m not planning on stopping this adventure just yet. I’ve recently looked around for an NVIDIA GPU to do a comparison against the AMD RX 570 and picked out an NVIDIA GTX 1060, which is from the same era and performance bracket. Should make for an interesting comparison in both Parsec vs Parsec and Parsec vs Moonlight scenarios.

I’m also hoping to eventually move to a more modern client PC that can support H.265 decoding and output at higher resolutions. Yes, 4K at 60Hz is demanding for a laptop from 2012. Given that the laptop is an older ThinkPad, I expect that to happen after 2025.

Regarding the storage setup, I’ve got input from a friend saying that I might want to try setting up a Samba share on my “NAS” VM and host my Steam library and other game files on that. After all, the virtual LAN has managed hit 2-3 Gbit/s in my testing and with L2ARC being persistent in ZFS 2.0, I might be able to take advantage of that as well ( assuming that I even need L2ARC, ARC efficiency is pretty good!). Currently I just have a Syncthing sync set up between the “NAS” VM and the gaming VM so that there exists at least a basic backup of all my games.

Setup

After I had a perfectly functional, quiet and performant server up and running, I fell victim to my imagination and completely changed my self-hosting setup. Again.

Here it is, in all its glory:

Well, at least that’s what’s on the table.

This machine is a true all-in-one: it hosts all my services in a VM while also acting as my main workstation.

Specs

CPU: AMD Ryzen 9 3900X. 12 cores and 24 threads in a consumer platform is absolutely bonkers.

RAM: 32GB DDR4 @ 3200MHz

Motherboard: Gigabyte B450 I Aorus PRO WIFI. It’s OK, but oh man are the newer BIOS releases buggy. At least it gives 0 hoots about flashing an older BIOS version.

GPU: AMD RX 560. It gets the job done.

HDD: 2x 12TB WD white-label (WD120EDAZ) drives, shucked from WD My Book external drives.

SSD: 240GB Kingston A1000 NVMe for OS, 2x1TB Samsung 870 EVO SATA SSD for fast solid state storage

PSU: Seasonic Focus something-something. Doesn’t really matter, it’s reliable and has plenty of power.

Cooling: Noctua NH-D9L on the CPU, 200mm fan at the front and 2x80mm beQuiet fans at the back to keep the system cool.

Case: ThermalTake Core V1. Not the smallest case, but super simple to work in!

Why?

Before setting this up, I had three machines running:

• a laptop that was always on so that there won’t be any conflicts with Syncthing.
• a server with server workloads
• a desktop that I mainly used for work

At any point in time, I had at least two machines running and the desktop was fired up for a good part of the day due to work. Since trying to reduce my electricity usage was one of my goals with going low-power on my server, this sounded a bit wasteful. At this point I got the brilliant idea of trying to combine this into one machine that does it all, but before that, I had some concerns. The hardware was simple: just shuck the 12TB drives in there and get some big SSD-s and partition them, but the software part required some thinking and testing.

The software

To keep the work to a minimum, I found that the easiest way to move my setup was to just put the existing server installation into a VM. I’ve had plenty of experience with VM-s so this was not a difficult task, just install libvirt and all the dependencies will be settled automatically.

The problem is the networking. If I want the VM to show up as a separate machine on the network, I need to add a virtual network interface as a macvtap device, which means that the desktop and the VM can both connect to the internet and to other devices on the local network, but the desktop cannot reach the VM directly. There are other solutions, such as the default NAT-based solution, but with that, forwarding ports is quite tricky to someone not familiar with networking.

To overcome this limitation, I came up with a pretty simple solution: add another network interface to the VM! This time, the interface was set up as the NAT-based solution. To make this work, I had to make sure that the Debian 10 install was configured to use that interface as well, which meant a little copy-pasting in /etc/network/interfaces. To make the host connect to this machine using the hostname, I also added an entry to /etc/hosts on the host machine to route all requests to the hostname via this second virtual network interface.

With this, the main obstacle was resolved and the host machine had no issues connecting to the VM.

The storage

Since this machine was also used as a workstation, I had to be a bit clever with my storage setup. The bulk storage in the form of two 12TB hard drives was 100% managed by the VM.

The 1TB SSD-s were partitioned as such:

• 2x 500GB partitions for my workstation running in BTRFS mirror, since I don’t trust ZFS updates on Fedora.
• 2x 250GB partitions for fast SSD-based ZFS mirror in the VM
• 2x 250GB free space for overprovisioning.

In the event that something does go wrong, it should not have too much of a negative impact either on the host or the VM. If the drive quality does become a concern, there is always the option to upgrade to PRO series drives that should have a much better endurance rating.

Pros and cons

With that, the setup was up and running. The VM initially had assigned 6 CPU cores and 8GB of RAM, but lately I’ve decided to assign 12 cores and 16GB of RAM to make the most out of the available resources.

So, how well does it work?

Quite well, in fact. There were some hiccups, though:

• the latest BIOS update for the motherboard was very buggy, which resulted in the ability to permanently get rid of boot devices in the boot device selection menu and the annoying issue of getting stuck in POST when rebooting the machine
• the VM would sometimes have issues with networking, which I attempted to fix by treating the NIC-s within the VM as non-hotpluggable devices. This seems to have done the trick, as I have not had any issues since.

With this setup, there are some tradeoffs that you should be aware of.

Pros:

• Everything runs in one machine with reasonable power usage : ~70 W when idle and monitors off, ~120 W with monitors on and a light workload
• Optimal utilization of resources. Instead of having to upgrade my workstation or server separately, I can just share the resources and modify limits any time I want. Should I upgrade to 64GB of RAM in the future, I can make some more available on the server by changing only one configuration parameter in the VM. The same goes for storage. If I were to upgrade to something bigger and faster, then it will benefit both the workstation and the server workloads.
• Easier troubleshooting. In case things go wrong with the server, I don’t need to grab an external monitor and hook it up to the server, the console is accessible via virt-manager on the host.

Cons:

• All eggs in one basket. If your host OS breaks something, then this will affect your server workloads as well.
• Noise. While not so bad with noise-cancelling headphones, it can still be annoying when you have hard drives running at your desk.
• Your server workload is dependent on your host OS booting up first, which will require manual input from you if your host installation is encrypted.

Conclusion

This setup worked out better than I expected, with no further issues occurring over a couple of weeks. The availability issues during restarts are not a concern for me, as this is a hobby to me, and the flexibility and ridiculous amounts of power this setup has outweighs all the cons.

Let’s see how long this setup lasts.

2021-06-28 update

It still works like a charm!

Anyone that has bought themselves external WD drives from the Elements/My Book/Easystore series are probably familiar with the acoustic characteristics of the drives. The drives have a loud hum caused by WD running the drives at 7200rpm while claiming the drives to be “5400rpm-class” and the clacking of the read-write heads is audible as well. In a small space, such as an apartment, the hum is maddening, especially when you have more than one drive running at the same time.

After running such a setup for months, enough was enough. I bought some sound dampening foam and used that to try to limit the noise that my setup was making. However, that didn’t do much and the before/after noise comparisons didn’t have much of a difference. As a result of this testing, I did have various pieces of acoustic foam left, and after I noticed that my Lenovo M73 Tiny PC had a similar shape to the WD My Book 12 TB hard drives that I got recently, I had an idea.

I used four pieces of foam and placed them so that I could stack the hard drives on top. This resulted in the humming noise being completely eliminated. The sound of hard drive read-write heads is still there, but it is so much less audible now. After this change, I have started hearing the cooling fan more than the hard drives themselves.

This setup could probably be improved by switching around the setup so that the hard drives were at the bottom with the bottom drive having additional foam or rubber feet below it to support it better. For the time being, I’m perfectly happy with this arrangement, as long as it doesn’t fall over.

2021-04-23 update

I have now added a fan to the setup!

The fan itself is powered by one of the USB ports with the help of a spare sacrificial USB cable and a Noctua omni-join kit that I had left over.

The drive temperatures are now reading around 42C and 46C.

The Orange Pi Zero is one hell of an SBC. It has served as a Wi-Fi access point for months without issues and as a testbed for playing around with MySQL.

I wanted it to be a useful device again, but had trouble finding an use case for it due to its hardware limitations. After messing around with my amateur archival work, I had accumulated some data that might be worth keeping around. Due to my network being quite unreliable and slow, I decided to go with torrents as the main distribution method. I could have just hosted the torrents on my main server box, which is what I later did anyway, but then I realized that hosting torrents is a workload that a weak machine could handle just as well.

I took a board that was left over from a shelf, grabbed some twine and nails and banged together this monstrosity:

This brave little machine runs on Armbian and hosts one Transmission instance. This instance is a mirror for some of the data that I collect as part of my archival effort. The OS lives on a 120GB Crucial SSD with the help of the nand-sata-install script that Armbian provides, and the data lives on one 4TB Seagate hard drive that is somehow still working.

The board is pressed against the metal part of an AMD AM4 socket stock cooler with the help of a thick 5mm thermal pad and solid copper wires taken from an electrical cable. This looks very janky, but the cooling performance is absolutely fantastic: 30C when idling and not more than 50-60C under the heaviest workloads.

The total power consumption of this setup is typically between the 5-10W range, which isn’t the lowest, but still acceptable. The CPU is a 32-bit one, so let’s see if this board can make it to 2038.

2021-04-02 update

The 4TB Seagate HDD decided to start a career in experimental music production and started clicking and not showing up when connected to a PC. And thus, the seedbox on a wall has been put on pause for now.

Reddit is a great starting point for getting new ideas for your homelab: racks full of machines in /r/homelab, storage measured in terabytes (or even petabytes) over at /r/datahoarder, all the different services that people host over at /r/selfhosted. This can be a bit overwhelming for someone just starting out in this area, which is why I decided to write a small guide on how to start off on your self-hosting adventure.

Why self-host in the first place?

Hobbyists and professionals have a lot of different reasons for self-hosting services. These include (but are not limited to):

• it’s an excuse to play around with computer hardware and rationalize the purchase of thousands of dollars of hardware
• reduce reliance on “big tech”
• the services solve an actual problem for the user (home automation, secure file storage, backups etc.)
• learning new tech and solutions that otherwise have not come up during their day-to-day work

Regardless of the specifics, the main part of the self-hosting experience is to just have fun and work on challenges that you find interesting.

Step 0: find a problem to solve

One of the best motivators for setting up your homelab/self-hosting setup is to solve a real problem that you have.

Do you want to automate your home with HomeAssistant?

Or do you want your family to be able to back up their valuable photos and videos to your server using Nextcloud?

Or do you want to share your media collection with your family and friends using Jellyfin?

It doesn’t really matter what solution you end up using or how you do it, as long as it is a secure solution and that solves your original problem without creating 10 additional ones.

Step 1: find a place for running your software

Once you have a problem that you are going to solve, you will need to also find a home for your project to live in. For this, you have a lot of different options:

• using a virtual machine at a cloud service provider (DigitalOcean, Linode etc.). Performance is not the best and beefier configurations can be quite expensive, but this is a reliable option with a fast network.
• renting a server from a service provider (example: Hetzner). More performant than a virtual machine at most cloud service providers, but probably more expensive as well. Usually comes with a fast network.
• an old laptop that you have collecting dust on your shelf. Not the fastest or the quietest option, but it does not use a lot of power.
• an old desktop PC. Probably plenty fast for any task that you can throw at it, but its power usage is higher compared to laptops.
• a tiny desktop PC. Low power usage, but can pack quite a punch. For inspiration, check out Project TinyMiniMicro by ServeTheHome.
• a single-board computer, such as a Raspberry Pi or alternatives. Plenty of performance for most workloads, uses very little power (5-15W generally) and silent as well.
• a second-hand rack-mounted server. Probably quite performant, but uses a lot of power and is very noisy.
• a NAS box from one of the more popular providers (Synology, QNAP, TerraMaster etc.). Useful for when you need a lot of storage on your machine or something that just works.

Your choice will likely be affected by other factors as well:

• your home internet connection. If your ISP does not allow opening ports or if your upload/download speeds are not that great, then using a cloud service provider or renting a server might be a better option for you.
• living in a small apartment will likely rule out noisy solutions, such as full-blown rack-mounted servers or powerful desktop PC-s.
• if all you want to do is local testing, then a virtual machine on your already existing desktop/laptop PC might be a better fit, as it does not require you to buy any new/used hardware.
• reliability and price of electricity. In some countries, running a server in your home might result in a huge power bill or might not be possible at all due to the power grid being unreliable.

The workloads that you plan on running will also have an effect on your chosen approach. You will need to take into account the resources that your workload requires. It does not make much sense to get a rack-mount server for hosting your Wordpress blog if a Raspberry Pi can do the same job just as well. On the other hand, running a CPU-heavy service like Jellyfin on a Raspberry Pi will be a painful experience when compared to running it on a desktop PC or an enterprise-grade server. If your service will be used by a high number of users, then you might want to opt for a more powerful machine.

Some solutions may also benefit from faster, SSD-based storage. Loading a lot of smaller files or running a big database off of hard drives will be noticeably slower than running them off of fast SSD-s.

There is no one size fits all solution. It’s better to do a little bit of initial research and use the right tool for the job. If in doubt, ask for help from someone more experienced.

Step 2: find the software solution that fits your use case the best

Just like with hardware, there is also a lot of choice when it comes to software.

Some solutions are pretty low-level and require setting up the OS and the services manually. This is not very user-friendly, but it offers the most potential for learning about your system and how to manage it. This approach will likely require you to learn new things and pay more attention to automating common tasks (updates, monitoring, backups) and making sure that everything is secure (no weak passwords, services properly isolated, regular and frequent updates etc.). If something goes wrong, then you will need to fix it yourself, but at least you have all the tools and knowledge to do so.

Example: Debian with ZFS for your main storage and services running using Docker or Podman (my personal preference).

There are also solutions that cater to people who just want to get work done. These solutions usually provide the user with a web-based user interface that can manage the system and any services that run on it. Do you want to set up your very own Wordpress blog? Just click this button and you will soon be set up with one without having to even know what the hell a “Docker” is. The downside of this approach is that when things go wrong, you will still need to familiarize yourself with the inner workings of the system. Troubleshooting issues with no preparation and learning new things under stressful circumstances is not very fun.

Cloud service providers have also started offering one-click solutions to setting up the more popular services. your very own Minecraft server is just a click away!

Examples:

• a NAS box from Synology that is aimed towards consumers
• VM provided and set up by Linode using a one-click solution in their web UI.

And then there are solutions that fit in somewhere in-between. Solutions like TrueNAS (formerly FreeNAS) and OpenMediaVault are made for users that can probably set things up themselves as well, but are just too lazy to do it. Using these solutions is not in any way worse than the alternatives. If it solves a problem and fits your use case, then go ahead and use it! Just keep in mind that you might need to peek under the hood when there are issues where the GUI cannot help you out.

As always, if in doubt, ask for assistance. It is perfectly fine for you to pick one of the user-friendlier options first and dig deeper once you feel more comfortable. Self-hosting is not a race, you can do everything at your own pace.

Step 3: managing your setup

Once you have everything up and running, there are some topics that you really should pay attention to in order to avoid big problems down the line.

Backups

If the only copy of your files lives on your NAS and something should happen to them (accidental delete, hard drives suddenly die, you get hit by ransomware etc.), then you are in a world of hurt. Having backups is a mandatory part of your self-hosting adventure. A mix of automated and manual backups is better than having no backups at all.

Some ideas for backups:

• external hard drive attached to your server that receives automated backups that run on an schedule. Just make sure that any failures during the backup will be communicated to you (over e-mail, for example). It’s also a good idea to occasionally check if the last backup was successful or not. Downside of this approach is that since this drive is still connected to the server, it is subject to other events that can shred your data (ransomware, power surges etc.).

• offline backups that you will manually run on a regular schedule. Example: copying all files from the server to an external hard drive on the 1st day of each month. If you accidentally delete all files on your server or a piece of malware does this for you, or if your server is toast due to a power surge, you will still have a copy of the data that cannot be affected by those kinds of issues.

• snapshots. While technically not a backup, snapshots do help against accidental deletes. Depending on your setup, you might be able to retrieve files deleted a week ago if you keep snapshots of your data for the last 30 days.

• a hard drive at your friends house. Helps protect against the worst case scenario where your house burns down.

• (encrypted) backups to cloud storage providers, such as Backblaze, OneDrive or Google Drive. It will require a monthly subscription, but at least you will likely have a copy of the file somewhere. Just keep in mind that if you should get locked out of the service or fail to pay for it, then you might not be able to access the data hosted at that service.

Security

If you are exposing a service to the wild wild west that is the internet, then you will be a target. Automated scanners will poke your services and if they find a vulnerability, they will use it. To prevent the worst, consider the following:

• keep everything up to date. Don’t forget that for some updates to apply (example: kernel updates), you need to restart the machine. For other services (example: ssh), they can be restarted without restarting the whole machine. Regular updates will help prevent most security issues.

• only expose services that you absolutely need to expose. With something like a web server, you really don’t have a lot of choice here, but with services aimed towards a known set of users, you can limit the exposed surface area by using a VPN or enforcing other restrictions (whitelisted IP addresses, fail2ban to block most intrusion attempts, etc.).

• make sure that your configuration is correct. A lot of intrusions are caused by configuration issues that usually end up with a service being exposed to the web without any passwords or other protections set up. It’s a good idea to check if you are accidentally exposing too much by checking what others are seeing when they try to connect to your service.

• try to set up your services so that you are prepared for an intrusion. If the attacker is able to take control of service X, but it is completely isolated from other services and devices on the network, then the damage they can do is likely limited.

Monitoring

Your setup will likely have issues. Hard drives might die, or the CPU might be overheating due to years of dust accumulation, or one service might be using up all the available resources, causing others to slow down as well. A better understanding of what your system is doing will help out a lot here.

Cloud hosting service providers usually provide at least a basic level overview of your resource usage: CPU, RAM, disk, network etc. If you are running your own machine, then you will need to figure this out on your own.

To understand what your system is doing, a tool like Netdata is a good starting point. If your server slows down, then you can most likely pinpoint the reason by looking at the resource indicators.

It is not realistic for you to keep an eye on all the indicators all the time, so having some kind of alerting set up is also a great idea. Events, such as a hard drive starting to throw errors, the CPU usage being at 100% for more than an hour, or a service not being reachable, are all something that you might be interested in hearing about.

You don’t have to monitor everything and have alerts set up to catch everything, but it is a great starting point for responding to issues proactively. Ending up with all your data lost because you did not hear about hard drive failures is not a great situation to be in.

What next?

If you have reached this point, then you are probably aware of most of the topics that will come up during your self-hosting adventure. You will encounter issues that will be specific to the approach that you have chosen at the hardware and software steps, but that’s perfectly normal. Most of these issues have solutions that are one search away, and if you have trouble figuring something out, then feel free to ask for help!

If you need new ideas, then check out the subreddits mentioned at the start of this post, you might find some gems in there! Do keep in mind that you will likely end up in an endless loop of building something, being happy with it for a week and finding something else to try out. It’s really fun, though.

My self-hosting adventure started during my university days. Once I actually had money to buy myself a decent used laptop (ThinkPad T430) that had more than 3GB of RAM that my ThinkPad T60 had, I could use some of my older hardware as testbeds for any of the stupid ideas that I had. For someone that liked to learn more about Linux and wanted any excuse to play around with hardware, something like a RAID0 setup over 4 80GB Samsung hard drives on an old PC setup sounded great.

I also had a small collection of data that I had on an 1TB external hard drive, and I wanted to also have backups of the data that lived on my laptop. This was the main motivation for setting up a machine that would serve as a NAS and a testbed for anything that I wanted to try out.

The first box: AMD Athlon 64 X2 4200+ based PC

At this point in time, I didn’t really have any names for my systems. They were simply named by the model of the machine or whatever else I had in mind.

One of the first “server” builds that I built was based on some old parts I had lying around. The CPU was from an old family PC that we got in 2006, and to this date, it is the only surviving component from that build. The motherboard was from a place that collected old hardware and resold some of it and if I remember correctly, it was the abit KN9 Ultra. AM2 socket, DDR2, 6 SATA ports, all that good stuff. This was coupled with 4GB of DDR2 memory. Not much, but it is a start.

The details regarding the hard drive situation are a bit hazy, but at least initially it hosted whatever drives I had around. At some point I managed to buy 8x 500GB hard drives to test out ZFS on a big array. This was mainly inspired by the glorious 32TB server build in an NZXT S340 case that someone showed off in Reddit. I’m still surprised that it was acceptable for me to be in the same room as that monstrosity.

This setup ran well for a while and there weren’t any hardware issues with this setup. Drives were OK, speed was acceptable, and ZFS did its job well, since this was all a big RAID-Z3 array. I even removed two drives temporarily to put in a GPU at one point and ZFS did not complain. After putting the drives back in, the drives were resilvered and all was good.

The engoodening of my server: AMD Ryzen

2017-2018 was wild in the world of CPU-s. AMD came in with some well-designed CPU-s that provided a much better value proposition than the competing Intel CPU-s at the time. The value was insane: double the cores for the same price! It didn’t take much for me to be convinced that the next server build will be an AMD Ryzen based machine, and as soon as I had the resources, I got myself parts for doing that upgrade.

The build wasn’t without hiccups, though. I discovered that cats really like playing around with screws and walking on all the expensive computer parts that took up half of the free space in the apartment. Nevertheless, I managed to complete the build.

Laptop-as-a-server

Due to reasons that have no rational explanations, I started looking into switching out this perfectly working server machine into something more power efficient. The CPU sat idle most of the time and I figured that I could use the machine somewhere else.

And thus, one of the stupidest server builds was born: a ThinkPad X230 serving as a server, with two 4TB WD Red hard drives attached over USB 3.0, running BTRFS RAID1. I do not have pictures of the setup, but this was pretty much it. I learned soon after that cats like climbing on things and that they have no understanding of the amount of force that is required to temporarily disconnect the USB 3 cable from the laptop. This is the part where BTRFS failed on me and I actually lost data in the array. Luckily I had working backups.

Not long after this incident, I started looking for alternatives.

Low power, but still somewhat usable?

My search criteria was quite simple:

• low power (ruled out any desktop builds)
• x86 based (no OS lock-in)
• hosts at least 2 drives over SATA
• reasonably priced

The closest match I could find was the TerraMaster F2-221. It has a low-power Intel CPU, standard UEFI implementation, fits two drives and has two SATA ports. The only part that it really didn’t fit under well was the reasonable cost aspect, but it was still much more affordable than the alternatives.

Around this time I also discovered the world of shucking. I got myself two 8TB WD MyBook drives and used them in this build. However, what I did not take into account was the noise. These 8TB drives were much louder than the 4TB WD Red drives that I used previously, and allegedly this is due to WD labelling these drives as “5400-rpm class”, even though they actually run at 7200 rpm. The TerraMaster NAS unit had no vibration dampeners or other methods to isolate the noise, so I had to stick it in a closet.

The TerraMaster build was otherwise great. I could upgrade the RAM to 10GB using the available DDR3 SODIMM slot on the board, and the services that ran on it were fine. This thing could even run Jellyfin, assuming that you utilized the integrated GPU to provide hardware acceleration for transcoding video.

However, there were some limitations for this setup:

• OS lives on an USB drive. Generally fine, but depending on your setup you could pick between internal USB 2.0 ports or one of the USB 3.0 ports at the back.
• When you have lots of small files and a database on the system as well, the lack of SSD base storage became quite painful. You could do it over the available USB 3.0 ports on the back, but that’s it.

Back to the desktop

Due to a combination of me getting two ITX builds and having workloads that are performance-sensitive (PostgreSQL, Nextcloud, Jellyfin), I decided to go back to where I started from and put the setup back into a desktop case. For that purpose, I picked the Thermaltake Core V1, mainly due to how easy it is to build in it and it having room for 2x8TB hard drives and 2x 2.5" SSD-s.

This box was very fast. Too fast, in fact. It also used quite a lot of power. 70-120W doesn’t sound like much, but when this box is supposed to run for 24 hours a day, this really adds up. I had reached my storage target, the performance was acceptable and everything was working, which meant that the next logical step is to change the setup and hit some limits again.

Quest for low power usage

At this point, I had made it a game for myself to try to go as low as possible with the power usage of my setup while also running my workloads at an acceptable performance level. From my previous meddling, I had a ThinkPad T430 that now boasts 16GB of RAM and a 4-core Intel i7 3820QM quad-core CPU, which is plenty of performance. This laptop also houses up to three SSD-s, which meant that having performant storage around over SATA was also not an issue. The main storage (2x 8TB) was attached over USB 3.0. Not the best solution, but ZFS has handled these kinds of setups very well in my past experience.

Can we go even smaller?

Recently I have learned about tiny form factor machines produced by big OEM-s, such as Dell, HP and Lenovo. These machines are very compact, but can pack quite a punch with a desktop-level CPU and the ability to put in 16+GB of RAM while being powered by a laptop power supply. The only downside of these builds is that the internal storage can be limited to 1-2 drives (SATA or M.2 form factor) and not all USB ports might be USB 3.0.

Once I stumbled upon one offer for a Lenovo M73, I knew that I had to try this kind of setup out. Initially I was thinking of using such a machine for media center purposes, but after I accidentally configured one too many UEFI settings for my ThinkPad X230 in the advanced BIOS settings, I needed a replacement laptop. And thus, the ThinkPad T430 based server was replaced by the M73. I did have to limit the CPU power usage using intel_pstate driver and the USB 3.0 ports needed a bit of bending so that they aren’t loose, but other than that it has been smooth sailing.

This is where the story ends for now. The machine takes about 13-15W when idling, and with 4 8TB external hard drives attached, the typical power usage is between 40-55W. I am planning on replacing the 4 drives with 2 high-capacity drives, something like a WD Elements/MyBook 14TB once those go on sale.

Future plans

This is probably where the setup changes will stop for a while. I occasionally play around with some alternative setups to try to go even lower (Raspberry Pi 4), but so far this setup has proven to be the minimum viable server. It does everything that I need, the power usage is low and it has room for expandability with regards to services that I can run on it.

If I had to start from scratch, then I would probably opt for an AsRock Deskmini X300. It has the same power efficiency benefits, but you could install a more powerful CPU, a lot of RAM and up to 4 SSD-s, two of which can be NVMe drives.

The bulk storage would still be over USB 3.0, but that comes with its own benefits:

• you can not go over a reasonable amount of drives, meaning that your wallet will be happy.
• you do not need to shuck the drives, you can use them as-is and get warranty service without issues.

This has been my self-hosting journey throughout the years. I hope that it was at least somewhat interesting and maybe gave you some ideas to test out on your own.

2025 June update: the contents of my archive are now easily accessible over at nfs-cars.com, a read-only static website that hosts content that was previously on nfscars.net. I’m not affiliated with that website, but I’m very happy that it exists!

Background

My mother once had a laptop: the Compaq Armada 1592DT. It came with Windows ME which later got “upgraded” to Windows 98 SE after I managed to completely screw up the OS, a whopping 96MB of RAM and a hard drive that probably wasn’t much larger than 1-3GB. It wasn’t powerful or anything, but some of my earliest memories of playing video games on that thing were related to Sports Car GT and Need for Speed III Hot Pursuit. Yes, both were demos, and the performance was probably in the 10-20FPS range, but it was still great for a younger me. That laptop eventually got replaced by a Windows 98 based desktop which wasn’t a lot better with its 64MB of RAM, but it could play RuneScape (barely) and also Need for Speed III.

Need for Speed III Hot Pursuit holds a very special place in my heart. It looked great at the time, the police chases were fun and challenging, the soundtrack was fantastic and the track featured in the demo (Rocky Pass) is still something I could probably draw up from memory, assuming that I could draw. At some point I discovered that it is possible to mod the game to include new cars and tracks, which resulted in ridiculous speeds, cars that could absolutely destroy the police cars, and lots and lots of game crashes. One of the sources for these mods was nfscars.net.

nfscars.net

Depending on when you will be reading this, nfscars.net is/was a website that hosts/hosted thousands cars, tracks and other mods for various Need for Speed titles. In 2020, I decided to check this site out again to see if it is still operational.

The last post referred to an “upcoming” NFS title that was already released a year ago.

The link to the forums resulted in an error page being shown.

Should the site suddenly disappear without any warning, all the content would be lost. Since I have a small datahoarding bug, I decided to take matters into my own hands and do my best to archive the contents of the site. That, and nostalgia for the old Need for Speed titles was what pushed me to start this archiving adventure.

Initial efforts

Before trying any fancy technical solutions to crawl the site and get all of the assets that way, I decided to start from the source and tried contacting the site owner to arrange the download of the assets. That did not work out at all, since the contact e-mails were either not active or the inboxes were full.

I signed up for the site and tried to message some user accounts which hopefully were related to someone who was in charge. No dice.

There is a small chatroom on the site named “Shoutbox”, so I tried asking about the site owners there. It seems that a couple of users are still occasionally chatting there, mainly about random topics and spammy users. Unfortunately none of the users there - even the moderators - could not assist with setting up a backup of the site.

With the domain (potentially) expiring on 2021-05-02, I felt like I had to act now, because the site was probably running on borrowed time.

Technical solutions

I checked a selection of solutions for performing a crawl on the site and decided on Heritrix, mainly due to this being backed by the Internet Archive. If they cannot get archiving right, then who can?

Setting up Heritrix turned out to be quite a hassle. There does not seem to be a quick guide that you can follow to get started in 5 minutes. After stumbling around, learning the config file format and understanding that the best way to configure crawls was to do it externally with a text editor, I was set up to launch my first crawl. Even then, I hit some roadblocks:

• The crawl was slow. I understand that Heritrix is just trying to be polite here, but if you have millions of URL-s to crawl and the site might go down at any moment, then time really isn’t on your side.
• Checkpointing is supported, but for some reason the default configuration does not enable it. If the machine that you are running this on should restart, then your crawl will be stopped and you cannot resume at that point, you can only start over.
• Heritrix is based on Java, which means that theoretically it should run on anything that runs Java, such as an ARM CPU based SBC. However, I ran into weird java.nio related issues with symlinking on those platforms and I just could not get them resolved.
• If you run too many crawls at once, then Heritrix is perfectly capable of killing your LAN by using up all the available connections. It managed to rack up 16000+ active connections, which hit the 16384 connection limit on my consumer-grade router. That was not fun to debug.
• Automatically continuing crawls from a checkpoint on system startup requires you to use the Heritrix API, which isn’t too complicated, but I just wish that this was another feature that it supported out of the box.

After messing up a couple of times and trying again, I finally managed to complete a crawl of the site. This only took 6 months, where the crawl itself ran for 3 months. MONTHS!

What the hell is a WARC?

Heritrix collects the results of the crawl into a special format called WARC, which it then compresses, resulting in lots of little .warc.gz files. If you extract one of these, you will end up with a .warc file.

To extract files from a WARC file, you need one of these tools listed here. After trying a couple, I decided to go with warcat since it seemed to mostly work and it supported combining smaller WARC files into bigger ones and extracting the contents.

Depending on the size of the site, the extracting process might take a while. Crawls can contain thousands or even millions of small little files, and if you are using hard drives for storing these, then it will be slow. I have not done testing on an SSD, but it will probably be much faster due to its superior random I/O performance.

Once I had extracted the contents, I did some checking to see if the assets that I care about are present in the crawl. Did a file search for a modded car in Need for Speed III.

No results.

Okay, that’s odd. Tried it with another file name.

No results.

Well, what do we have? I checked the extracted contents using ncdu and found that while Heritrix managed to grab a lot of the webpages and the images from the galleries, it failed to download files, such as .zip and .rar files that contain the modifications themselves.

Getting the final pieces of the puzzle

I started investigating the download functionality of the site. On every page load, the download link would have a new identifier attached to it. Upon clicking on it, the download would start. If you wait too long, then the download link would expire.

Firefox has a handy feature where you can copy the request into a curl command. I did that, added some parameters to it so that it would use the filename that comes from the response header and made it save the file to disk. It worked!

Now I had to figure out how to collect all of these download links. The site has sections for each game, which is also reflected in the URL. However, I found that when you navigate to the page of a mod and change the numerical ID at the end, you could cycle through all the cars, tracks and other tools for all games, even if the URL was technically referring to one game in the series.

After bashing around for a while, I came up with this masterpiece:

#!/bin/bash
cd /your/download/destination || exit 1

for index in {1..18050}; do
  echo "Index at $index, starting."
  filePath=($(curl --silent https://www.nfscars.net/need-for-speed-hot-pursuit/1/files/view/$index/ -H 'Cookie: csrftoken=XXX; sessionid=YYY' | grep -oE "files/download/send/[0-9]+/[0-9a-zA-Z]+"))
  echo "File path: $filePath"
  echo "Making directory $index"
  mkdir -p "$index"
  echo "Going into directory $index"
  cd $index

  for path in "${filePath[@]}"; do
    echo "Starting download on path https://www.nfscars.net/$path"
    curl --silent "https://www.nfscars.net/$path" -H 'User-Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:85.0) Gecko/20100101 Firefox/85.0' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' -H 'Accept-Language: en-US,en;q=0.5' --compressed -H 'Connection: keep-alive' -H 'Referer: https://www.nfscars.net/need-for-speed-hot-pursuit/1/files/view/1/' -H 'Cookie: csrftoken=XXX; sessionid=YYY' -H 'Upgrade-Insecure-Requests: 1' -OJ
    echo "Download $path done."
  done

  cd /your/download/destination 
  echo "Back in home dir. Index $index end."
done

Not the prettiest script around, but it did the trick. In a day or so the assets were downloaded and you could use the index to connect these to a page that is contained in the crawl.

Results

All this effort would be for nothing if I didn’t share my results, so I decided to pack these up and serve them for anyone that wants to keep a backup of at least one part of Need for Speed history.

The archived content can be found on the Internet Archive. Here’s a newer crawl from 2021.

The torrent files are also mirrored here:

• 2020 crawl

• 2021 crawl

Pieces of history get lost all the time with sites going offline. I’m hoping that at least with this effort I have managed to do my part in preserving a tiny piece of it.

2021-03-15 update

The site seems to be down at the moment and has been for a couple of days. Oh no.

2021-03-20 update

Somehow the site is alive again.

2024-08-20 update

It’s dead.

However, I’m delighted to find people using this archive to get their favourite mods!

This post includes a guide on how to match up your favourite mod with the correct filename in my archive. I do wish that I had more time to properly organize that archive while creating it…

2025-06-10 update

I learned that there exists a browsable mirror of the nfscars.net archive! You can find it on nfs-cars.com. I’m very happy that it exists.

This is a list of lessons that I’ve learned while playing around with my computer setup during the last 6-7 years or so.

• USB connected storage is a bad idea when you are using BTRFS.
• USB connected storage is still a bad idea when you are using ZFS, but it is at least much more resilient.
• USB drive enclosures are a total crapshoot when it comes to the quality. Some are good, some will have random issues.
• SMR based hard drives can be unusably slow for certain workloads.
• Desktop PC-s use quite a lot of power when sitting mostly idle (50+ watts).
• You might be OK with Intel QuickSync for your transcoding needs, try it out before opting for a much more powerful hardware configuration.
• Some hard drives are very, very loud. If you live in an apartment, keep this in mind, otherwise you will start trying to rationalise the purchase of 8TB SSD-s to make your setup quieter.
• ZFS snapshotting with a tool like Sanoid is invaluable.
• You will need to use your offline backups, don’t think that you can get away without having some.
• Five nines uptime is perfectly doable in a homelab environment, if you are willing to move the decimal point.
• It’s all fun and games until you have actual people using services on your server.
• Who needs a fancy monitoring setup when you have your friends and family members yelling that the server is down?
• VM-s are great for service isolation, but terrible for flexibility and efficiency.
• Containers, if kept up to date, are an amazing way to manage your numerous services.
• BTRFS is one of the few filesystem options on 32-bit ARM CPU based machines that can still provide data integrity benefits.
• Some workloads, such as databases and services with lots of random I/O, require an SSD to perform well.
• If your workload is very heavily dependent on random I/O, then the “pro” branded SSD-s are actually worth the money.
• SSD power usages can vary wildly, some may require 5V@1A, some up to 5V@1.8A. This will be problematic for hosts that cannot supply much power through the USB ports, such as a Raspberry Pi.
• Armbian is great for SBC-s that support it.
• Try running your database on a slow machine first. Any optimizations you will make will be very noticeable and your database will fly once you move it to a more powerful machine.
• Old laptops are a great way to start your self-hosting adventure, especially if you can limit the fan noise with a tool like thinkfan.
• External hard drives, such as the WD Elements or WD MyBook, are a great way to build multi-terabyte storage setups without breaking the bank.
• Raspberry Pi 4 specs sound good on paper, but its limited I/O capabilities prevent it from doing too many things at once.
• When setting up your system, document every command and installed package for future reference. You never know when it may come in handy.
• Configure your setup for when you get hacked.
• Things will break at the most inconvenient time.
• Some computers can be very temperamental. However, it might be easier to handle the quirks of one system rather than getting a replacement and learning about its quirks the hard way.
• Some workloads, such as Heritrix, might not run well on all platforms, such as arm64 or armhf.
• The original Raspberry Pi 1 B+ is just too slow for most useful workloads in 2021, even when overclocked.
• Having a filesystem that can protect from data corruption is a must-have.
• Only use ZFS SLOG and L2ARC devices if you are absolutely sure that your workloads will benefit from them. If your current ZFS ARC cache hit rate is already over 90+%, then it probably is not worth it.
• Netdata is both beautiful and useful.
• OS support cycles mean nothing and can be changed on a whim.
• If you want a stable ZFS setup, use a distro that has a stable kernel version. Debian and Ubuntu are great for this purpose.
• Update and restart as often as possible. It’s much easier to rule out what broke your setup if the amount of changes that have built up is small.
• Automate the way you deploy configuration changes to your server.

Issue description

You are running a Linux-based machine with an install of ZFS on Linux. Everything seems to work correctly, but after restarting your machine, the ZFS pool is not visible. You can still import your pool manually using zpool import poolname or zpool import -a.

In my situation, this issue does not occur with Debian 10 on an x86-based machine, but it does occur on a Raspberry Pi 4 running Ubuntu Server 20.04.1 LTS.

Setup:

• Raspberry Pi 4 (4GB)
• Ubuntu Server 20.04.1
• 32GB microSD card for root filesystem
• 2x 8TB WD Elements external USB 3.0 drives (ZFS mirror)

Potential causes and solutions

When looking this issue up, there were many potential causes and solutions provided:

• ensure that all ZFS related services are enabled
• add a startup delay in ZFS related configuration files
• just import the pools (duh)
• recreate the ZFS cache file

In my specific scenario, none of these seemed to do the trick.

Investigation

The ZFS pools should be automatically imported on boot by a service named zfs-import-cache.service. The cache file should contain information about the imported pools.

Once you have started your system and don’t see any ZFS pools available, try running systemctl status zfs-import-cache.service or journalctl -u zfs-import-cache.service. This should show you if the automatic import succeeded or not.

The pool tanker is a ZFS mirror that is supported by two 8TB WD Elements external hard drives. They are big, loud and require some time to spin up the platters before you can actually read data off of them. At this point in the investigation, a good friend recommended that I check what disks were recognized by the system at the time when this import service was starting up.

For that, you need to modify the systemd service by running systemctl edit zfs-import-cache.service and put these contents in there:

[Service]
ExecStartPre=/usr/bin/lsblk

This addition will run the lsblk command before the command specified in the systemd service is actually executed. The output of this command will be visible in the service logs. After making this addition, reboot your system and run journalctl -u zfs-import-cache.service again. You should be able to see all the connected drives on the system.

In my case, the system recognized the microSD card and an SSD that was connected over USB, but not the hard drives. This was a very clear indication that the system was not waiting until the drives had spun up before attempting to import the ZFS pool.

Hacky solution

To resolve this issue quickly and move on with more interesting things, you can use the same trick that we used for troubleshooting this issue, but instead of listing the connected hard drives, we can make the systemd service delay its execution by putting a sleep statement in there.

Edit the systemd service again with systemctl edit zfs-import-cache.service and replace the contents with this:

[Service]
ExecStartPre=/usr/bin/sleep 15

This will make the service sleep for 15 seconds before importing the ZFS pool(s). It could be 10, or 30, or whatever value works for you, 15 seconds was chosen here because it is more than enough time for the disks to spin up.

Proper solution (probably)

There are probably better solutions, such as messing around with udev rules or enforcing the delay somewhere else in the ZFS pool import chain of operations, but I have not investigated them at this point. Feel free to contact me with a better solution, and I will update the post accordingly.

One day I stumbled upon a Craft Computing video about his new server build, and on the parts list was a Sedna PCI Express 2x SATA adapter card:

There are different designs out there, including a 4x SATA card, which is bonkers.

Anyway, I thought that I might as well give this a shot, since I have two 4TB 2.5" 15mm Seagate hard drives running, with one of them being in the process of accumulating bad sectors.

The installation procedure itself is simple, just take your drives, screw them in and plug this into a PCIe slot.

So, how well does it work?

Quite well, actually.

The PCIe 2.0 x1 link does mean that your speeds will be capped at around 500MB/s, but with current hard drives you are unlikely to reach this limit. SSD-s, on the other hand, will be bottlenecked in sequential read/write workloads.

lspci -nnk output:

01:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 01)
        Subsystem: ASMedia Technology Inc. Device [1b21:1060]
        Kernel driver in use: ahci
        Kernel modules: ahci

The drives are individually addressable and show up as normal drives (like /dev/sda). S.M.A.R.T info is functional and readable.

This adapter has been running in my system for over half a week now and so far there haven’t been any stability issues whatsoever.

Should I get this?

This adapter is a good fit for the following use cases:

• there are no more HDD/SSD drive mounts in your system left and you do not want to use double-sided tape as one
• you have run out of SATA ports and don’t mind using up a PCIe slot
• you want to install additional SATA drives in your system, but also would prefer to keep the amount of cabling in your system at a minimum
• flexing on your friends with your fancy new 8/10 TB storage card

I would recommend looking into alternative solutions if your use case matches one of these:

• using this adapter with SSD-s on sequential read/write workloads that exceed 500+MB/s
• adding more than two SATA drives into your system (get a simple PCIe -> SATA adapter that only provides the data connectors instead)

LinusTechTips has some pretty great videos about building insane PC and server setups, such as:

• unboxing and deploying petabytes of storage

• 7 gamers, 1 CPU

• building sleeper PC-s

• testing sketchy CPU-s

• opening up a 100TB SSD

I have the ideas, but not the budget of LinusTechTips, so writing them down and hoping that one day I can test this out is the best I can do for now.

The project itself

The main idea is to have a laptop that has ridiculous amounts of storage and the performance to back it up. The idea started gaining traction in my head after I realized that my Lenovo ThinkPad T430 can house up to 3 SSD-s, which makes it a perfect candidate for a system with a ZFS mirror configuration. It also helps that it is powered by a quad-core CPU and 16GB of RAM.

Additional points:

• low power, should stay below 60-90W even under the worst case scenarios
• portable, just plug in the power and an ethernet cable and you are good to go
• built-in UPS in the form of the laptop battery
• well built, can survive a fall or two (already has done so, but that’s a story for another time)
• includes a screen and a keyboard for those times where you really screwed something up and need to fix it fast
• supports 1x mSATA SSD (great for / partition) and 2x SATA HDD/SSD (one in the usual drive bay, the other in the HDD caddy that is replacing the optical drive)

Storage

With SSD prices having dropped quite low and capacities increasing with time, it might not be completely unreasonable to get SSD-s with capacities reaching multiple terabytes. At the time of writing, Samsung, for example, is shipping SATA SSD-s with capacities up to 8TB, which is the same size as the main hard drives in my server right now.

However, these SSD-s are also ridiculously expensive, with the 4 TB model costing ~400 euros and the 8 TB costing almost 800 euros.

Hard drives are generally cheaper per TB, but in this case we don’t actually have much better options. 2.5" 9mm hard drives seem to go up to 2TB at most. You will also have to take into account the fact that hard drives do not like impacts, such as falls, and they are much slower than SSD-s in most workloads.

Cooling

Since I have fitted this T430 with an Intel i7 3820QM CPU with a 45 watt TDP, cooling is an issue. Liquid metal, beefier heatsinks, fresh thermal paste, I have tried it all, but the only solution that has actually worked is disabling the turbo boost feature. There is also this other solution I created that automatically throttles the CPU at a chosen temperature limit using Intel p-state driver, but that solution requires a rewrite.

With poor cooling comes great noise, so ideally this server should run in a place that does not bother anyone with its high-pitched fan noise.

Performance

With a quad core CPU, this system can do quite a lot. Running Nextcloud, PostgreSQL, Jellyfin or any other service should be a breeze, especially since the main storage is SSD based. The amount of RAM is also perfectly suitable for doing quite a lot. Run a couple of VM-s, transcode your media, it can do it all (with reasonable expectations, though).

Portability

One of the best features of this setup is that you can just grab it and go, should the need arise. Its low power requirement makes it a much more suitable candidate for powering it in situations where the electrical grid is down and all you have is a generator or some form of renewable energy. In addition to that, everything you need to work with the machine is already there, just open up the laptop and start working using the built-in keyboard and screen.

The good build quality of the laptop also means that you do not need to worry much about putting it in your backpack and throwing it around. Sure, the plastic might break in some places, and if the impact is strong enough then it can still break, but it is less likely to break when compared to alternative thin and light types of laptops.

Practicality

It’s not all fun and games, though. This setup does have quite a few downsides:

• no way to upgrade CPU and RAM any further. What you have is what you will have to live with.
• it’s expensive simply due to the fact that the SSD-s get very pricy if you require high capacities, hard drives are still more economical for storing huge amounts of data
• the cooling issue will remain unsolveable, unless you invest time and resources into designing a custom solution

Conclusion

Do I still want to build this?

Yes.

Can I build this?

Not unless a delivery truck full of 8TB SSD-s happens to crash in front of me.

Or someone sponsors me. Take a hint, WD/Seagate.

Maybe I am unlucky with computer hardware.

Maybe I have the opposite of the golden touch of Midas.

Or maybe it’s just the fact that I use my hardware for things they were never meant to be used for.

Anyway, I have acquired two Seagate 4TB external hard drives for doing some testing with ZFS. I shucked them (took them out of their enclosure) and started using them as internal drives in many different configurations.

Funnily enough, even though the two external drives had different branding (Backup Plus, Expansion) and enclosure design, the hard drives in them were identical.

Some info about the drives:

• Model: ST4000LM024-2AN17V
• Recording type: SMR (shingled magnetic recording)
• Form factor: 2.5"
• Interface: SATA
• Thickness: 15 mm

Before I get to the stupid stuff, I would like to mention what these drives are good for, and where they suffer heavily.

Good:

• WORM (write once, read many) type of workloads, such as a Steam library, your media collection etc.

Bad:

• random read/write workloads (performance is absolutely rubbish and makes the system unusable)
• write heavy workloads (SMR requires some clean-up activities that are managed by the drive, and you cannot do anything while those take place)

The stupid stuff

I have done some pretty dumb things with these, such as using them in an InWin Chopin computer case and letting them hit 60C under load. Fun fact: these drives actually fit almost perfectly in the case, as long as you bend one small metal section for one of the drive cages. Not so fun fact: they will overheat if you have a high CPU load and you cannot do much about that.

Currently these drives are sitting in my main desktop. Initially I tried using them in a mirrored configuration under Windows, because having your whole Steam library downloaded with disk space to spare is pretty cool. Unfortunately, one of the disks started having issues and it got kicked out of the mirror. It kept doing this every time it accumulated bad sectors.

Now these two sit in a ZFS mirror configuration inside a VM, acting as some sort of a temporary scratch disk. Another free fun fact: kvm (or qemu, or libvirt, one of those) will pause the VM if it detects that one of your disks has encountered I/O errors, and seems that the only “fix” is to force a restart of the VM (could not unpause the VM).

It has become some sort of a fun pastime for me to keep an eye on one of the failing drives and the status of the ZFS mirror.

Will ZFS detect some errors?

How large can the bad sector count get before the drive dies?

When will it actually die?

Why is S.M.A.R.T still reporting that the disk is OK?

So far, so good.

Expect this list to grow with time.