./techtipsy

ThinkPad as a server: third time's the charm

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Wed, 16 Oct 2024 06:00:00 +0300

This is a follow-up to my two previous attempts on this topic:

Since then, I’ve had quite a few changes to my home server setup:

Zimaboard was great, until the CPU became just a little bit too slow for my needs.

ASRock Deskmini X300 was great, until I learned that it’s actually relatively bad at idle power usage due to lack of lower sleep states.

Combined with my recent tech cleanup, I am now proudly running all of my services on a single home server, powered by my trusty ThinkPad T430.

The promise of USB-C everything was too good to pass up, so I stopped resisting.

Doesn’t mean that I won’t run hardware into the ground.

Compared to last two attempts, I’ve made a couple of important changes to the setup.

There is a weekly tlp recalibrate BAT0 job scheduled that completely drains the battery and charges it up again, which should make sure that the laptop battery survives for a longer time. The battery charge threshold is set to 80%.

I’ve used a knock-off Honeywell PTM7950 thermal pad on the CPU. The cooling performance is comparable to using thermal paste, but hopefully it doesn’t drip out.

The whole setup is mounted to an IKEA pegboard using a Nillkin ProDesk laptop stand and a generous amount of zip ties. The combination of zip ties and the rubber feet on the stand result in the stand barely moving once mounted. The laptop stand itself is very sturdy, and you’ll have to use a lot of force to change its angle.

The CPU uses less power compared to the previous one. The one in the server is the Intel i7-3632QM, a quad-core CPU with a 35W TDP.

I still have some scripts running to limit the CPU temperatures to 85°C, just to be on the safe side and avoid overheating at all costs.

Took me 20 minutes to migrate the server along with the drives, and 2 hours to get the zip tie mounting just right.

The small shelf attachment holds two external drives and the power adapter.

Another angle.

Plenty of clearance for adequate cooling.

The power consumption is improved on this setup:

mean: 27 W
min: 23 W
max: 70 W

Results on the DeskMini on a typical day:

mean: 34.8 W
min: 28 W
max: 89 W

All measurements include the power consumption of the UPS, fiber PON and the router, which is around 10 W.

And the performance is good enough.

I’m happy with it.

It’s also handy that I now have that one laptop that I can sticker bomb without feeling guilty about removing them later on.

The Fedora Linux starter pack: everything you need for a smooth experience

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Mon, 14 Oct 2024 06:00:00 +0300

I like Fedora Linux.

It’s the Linux distro that stopped my habit of distro-hopping. Big deal? Ooh, big deal!

It’s stable in day-to-day use, ships with up-to-date software, and the software selection is adequate out of the box.

It also ships with a fresh Linux kernel version¹, resulting in a good experience even on modern hardware.

I’ve been happily using it for about 4 years at this point, both on personal and work computers.

However, every time I recommend it to my friends, family and colleagues, I have to mention a few things that a new Fedora Linux user might want to do first so that the experience is as smooth as possible.

If it weren’t for those aspects, Fedora would be pretty much perfect.

Set up RPM Fusion

There are some packages that are not available on Fedora by default, likely as a result of copyright laws, lawyers and Red Hat (IBM) not having the appetite to fight it out in the courts.

This is why RPM Fusion exists:

RPM Fusion provides software that the Fedora Project or Red Hat doesn’t want to ship.

The installation instructions are here, and I personally prefer this one-liner setup:

sudo dnf install https://mirrors.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm https://mirrors.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm

This step is a prerequisite to the next one.

Enable hardware accelerated video playback

This is the part where I blame lawyers.

To have a good chance at smoothly playing back videos in a browser or using VLC, you have to do a few extra steps, because someone fucked up.

Check this guide for up-to-date information.

Switch to the full version of ffmpeg:

sudo dnf swap ffmpeg-free ffmpeg --allowerasing

AMD GPU users:

sudo dnf swap mesa-va-drivers mesa-va-drivers-freeworld
sudo dnf swap mesa-vdpau-drivers mesa-vdpau-drivers-freeworld

Intel GPU users (Broadwell and newer, 2014+):

sudo dnf install intel-media-driver

Old Intel GPU users (Ivy Bridge and older, <=2013):

sudo dnf install libva-intel-driver

NVIDIA GPU users: good luck.²

You can verify the enabled profiles using vainfo.

Here’s the output on my machine before completing the steps (AMD GPU).

Note the lack of H.264 and HEVC profiles.

me@mymachine:~$ vainfo
Trying display: wayland
libva info: VA-API version 1.21.0
libva info: Trying to open /usr/lib64/dri-nonfree/radeonsi_drv_video.so
libva info: Trying to open /usr/lib64/dri-freeworld/radeonsi_drv_video.so
libva info: Trying to open /usr/lib64/dri/radeonsi_drv_video.so
libva info: Found init function __vaDriverInit_1_21
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.21 (libva 2.21.0)
vainfo: Driver version: Mesa Gallium driver 24.1.7 for AMD Radeon 780M (radeonsi, gfx1103_r1, LLVM 18.1.6, DRM 3.57, 6.10.8-200.fc40.x86_64)
vainfo: Supported profile and entrypoints
      VAProfileJPEGBaseline           :	VAEntrypointVLD
      VAProfileVP9Profile0            :	VAEntrypointVLD
      VAProfileVP9Profile2            :	VAEntrypointVLD
      VAProfileAV1Profile0            :	VAEntrypointVLD
      VAProfileAV1Profile0            :	VAEntrypointEncSlice
      VAProfileNone                   :	VAEntrypointVideoProc

Here’s what it looks like after the tweaks are applied.

me@mymachine:~$ vainfo
Trying display: wayland
libva info: VA-API version 1.21.0
libva info: Trying to open /usr/lib64/dri-nonfree/radeonsi_drv_video.so
libva info: Trying to open /usr/lib64/dri-freeworld/radeonsi_drv_video.so
libva info: Trying to open /usr/lib64/dri/radeonsi_drv_video.so
libva info: Found init function __vaDriverInit_1_21
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.21 (libva 2.21.0)
vainfo: Driver version: Mesa Gallium driver 24.1.7 for AMD Radeon 780M (radeonsi, gfx1103_r1, LLVM 18.1.6, DRM 3.57, 6.10.8-200.fc40.x86_64)
vainfo: Supported profile and entrypoints
      VAProfileH264ConstrainedBaseline:	VAEntrypointVLD
      VAProfileH264ConstrainedBaseline:	VAEntrypointEncSlice
      VAProfileH264Main               :	VAEntrypointVLD
      VAProfileH264Main               :	VAEntrypointEncSlice
      VAProfileH264High               :	VAEntrypointVLD
      VAProfileH264High               :	VAEntrypointEncSlice
      VAProfileHEVCMain               :	VAEntrypointVLD
      VAProfileHEVCMain               :	VAEntrypointEncSlice
      VAProfileHEVCMain10             :	VAEntrypointVLD
      VAProfileHEVCMain10             :	VAEntrypointEncSlice
      VAProfileJPEGBaseline           :	VAEntrypointVLD
      VAProfileVP9Profile0            :	VAEntrypointVLD
      VAProfileVP9Profile2            :	VAEntrypointVLD
      VAProfileAV1Profile0            :	VAEntrypointVLD
      VAProfileAV1Profile0            :	VAEntrypointEncSlice
      VAProfileNone                   :	VAEntrypointVideoProc

Enable Flathub

This step is only necessary if you did not enable third-party repositories during the inital setup of your Fedora Linux installation.

Setting up Flathub is just one command:

flatpak remote-add --if-not-exists flathub https://dl.flathub.org/repo/flathub.flatpakrepo

Check the Flathub setup instructions for Fedora for up-to-date info.

🇪🇪 Specific to Estonia: ID card support

Estonia has an ID Card system (Eesti ID kaart), and the desktop application is also officially supported on Linux. Ubuntu Linux.

To use the ID card in Fedora, you’ll need to install the necessary packages:

sudo dnf install -y open-eid

Firefox users will need to make sure that the Web eID extension is installed. If it wasn’t installed with the above command, you can also get it here.

The desktop application used for opening and digitally signing documents, DigiDoc4, has a bad habit of breaking sometimes due to changes. Luckily there exists an unofficial version of DigiDoc4 as a Flatpak and it’s available on Flathub.

It’s mentioned in the official open-eid/linux-installer repo as well, which lends some credibility (plus I know the person that built it).

As of 2024-10-14, ID card works on Firefox and the DigiDoc4 client from the Fedora official repository does not crash and burn.

GNOME: fix the “Alt+Tab” behaviour

GNOME is a great desktop environment, but it, too, has its quirks.³

If you’re used to “Alt+Tab” keyboard shortcut simply switching windows, then you’ll have to tweak it in settings.

Open “Settings”
Navigate to “Keyboard”
Click on “View and Customize shortcuts”
Use the search bar to:
- disable “Switch applications” shortcut
- set the “Switch windows” shortcut to “Alt+Tab”

Here’s what it looks like on GNOME 46.

Navigate to 'Keyboard' and click on 'View and Customize shortcuts'

Select 'Navigation'

Disable "Switch applications" shortcut.

Set the "Switch windows" shortcut to "Alt+Tab".

GNOME: minimize, maximize buttons on windows

By default, you’ll only see the “Close” button on windows, but it is possible to also show minimize and maximize buttons.

Install GNOME tweaks with sudo dnf install -y gnome-tweaks.

Open it, navigate to Windows, and toggle Maximize and Minimize settings.

GNOME tweaks settings as of GNOME 40.

GNOME: app status indicators

Some apps, such as Solaar, like to run in the background and show you a status indicator on your taskbar. This functionality is not present on GNOME by default, but you can easily fix it with the AppIndicator and KStatusNotifierItem Support extension.

I used to install a handful of extensions in the past to "fix" GNOME, but in 2024 this is the only one I actually need.

GNOME: corner tiling

As of GNOME 46, you cannot do corner tiling out of the box. The best you can do is split windows horizontally, which is fine for single ultra-wide monitor setups. There are situations where corner tiling comes in really handy.

The best solution I have found so far seems to be the Tiling Assistant extension. It’s configurable and will probably suit your workflow well.

At the time of writing, it does have a few small but annoying enough bugs that are a show-stopper for me, but compared to other solutions this one at least works.

Optional: disable SELinux

There are two types of people: those that swear by SELinux, and those that swear due to SELinux.

If you’re running into weird issues and you don’t know why, then see if SELinux is to blame.

I run into issues with SELinux every time I try to move files around with rsync, resulting in a simple rsync -aAXv source/ destination/ command becoming rsync -aAXv --filter='-x security.selinux' source/ destination/. It’s frustrating.

To fix this, I simply disable SELinux. Open up /etc/selinux/config, find the line SELINUX=enforcing and change it to SELINUX=disabled.

You’ll need to restart your machine for it to take effect.

Don’t use ZFS

ZFS is a great filesystem, but not on distros that ship new kernel versions.

This is due to OpenZFS needing to be built against the Linux kernel, and new kernel releases often break compatibility. Those breakages may lead to your ZFS pools not being imported at boot as a result.

You can use ZFS if you use LTS kernels built by someone else, or if you simply don’t update your system, but I don’t recommend either approach.

Use a different distribution, such as Debian, if you really need to use ZFS daily.

If you don’t care for ZFS but like data integrity checks, snapshots and don’t use a lot of disks in an array, then btrfs is totally fine.

Don’t use NVIDIA GPU-s

I’ve never had a good experience with the open source driver nouveau, and the proprietary drivers are a headache and break things.

This might be about to change, but I’ll believe it when I see it.

Conclusion

I hope that this has made your Fedora Linux experience just a little bit better.

Here’s to hoping that IBM does not fuck up Fedora Linux. My expectations are low.

If you have any other tips to share then please contact me! You can find the contact details below.

one thing that I appreciate with Fedora is that the kernel versions are ever-so-slightly lagging behind the likes of Arch Linux, meaning that you’ll be less likely to hit any kernel bugs, and yet you don’t have to worry about being too far behind (like on Ubuntu or Debian). ↩︎
no, seriously, I wish you all the luck in the world. I have never managed to have a good experience with NVIDIA on Linux. I’ve completely given up. ↩︎
use whatever desktop environment floats your boat, I don’t judge! ↩︎

So you want to migrate to Kubernetes: observations from a software developer

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Tue, 01 Oct 2024 06:00:00 +0300

Kubernetes: everyone wants to do it, regardless of their scale and business objectives.¹

Common justifications include better scalability, cost savings, standardization and being super modern and stuff. It’s the future!

In my personal experience, Kubernetes is far from the magical uptime machine that a lot of people think it is, and migrating it to it comes with a lot of hidden costs and potential downtime.

I’m not a Kubernetes expert, but I’ve been involved in a few Kubernetes migration projects and I have opinions. Here are all the learnings and observations that I’ve personally witnessed.

Migrations are complex

For most companies, using Kubernetes will mean planning and executing a migration project.

Assumptions will be made, estimates communicated and then the work begins. 90% of the migration will likely go relatively smoothly, but the last 10% will result in the migration project blowing past any initial estimates that you had.

There will always be those teams and services that require more time due to conflicting priorities or unexpected technical nuances popping up during testing.

It’s the long tail that gets you.

Kubernetes is complex

Kubernetes is so complex that most people point you towards managed Kubernetes clusters provided by the big cloud providers as a starting point.² To me, this is the best indication that we’ve lost the plot.

Kubernetes is an abstraction layer on an already complicated stack, and abstractions tend to leak at the most inconvenient time.

There is nothing wrong about running plain old virtual machines as container hosts and scaling them vertically. Load balancers and containers are a stable and reliable technology by now, and individual servers have made a big leap in performance over the past decade.

You’re going to have to know about the fundamentals of where your service is running either way, so you might as well keep the stack simple, understandable and easily debuggable, avoiding all the extra complexity.

There is no shame in choosing boring technology.

Shout-out to all the madlads who run Kubernetes at home for fun. I respect the hustle.

Kubernetes will only start making sense at scale

If your company doesn’t have a fully staffed platform team (6-8 full-time employees), then you probably don’t need Kubernetes.

If you do, then you can start considering it, but know that it won’t magically solve every issue that you have in your tech organization. Your time might be better spent on tackling those issues first.

Kubernetes is great if you want to standardize how your workloads run, and with additional tooling and setup you can end up with a pretty neat system where developers can set up new services on their own and easily monitor them using your observability stack (Grafana, Prometheus etc.).

This requires a lot of effort though, from both your platform team and developers. This effort will be unreasonably high for small startups and organizations, and my guesstimate is that using Kubernetes will start making sense if you have 100+ developers in your tech organization.

If you’re a small team that has a setup that works for you, then continue using it. You’re doing great!

If you only need a few Kubernetes features, such as autoscaling, health checks and rolling deployments, then you can probably find a simple solution that works on your existing stack.

If you’re just starting up, then don’t use Kubernetes. My recommendation is to start with a stupid simple stack that you know really well and scale it up vertically for as long as possible. Once that setup does not work for you, you will probably have enough money and people to do the Kubernetes migration. It’s a good problem to have!

Let your developers learn Kubernetes before migrating to it

If you skip this part, then expect a lot of questions, blocking issues, missed deadlines, hasty debugging, lost productivity and multiple production outages.

When I first dealt with Kubernetes, I had no idea what I was doing. I barely got to search what was the difference between pods and nodes, how to package applications into containers and what the hell an ingress was³. There was no formal training or opportunities to take a few days to play around with Kubernetes before rolling it out to production. I had to make sure my other work commitments got done in parallel.

It sucked.

I eventually got better at working with Kubernetes, mainly as a result of learning from production outages. This is also training, but much more expensive compared to simply giving developers the opportunity to learn and experiment in a sandbox.

After doing most of “Kubernetes The Much Harder Way”, I have a vague understanding of what I’m doing, but if the Kubernetes cluster were to completely fall over in an unexpected way, I would still have no idea on how to even approach fixing it, or how to make sure that the cluster is properly secured.

One day long Kubernetes workshop organized by the company can go a long way in helping everyone get up to speed. Just organize one before the migration.

Your overworked developers won’t like it

During my 8+ years as a software developer, there was a push for developers to be full stack and to embrace the operational side. I don’t have anything against that, developers should be responsible for what they deploy and observe the behaviour of their services diligently.

However, I’ve also learned that a good chunk of developers don’t want to mess with the full stack and want to focus on their area of responsibility, which may involve more product-focused work. Throw in some inefficient meetings, absurdly high expectations from the business side, and the time and tolerance for handling anything else goes way down.

The cognitive capacity for the average developer is limited. If your developers are already on that limit, and you decide that we need some Kubernetes and developers need to be responsible for their own deployments, then you need to expect some resistance and contempt towards you.

Even before you get to the Kubernetes part, you may also have to make sure that developers know the fundamentals about where their service runs and what amount of resource consumption is appropriate for their service. Turns out that this is not a given, especially in a fast-growth environment where teams and ownerships change often.

Oh, and developers might get very angry with you as every Kubernetes-related frustration will be attributed to your platform team, even if it’s an issue they themselves caused. It’s not fair, but it’s how it may play out.

Your application code has made assumptions about the platform it’s running on

If you don’t have expert knowledge about the service that you’re about to migrate to Kubernetes, then you’ll likely miss any assumptions that have been made in the application code itself.

Most common one is the assumption that there exists only one instance of your service at any time. You can lift-and-shift it to Kubernetes as-is, but then you won’t be taking advantage of any scalability benefits that Kubernetes offers, so what’s the point?

There was also a case where a service was relying on local storage for temporarily storing tasks that had to be picked up later. This made perfect sense on a virtual machine, but on Kubernetes the storage on the pods is ephemeral, and pods have a habit of restarting for all sorts of reasons. This issue went unnoticed for quite a while and only became known after someone familiar with the service asked about it. Adding a persistent volume fixed this issue.

Some libraries and solutions can also make assumptions about the number of instances that your service has, or the internal IP addresses that point to your service being static and predictable. Kubernetes breaks all of those assumptions.

You still need a platform team

I’ve seen claims that using Kubernetes will mean that you’ll need fewer people on your platform team, especially if you use a managed Kubernetes offering.

The reality is that you still need someone to make sure that even a managed Kubernetes instance stays up and running. This involves mundane work, such as making sure that updates are applied correctly without breaking every workload, or making sure that additional tooling bolted on to your Kubernetes cluster doesn’t wreck the services that are running on it.

Before the migration, your platform team answered questions and requests from developers, and wrangled whatever infrastructure you had running.

During the migration, your platform team will be answering questions and requests for both setups while also setting up Kubernetes and related infrastructure-as-code solutions, and unless you brought in more people before the migration, they’ll be overworked.

After the migration, your platform team will still be answering questions and requests, maintaining whatever infrastructure-as-code solutions you put in place, and making sure that Kubernetes stays running, which seems to take about the same number of people as before, if not more.

If you managed to avoid burning out any engineers during the migration, then that’s great!

If you managed to reduce headcount after a Kubernetes migration and it did not bite you in the ass years after the fact, then please do let me know.

Kubernetes won’t fix your legacy monolith

Kubernetes works really well with small services that start up within seconds and use relatively few resources.

The start-up time of your monolith is probably measured in minutes, and it likes to use all the CPU cores and RAM that you give it.

It can still run on Kubernetes, but certain aspects, such as scaling up fast in response to a spike in load, won’t work due to the long start-up time, or due to existing Kubernetes nodes not being able to accommodate your monolith without slowly starting up new nodes. By the time more instances of your service start up, that temporary increase in load might have already passed. Your performance still sucks and your resource usage graphs look like a poorly maintained saw.

Your platform team will also be unhappy with these types of services as these big resource-hungry monoliths tend to require the use of bigger nodes, and they might even end up impacting neighboring pods if configured improperly.

If you have set up tooling to ship service logs from your pods to a centralized location, then you might also find that your high-traffic monolith is logging so much that the tooling can’t keep up, resulting in logs going missing. The root cause can be something as basic as a default configuration value not working out for your thicc monolith, but by the time you get to that discovery, you’ll have wasted a good number of hours or days of productive work time.

Kubernetes won’t magically fix your performance issues

Autoscaling is one of the features that a lot of Kubernetes users like.

You’re having lunch and your service got really popular all of a sudden? No problem, your properly configured HorizontalPodAutoscaler can take care of it!

Autoscaling can save your butt, but it can also introduce additional issues.

For example, deploying a new version of your service can fail because you have too many instances of the service running. Databases, such as PostgreSQL, have a limited number of database connections available. Each instance of your service using up N database connections. If you don’t account for deployments or autoscaling scenarios, then the new instances will fail to start up because they cannot establish new database connections. It’s a good idea to have a few instances' worth of database connections set aside as a buffer.

Unless you’re actually limited by physical constraints, such as CPU time, memory and network bandwidth, then Kubernetes is unlikely to fix any performance issues. You’re better off profiling your application, network and database performance first and making sure that your observability stack gives you enough information to troubleshoot performance issues.

Kubernetes is not a magical uptime machine

It really isn’t.

At some point, you will have downtime because of a Kubernetes configuration issue, taking down your whole service.

Sometimes you’ll involve additional tooling to make working with Kubernetes easier. That can also horrifically backfire due to circumstances not under your control.

You’ll probably have system-wide latency spikes because a critical service got its pods restarted one by one, and the new pods need to warm up their caches again. This is especially true for JVM-based services.

Misconfigured tooling can wreak havoc on your Kubernetes cluster. It’s not fun to troubleshoot why all your pods suddenly disappeared, only to find out later that Karpenter went on a pod massacre.

Vertical scaling can go a long way

If your current stack has a load balancer and a few containers, and you’re not doing anything too inefficient, then you can probably scale up vertically for a very long time.

Servers have made a big leap in performance and capability, resulting in machines with 128+ CPU cores, terabytes of fast memory and lots of room for adding ridiculously fast SSD-based storage.

You can already take advantage of this using your favourite cloud provider by picking a higher-tiered VM. You’ll still be paying the cloud tax, but it’s going to be cheaper than a Kubernetes cluster, and your stack will remain simple, fast and portable.

If you want to go even further, you can buy 2+ physical servers, find a suitable location to host them, and take full advantage of modern hardware. At a certain scale, this will be much cheaper than the cloud, even if you need to hire somebody to manage, maintain and replace them. Physical servers aren’t scary, and you’ll need knowledgeable platform people working for you either way, so why not cut out the complexity and expense of the cloud?⁴

Conclusion

Kubernetes is a perfectly good option to go with, but only at the right level of organizational size and maturity. Unless you’re at that level, you really don’t need to worry about using it.

right after they’re done implementing “AI” and LLM-s on a completely unsuitable use case. ↩︎
the only thing worse than managed Kubernetes is a poorly managed self-hosted one. ↩︎
turns out that it’s a fancy name for a reverse proxy. You know, like nginx. ↩︎
there are benefits to using the cloud, but just like Kubernetes, cloud services have a narrow set of circumstances where their use is appropriate. ↩︎

My blog successfully survived a scheduled power outage

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Sat, 14 Sep 2024 11:00:00 +0300

I had the opportunity to test the resiliency of my home server setup due to a scheduled power outage on 2024-09-13.

It was also Friday the 13th. I’m not superstitious, but I’m a little stitious.

My setup usually consists of the home server, a Wifi AP/router combo box, a converter box for the fiber line, and a CyberPower UT850EG UPS.

The planned power outage was communicated a week in advance and was supposed to take up to two hours. It ended up taking about 1 hour 20 minutes.

The CyberPower UPS is good for temporary loss of power and can probably run my whole networking and home server setup for about 10-15 minutes, which is not enough. Luckily I have one of those Jackery power banks that can provide 230V output for a much longer time. During a trial run, I found that this Jackery box can sustain my homelab for about 4 hours without any issues.

At 11:02, the power went out and UPS did its thing and kept the setup alive. I then turned on the Jackery power bank, connected the UPS to the 230 V power line and everything remained operational. I continued working on my laptop as usual.

My blog and all the other services I run on my home server kept working.

Great success!

I hope to one day do something similar but on a bigger scale, involving solar panels and big batteries. Would be wicked cool to keep a whole building running off of batteries during outages.

I encourage you to write a blog

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Fri, 06 Sep 2024 06:00:00 +0300

It’s been over 4 years since my first post on this blog.

During those 4 years I’ve written over 90 posts, received over 1 million clicks, a dozen legitimate reader e-mails and thousands of spam e-mails.

And I love it!

I’ve found that writing can be very fulfilling and I encourage you to at least give it a try.

This post covers the reasons why I write, how I write and some tips on how you can get started writing one yourself.

Blogging is so ~~2004~~ 2024.

Why I write

I’m a software developer by trade. However, some might find it surprising that I actually write very little code in my free time.

My GitHub activity.

Instead, I play around with computer hardware, self-host some services on my machines, and make stupid experiments that often result in interesting findings.

Most of that work had been invisible, until I started writing about them on my blog.

I write because other people find my posts useful. I’ve learned that some of my posts have been really useful to people who’ve faced similar problems in the past or who want to learn more about a relatively obscure piece of hardware. Tools like Marginalia search, Google Search Console and goaccess have been great sources for finding where my content has been referenced.

I’ve found my content linked in all sorts of places:

Gentoo wiki entry for the Lenovo ThinkPad T430
Libreboot installation instructions
a YouTube video with almost a million views
a YouTube video with ~1400 views
- it’s properly sourced, I love it!
other small blogs
link aggregators
various forums, including the ThinkPad one and QubesOS
Reddit, even the really obscure stuff
even Hackaday!
- also got a mention from the OpenPrinting project leader!
- same post was also featured on XDA Developers and hackster.io
there was even a case where my post was translated into German and published on golem.de, with my written permission.

Seeing my work out there helping people gives me an immense sense of accomplishment and motivates me to write about the challenges I face. It really does mean a lot to me. It’s not quite on the level of Jeff Geerling and the work that he has published, but perhaps I’ll get there one day.

People on Reddit finding my archival efforts to be useful. I love to see this!

I also often end up looking up my own posts because of some technical details that I’ve written down, or to share a post in a discussion because of its relevance. The one about the minimum viable fan control script has been very handy, for example.

This post has been performing well consistently for years!

Quite often I end up finding solutions in other indie blogs myself. There’s something special and authentic about those, and they often go more in-depth than places like StackOverflow. Ads and inappropriate sponsorships are also rare on those.

I write because I forget things. My blog serves as a public diary of sorts, and it has been extremely helpful in reminding me about all the fun things I’ve worked on, or experiences I’ve had. I sometimes revisit my older posts and I still find it surprising how much I’ve managed to do, or what I’ve worked on.

I write because I can do what I want.

No Synergy Greg.

No managers.

No meetings.

No quarterly plannings.

No unpleasant colleagues.

No conflicts.

It’s really nice.

Rules of engagement

Whenever I write something, I try to follow a set of principles. These have been influenced by my own ideological views and content creators that I look up to.

Write content that I would like to read myself. I love stumbling upon well-written posts covering topics that the author is passionate about. These posts have a special energy in them, which is very infectious and motivates me to put more effort into my own writing.

If you want to know what I like to read, then check out the list of good posts here, and anything else that’s cool goes here and here.

Straight to the point. I try to avoid unnecessary word salad whenever possible and try to get my point across in a clear and concise way.

Make the content accessible. The person reading my blog might be a seasoned developer, or someone just starting out in IT. It’s a balancing act and I’m unlikely to ever get it just right, but I try to remind myself of what I knew back when I was just starting out and try to address that person in my writing.

Listen to the audience. If the readers demand to see the cute cats you mentioned in a post, then you better deliver.

Keep performance in mind. I do my best to keep the page size down to a reasonable size. This results in my own bandwidth requirements being lower, allowing me to serve more traffic over the same connection. It is also helpful to readers who are on a slower network connection or are running slower hardware.

The Performance Inequality Gap series has inspired me a lot in this area. I’m not going to do every micro-optimization that tools like Lighthouse complain about, but I will do what I consider reasonable.

No ads. A couple of dollars a month is not worth compromising the privacy of my readers. Relevant and up-front collaborations with manufacturers in the style of Level1Techs and Jeff Geerling are still OK in my view.

No shady sponsors. Most sponsorship offers that I get involve gambling, and gambling but with extra steps (crypto-“currencies”). Anyone working in an area that exploits human psychology to make obscene amounts of money (at the cost of ruining actual human lives) should really reconsider their life choices.

No so-called “guest posts” or backlinks. I have received hundreds of these SEO spam e-mails at this point, and the pattern is the same:

hey ounapuu.ee, we really love your site about $NOT_WHAT_I_ACTUALLY_WRITE_ABOUT

are you ok with linking to $SHADY_SITE or a guest post? what’s your asking price?

Latifi Hamilton, SEO Spam Corp. Ltd

I am 100% convinced that SEO optimization in this form is an absolutely useless “job” and a net negative to the world. Please stop.

No generative AI garbage. It conflicts with the reasons why I write a blog in the first place. I’ve tried things like GPT-3 to see what it regurgitates, and it has been the most unoriginal, fluffy drivel that gets all the important details wrong.

I’ve also seen the same-looking generative AI imagery in lots of blog posts and presentations, it’s really off-putting and shows a lack of care and effort by the author, even if the rest of the content is solid.

Getting started

So you’ve finally decided to start your own blog. Great! Here are the steps that I believe make sense to someone just starting up.

Why are you writing?

This is the most important step. Don’t skip this.

A lot of techy people (myself included) get carried away with the technical details before writing the first post. It’s understandable, the technology aspect can be very exciting on its own. However, focusing on the blog setup itself distracts from the core of your blog: the content.

Start by figuring out the reasons why you want to write in the first place.

It could be anything:

you have a fun hobby that deserves more coverage
you find that there are products in your niche that don’t have any proper reviews
you want to share some stories and lessons learned from your professional career
you want to create content on a platform that you have full control over
you just want to share your work with the world

Now that you’ve got that figured out, I recommend writing up your first draft, even before you know where you’re going to host it. Any plain-text file will be fine for this purpose, and you’ll most definitely rewrite most of it and touch it up once you know where you’re going to host your blog anyway.

Find a place to host your blog

You have your goals in place and you have the draft for your first post. Great!

There are a lot of options out there for hosting your blog, each with their own strengths and weaknesses. You should pick the option that fits your needs the best.

Indie blogging platforms

Examples: Mataroa, Bear Blog

These come in all sorts of flavours and offer different functionalities for free or a small nominal fee. Usually run by one or more people as a passion project or side-gig. Easy to start out with, and the good ones offer easy ways to take your posts with you if you ever decide to move to another platform.

If I didn’t already host my blog on my own server, then I’d probably use one of these types of services.

Fun fact: Bear Blog is run by another Herman!

Big blogging platforms

Examples: Medium, Substack.

Quite popular and likely have good tooling for writing content easily, but you usually have less control over your work and you are at the whims of the platform. If they ever decide to completely change their business model and start charging you much higher prices for hosting your work there, then you better hope that they have an easy way to export your content.

Dynamic site generators

Examples: WordPress, Drupal, Joomla

You can host these on your own server, or pay someone else to do the hosting for you. Can be easily customized and should be quite beginner-friendly for content creation. Pages are created on the fly, so these are often not the most performant option. Can easily fall over under high traffic if the setup is not optimized and your post gets linked on a popular platform (“hug of death”).

If you ever have to pick one of these types of services, then you must make sure to keep it up to date. WordPress vulnerabilities can be exploited within hours after a vulnerability disclosure.

The performance and security considerations are the main reason why I avoid these options.

Static site generators

Examples: Hugo, Jekyll and many, many more

With static site generators, you build the site once and copy the contents to your web server. This comes with great performance benefits as serving static files on a web server is really darn fast. Even the cheapest virtual private server at any cloud provider can likely handle more than a gigabit of SSL-encrypted traffic with this option.¹

Security is also less of an issue. As long as your web server is up to date, then you don’t have much to worry about, just make sure to not accidentally push any sensitive files and information along with your blog.

These benefits come with a few trade-offs. The content is written in Markdown and while you can embed code examples and images, it’s not as user-friendly as other platforms. Adding dynamic content and comment sections is not included by default.

This is the option I’ve chosen. The site is built with Hugo, the web server part is handled by Fedora Server, Docker and this container.

It’s also possible to host your static site on GitHub, if you don’t want to deal with the hassle of having a web server around. The downside of this approach is that GitHub is quite often down, and their downtime is now your downtime.

Just do it.

Go ahead, write that first post! Here’s mine for reference, not great, not terrible.

Don’t worry, even I started a blog and abandoned it on my first attempt.

Sometimes I start writing when I don’t feel like it. If there’s a 20 minute time window, I can write the skeleton for a new post, or finalize a work-in-progress one. Once I get past the hurdle of starting, I’m in the zone. Some of my best posts have been written under these conditions.

Daniel Stenberg had a talk at FOSDEM 2024 that incidentally covered similar time management techniques that help him get work done on curl, I highly recommend giving it a listen.

Build a habit

I have a goal to write at least two posts per month and I’ve mostly stuck to it, with a few exceptions. Internal deadlines help me actually get my thoughts and work out there in a somewhat timely manner.

Having this cadence also helps in situation where I expect a busy schedule, as I can try to finish a few posts in advance.

A post can take anywhere from 1 to 4 hours in my experience. Some can take a few days, such as my FOSDEM 2024 post.

One post per month can be a good goal for a first time writer.

Tooling

You don’t need anything special for writing. Use whatever works for you. If it supports spell-check, then that’s even better.

I write my posts in Markdown and the best tool I’ve found for that happens to be IntelliJ IDEA. I already use it for development work and its Markdown support is good. That’s it.

I’ve used tools like MarkText but it went really poorly the last time. Other text editors haven’t worked for me for various reasons.

Optional: buy a domain name

You may want to get a domain name for your blog. Depending on your setup, it can make transferring your blog to a different provider or service much easier, simply point your domain to a new IP address!

This does add a yearly cost that might not be acceptable for everyone.

Optional: use a completely separate contact e-mail

If you add a contact e-mail to your blog, then you will get spam, guaranteed.

I use a completely different e-mail because of this, it helps me easily see where the spammers got my e-mail. I also find it funny that every time a spammer e-mails me, they’re blatantly lying without even trying. You see, my e-mail is ihavesomethoughtsonyourblog@ounapuu.ee and it’s rarely the case that they actually have thoughts on my blog.

If you have a domain name and a decent e-mail service provider, then adding additional e-mails should be quite easy.

Optional: self-host at home

A small blog is a great first-time project for someone that wants to get into self-hosting and building a homelab.

All you need are ways to open port 80 and 443 for HTTP and HTTPS.

A residential connection will likely have a dynamic IP address. It might be possible to pay extra to have a static IP address. It’s also possible to sign up with a dynamic DNS provider that handle IP address changes for you.

I’ve gone one step further and written a DNS updater Python script against my domain registrar. If you can read this text, then it means that the script works!

How I write

I’m not going to tell you how to write because I can barely manage to do it myself, and there are smarter people out there from whom you can learn that.

What I can do is to share my writing process in the hopes that there are a few tech tips that you find useful.

Writing

I have a list of ideas that I have written down. When I get the opportunity or inspiration to write, I look at the list and pick something from it.

Every post starts with a draft title and a bulleted list of points to cover in a post. That list is like a table of contents for the post, and then I start expanding on them in writing.

Pictures are usually added after the first draft is done.

I believe in POSSE: Publish (on your) Own Site, Syndicate Elsewhere. I have full control over my blog and can link to it on all sorts of platforms. Screw “big tech”!

Sharing my work has been tricky, however. I can always share it in platforms like LinkedIn, but links like that get down ranked compared to “native” posts on the platform itself.

Link aggregators, such as Hacker News and Reddit, discourage or forbid sharing your own content. I have done it, but only with content that I think is relevant to the communities. There’s a reason these rules are in place, simply check the newest posts on Hacker News to understand why.

I don’t recommend this type of behaviour, but if you have a good understanding of the community and what content they like to see, then it can be a good way to get some eyes on your content.

Once you get a following, you don’t have to worry much about sharing your work as your readers will probably do it for you.

How readers keep up with my blog

I’ve seen two ways that readers follow blogs: mailing lists and RSS feeds.

Readers can follow my blog using the RSS feed. For those that are unfamiliar with RSS, I link to aboutfeeds.com at the bottom of my posts.

It’s harder to track the number of subscribers with an RSS feed, but I believe it’s the approach that best preserves reader privacy.

Some feed reader services, such as Feedly, Inoreader and Feedbin, have a neat little feature where they actually send the number of subscribers in the user agent when making a request towards your RSS feed, and that user agent is visible in your web server logs:

127.0.0.1 - - [21/Aug/2024:10:40:19 +0300] "GET /index.xml HTTP/1.1" 304 0 "-" "Feedly/1.0 (+http://www.feedly.com/fetcher.html; 72 subscribers; )"

Mailing lists are also quite common for blogs, but I never found a good mailing list provider that costs a reasonable amount of money, so I never started one. I’m also not going to manually maintain one or build a solution myself.

Analytics

My main analytics tool is goaccess, a handy terminal UI tool that reads nginx web server logs and outputs basic statistics. This is just enough information for getting a basic understanding about how well my posts are doing.

goaccess can also track incoming requests live, which is very cool to observe when your post has gained traction on Hacker News.

Small glimpse of what goaccess can show.

I also check in on Google Search Console from time to time. It’s been quite handy for understanding how well my posts are doing in Google Search.

Example of Google Search Console results.

I can also get an understanding how my content is doing elsewhere. I don’t have Google Analytics on my page, but I can still see when my posts have gained traction on Hacker News or elsewhere.

I guess they track popular HN posts.

Observations and tech tips

My years of running a blog have resulted in some potentially interesting tidbits and tips that I can share.

Hacker News

Hacker News likes cats, cool hardware, tech museums and FOSDEM.

And they sure love old ThinkPads, including those that act like a server. Any old ThinkPad, really.

I’m happy with it because the discussions that spawn from these topics are great to read and I learn new things almost every time. Sometimes people post comments without reading the post and that kind of pisses me off, but overall it’s a nice community that has done a decent job moderating itself.

What I also like about Hacker News is that they also have a second chance pool. If your submission was great but didn’t get much traction initially, then it might end up being given a second chance to get to the front page. Two of my submissions have gone through this process, it’s a really nice feeling to be acknowledged like that.

People like hardware

I’ve noticed that no matter what hardware I talk about, there is someone out there interested in it.

For example, most clicks coming in from Google are related to products like Fairphone 5 and Zimaboard.

Screenshot taken in Firefox Private mode. It's beating The Verge!

Niche products, such as ExpressCard to NVMe adapters, are also bringing in a solid number of clicks.

Screenshot taken in Firefox Private mode.

I wouldn’t want to be a full-time reviewer like how most tech YouTubers operate, but it could be fun to get access to cool hardware that I could then ~~do incredibly dumb things with~~ test.

Feedback

You’ll probably get feedback.

Most of it will be good, constructive and informative.

Some of it will be from people who are complete tools.

The most important step is to figure out which bucket the feedback falls into.

You don’t have to address any of the feedback. You don’t owe anyone anything. It’s your blog.

If you’re brave, you can consider showcasing uninformed takes on your blog, in the style of Ludicity.

Follow your own blog

I use Miniflux as my feed reader because it’s simply the best one out there.

Counterintuitively, I also follow my own blog there as it helps me catch any issues with my RSS feed.

I’ve caught a few issues, such as my drafts being accidentally published, or new pages on my site being present in the RSS feed.

It’s also just a good way to see if the feed is OK.

Other blogs that I like

I follow a lot of blogs, but the ones that have had the biggest influence on my own blog are the following:

Jeff Geerling
- he covers a lot of cool single board computers and actually uses Ansible correctly!
- he also has an accompanying YouTube channel
Low-tech Magazine
- I love the concept of a solar-powered website and hope to do something similar in the future
- this post is what inspired me to write my most successful post to date
Louwrentius
- lovely little blog that covers all sorts of homelab-ish topics
- this one is also solar-powered!
Ludicity
- the aggressive tone is refreshing and gives me courage to say what I actually think
- the blogging platform itself is also very opinionated and minimal, which I love to see!

Discovering new blogs

Sometimes I am bored. Happens rarely, but when it does, I like to look around to see what other blogs are out there.

I’ve used the following tools to find new blogs:

It requires intent and time to go through these, but that’s also part of the fun. Similar to a dedicated music discovery and active listening session, if you think about it.

Closing thoughts

I find writing to be incredibly fulfilling, and I hope that at least some of you end up writing about the topics that you are passionate about.

If you feel that there’s a topic that I didn’t cover, or you have additional questions related to writing a blog, then don’t be afraid to reach out to me!

the cheapest ARM-based VM at Hetzner Cloud could easily serve 1.6 Gbit/s of HTTPS traffic, benchmarked using wrk against my own blog. ↩︎

How to copy media off of an iPhone the hard way (using Linux)

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Mon, 02 Sep 2024 06:00:00 +0300

I helped a family member upgrade to a newer iPhone and make some room so that the internal storage does not run out.

They had Nextcloud installed on the current phone, but due to limitations of the Nextcloud iOS app, the backups only take place if the app itself is open, meaning that we had hundreds of photos and videos that were not yet backed up.¹

The network was slow and I was in a time crunch, so I opted to copy the media off of the phone by connecting the iPhone to my laptop over a Lightning cable. After entering the PIN on the phone again and approving the connection, I could see the iPhone in my file manager, but opening it would show nothing at all. This was very unusual as other iPhones I used in the past just worked on my laptop.

This is what you'd expect to see when connecting an iPhone to a PC.

Rebooting the phone and connecting it again did nothing. Different known good Lightning cables did not improve things, either. Even a separate Windows 10 machine could not see any images, so it wasn’t a Linux thing.

Various Apple-related posts online were devoid of useful advice.²

I was not ready to give up just yet and I finally stumbled upon this Ask Ubuntu answer. This had potential, so I gave it a go, and I’m happy to report that it worked on my Fedora Linux machine!

These are the steps that I took. The commands need to be run in a terminal window.

Tested with Fedora Linux 40.

install the necessary packages
- on Fedora, run sudo dnf install -y ifuse libimobiledevice libimobiledevice-utils
connect your iPhone to your PC
- if your phone is not yet paired, then run idevicepair pair
- if it’s already paired, verify the connection by running idevicepair validate
mount your phone to an empty folder of your choice
- if an empty folder does not exist, run mkdir ~/my-iphone (use any folder name that you want)
- to mount the phone, run ifuse ~/my-iphone
you can now browse your media via your file manager or terminal by navigating to ~/my-iphone
- photos and videos taken by the camera will be in the DCIM folder
- other folders, such as Photos and Downloads may also interest you
- to make a full copy of your phone to a folder named iphone-backup, run cp -r ~/my-iphone ~/iphone-backup
- alternative way to create a full backup: rsync -aAXv ~/my-iphone ~/iphone-backup
once you’re done, unmount the iPhone by running umount ~/my-iphone
it is now safe to disconnect your iPhone from the PC

The directory listing of an iPhone.

Photos are located under DCIM folder, nested into a bunch of subfolders.

Ah yes, USB 2.0 speeds.

If you have copied everything over and want to make more space on your phone using this method, then you need to delete files under the DCIM folder and the cached thumbnails under the PhotoData folder. iOS seems to generate thumbnails for the photos you take and even if you delete all the actual photos off of the device, then the photo thumbnails will still show up in the Photos app and elsewhere. This confused the hell out of me after I deleted the backed up photos off the phone the first time.

Thumbnails are in a separate folder.

Alternatively, you can just delete files from the Photos app on the phone itself to avoid this nuance.

Linux has its flaws, but at least it provides you with the power and tools needed to get you out of tricky situations.

I can’t wait for Immich to get a stable release… ↩︎
Apple and Microsoft related forums are the worst for getting an actual solution to your problem, which is in stark contrast to FOSS/Linux related discussions. ↩︎

How I ended up working as a software developer

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Fri, 16 Aug 2024 06:00:00 +0300

I’ve officially worked as a software developer since August 2016, and by now I have a fair share of stories to tell from those years. But those are stories for another time.

Today I’d like to focus on where it all got started.

The early days

I never considered myself good with computers, or a nerd, or anything like that during my childhood. All my computing experiences can be summed up in a pretty short list, and most of the memories are around computer games.

I was simply too busy petting the kitty.

Starting off, there was that one Windows 95 box with no internet connection at home. Me and my younger brother defaced the startup screen in Paint because it was just one of the images on the drive, and there wasn’t anything else interesting for us to do there.

Then there was that one laptop the family had temporarily. It ran Windows XP, which felt really modern because the user interface had actual colors.¹ I remember being fascinated by one game on it: Siia sinna läbi linna. It was a simple educational game where you had to follow traffic rules as a pedestrian and walk around. I had the most fun with it when I tried to illegally cross the road and barely miss the cars.

My aunt had a desktop PC that I rarely could play with, but it had some great games, like The Need for Speed (the very first one!) and a PC port of Sonic the Hedgehog 3.

Later on there was a Compaq Armada 1592DT. It ran Windows Millenium Edition, which is commonly regarded as the worst Windows release ever. It was not great, I can tell you that.

It was around this time that I also got into gaming more. I played through the demos of Need for Speed III Hot Pursuit and Sports Car GT² hundreds of times. Free demos was all I got, and not many games even ran on the laptop. The two games in question also ran slowly, but I still loved them.

Then there was a Windows 98 desktop PC that I got around 2004-2005. This time in my life was characterized by family drama, so it was great to have a place to escape. My fondest memories include finally playing the full version of Need for Speed III Hot Pursuit, and hours and hours of RuneScape.³ I still remember one random event occurring in RuneScape where a tree pops up and you have to interact with them or try to kill them, however my computer froze at the time and no amount of hitting it with my foot helped. Lost my full mithril armor that day. The computer eventually gave up with what I assume was a hard drive failure.

Then we got an actual new computer for the first time in my life. It was 2006 and we got a Fujitsu tower PC with an AMD Athlon 64 X2 4200+ and an Nvidia GeForce 7300. The GPU was passively cooled and died soon, and the warranty service put in a Nvidia GeForce 6500 instead. That ran much better. This was the era of more unpleasant life events, working summers as a newspaper seller⁴, and playing a lot of GTA San Andreas and Need for Speed World. I could spend 9 hours playing every day while still keeping my grades up at school.

I had the time of my life playing games, and it’s probably what saved me from making stupid, irreversible decisions.

School, computers and me

I went to first grade in 2002.

The first time I could use a computer in a classroom was in 2010.

There was one classroom full of Windows XP boxes, and with something like 256 MB of RAM in them. The UI was very gray, likely as a result of switching to a “classic” theme to save some resources.

I remember two items from the curriculum:

creating a Word document
creating some pixel art in Paint

We had to remember which computer we used and hide our files somewhere in the folder structure if we didn’t want to lose it between classes. USB sticks were very expensive and not that popular as well.

Our teacher was what you’d think of when you thought of the most stereotypical sysadmin: probably good at their day job, but maybe not the best teacher. At least they did show us the insides of a PC, and I remember how they were raving about their IBM ThinkPad T20-series laptop and how the new ones were trash. I guess some things never change.

There was also a short robotics course where we could do things with LEGO Mindstorms robots, but we never quite understood what we were doing.

Not all schools in Estonia are made equal, and I experienced it first-hand. My next school was a completely different experience, as they had two computer classes, and the machines they were replacing were still years ahead of what we had at the previous school.

I’m very happy to see that we have companies like GreenDice who are motivated by similar experiences and want to make sure that everyone has access to computers. Most media consumption happens on phones, but the real work still gets done on PC-s.

The part where I started programming

When I went to 10th grade, I did so at a new school. It is considered one of the “elite” ones in Estonia, and I got there by pure accident.⁵

Ah, 2011. The iPhone was new, smartphones were evolving fast and I definitely did not know how to talk to girls.

Around 2012, our class teacher sent out a notice saying that Tartu University was offering a free extracurricular course named “Teeme ise arvutimänge” (roughly translates to “Let’s build computer games”). It was fully online with no scheduled mandatory hours and I liked games, so I signed up.

The course was about 7 weeks long. Every week you’d focus on one area, starting with the basics of Python 3, building up your knowledge with more complex parts of the language and creating a text-based game. At the end of the course you had the choice of building a text-based game or a 2D game with Pygame.

I had discovered retro gaming around this time, so I went ahead and recreated the final boss level of Sonic the Hedgehog 3. I “borrowed” sprites and the official soundtrack from various places online, and at the end I had something that didn’t run very well, but it ran.

And it still runs on my Fedora Linux 40 laptop!

I got a bit excited replaying this.

"original content do not steal"

After the final project was completed, everyone who participated shared their games with the group. I liked seeing a few games there where it was obvious that the author had put actual effort in and loved working on it.

I passed, in spite of the obscene number of copyright violations that I had committed.

It's clear that git was not part of the course.

Before I went to university, I also attended a one-off extracurricular programming class offered by the school. During that time, I showed my game to a classmate there and they were absolutely horrified at the code. For good reason. Whatever I did there, it was horribly inefficient. At least my modern CPU can now chomp through all that inefficiency.

With that feedback in mind, I rewrote the game from scratch, made fewer stupid mistakes and added new features as well. It was still around the same concept of the Sonic the Hedgehog 3 final boss level, but the obstacles and enemies were more varied and the game ran at 60 FPS even on an old laptop.

What’s funny is that the new version of the game doesn’t run. I had to manually set the resolution of the game, and even after that the game crashes randomly after 5-10 seconds.

Definitely not future-proof.

But hey, it looks so much better!

I even commissioned a drawing for the title screen. Big budget stuff!

Ain't nobody got time for replaying the whole game to debug the final boss!

The sprites are animated now!

Variety in bosses!

A bit harder to hit now.

The orange ones follow a sine wave pattern. Very advanced AI!

This Pygame adventure also spawned the only two StackOverflow questions that I have ever asked: the one where I ran into limitations of the library, and another one where I couldn’t understand why my background was funny.

I also ended up attending an event⁶ where I could show my game to others, and the article about it is still up! You might need to use your favourite translation service to understand it though.

This one course is the sole reason I chose computer science in university and ended up as a software developer. This sequence of events is purely accidental, and yet it sparked this fire in me that thrives on building new things and troubleshooting issues. I loved the immediate visual feedback that I got when building the game and had a lot of fun trying to figure out how to make the computer do what I want.

A lot of what made me love programming was also what I enjoyed during my first actual job as a software developer. I started out as a front-end developer, working with Angular 2 right when it got the first official stable release. It wasn’t easy to start with something like that as a junior developer, but I loved the immediate visual feedback and learning how to use the browser tooling to troubleshoot issues.

For a few years I also considered pursuing a career in game development. I love playing games, I love programming, so it would have made perfect sense, right?

Unfortunately the only things I kept hearing about game development were negative ones, involving poor working conditions, “crunch time”, and how most game developers end up with mental illnesses and severe burnout, all because some people in suits want to make even more money.

Things I wish I knew

The lead-up to my first actual job as a software developer included a lot of unknowns, anxiety and comparisons to more successful students, which is why I’d like to share some tech tips for those just starting out in this field or IT in general.

It’s OK to try out this role and end up deciding that it’s not for you. I know quite a few people that started out as software developers, but ended up transitioning into a different role that suited their interests better, such as team lead, product manager or data engineering. Even I had a two-year gig as a team lead! Change can be scary, but it might end up being the right thing to do.

Don’t feel pressured to do anything. Some YouTuber just posted a video that you have to learn this new framework or programming language or you will never get a job? Someone on Twitter keeps insisting that the blockchain is the future and everything else is now obsolete? That is pure grade-A clickbaity bullcrap that plays on the fear of missing out. Don’t fall for it.

There’s also a subset of developers who have an expectation that you also write code in your free time and regularly contribute to open source projects. Unless you want to work at Google, Meta or any of the other “big tech” companies, then you really don’t need to cave in to this unreasonable pressure. You’ll be fine.

Do things because you love to do them. I have the opportunity to do software development stuff 32 hours a week⁷, why would I want to do even more of it? In my free time I want to do dumb experiments with my hardware and try out new ideas in my homelab. I also like writing a lot, so that’s what I end up doing.

Take care of yourself, and learn about the symptoms of burnout. It’s OK to take a rest if you need it. I wish someone told me this while I was in university, would have prevented quite a few chronic health issues.

Programming skills don’t matter as much as think they do. They still matter, don’t get me wrong, but it’s a relatively small part of the job. The ability to work well with others and a problem-solving mindset will take you very far, and you’ll figure out the technical details along the way.

Don’t do it for the money. The money won’t cover the therapy that you’re going to need to get over the soul-crushing agony that you experience every day if you secretly hate software development and everything around it. There’s a lot to dislike in the industry even if you like this role, so I can’t imagine how bad it might be as someone who isn’t able to enjoy the good parts of it.

I also have a list of good articles that will hopefully give you a better idea about the industry, the role and the expectations to a software developer.

Everyone has their own path to becoming a software developer, and this one is mine. Yours will probably be different, and that is perfectly fine.

if you only knew grayscale interfaces all your life, then you’d be excited about a change like that as well. ↩︎
Sports Car GT launcher also went from reporting 1 MB of VRAM, to -1 MB, to -1535 MB. For some reason it’s a core memory of mine. I don’t know why, either. ↩︎
it’s called Old-School RuneScape now. I’m not old, you’re old! ↩︎
and I was damn good at it, too. ↩︎
it’s not a weird flex, I just never planned on switching schools, but the grades were good and the toilets in the new school weren’t thick with smoke, so I switched. ↩︎
it’s been 10 years. That’s a long-ass time. ↩︎
if you haven’t tried a proper 4-day work week, and you have the means to do it even with an effective 20% pay cut, then try it, it will be life-changing. ↩︎

Running my ThinkPad T430 with an eGPU in 2024

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Wed, 07 Aug 2024 06:00:00 +0300

I probably shouldn’t have written down my notes on the eGPU setup I had years ago.

I’d be lying if I wasn’t considering remaking this setup with everything I’ve learned 6 years later.

Oops.

I got access to an allegedly-faulty AMD Radeon RX 480 and an NVIDIA GTX 1650 for free thanks to my friend, so I ordered the EXP GDC Beast v8.5c, two sorts of 6/8pin power cables, and a 12V 12A DC power supply.

The idea was to take a ThinkPad T430, bolt a GPU to it, and use it as a stationary workstation until the laptop dies or gets too slow for modern tasks. The resale value of the T430 is too low for me to justify selling it, and if there’s anyone out there who can run a T430 into the ground 5+ years from now, then it’s probably going to be me.

Quick specs:

CPU: Intel i7-3820QM
RAM: 16GB DDR3
Storage: 4TB Samsung 870 QVO

Side-note: I also went ahead and replaced the thermal paste with a bootleg Honeywell PTM7950 phase-changing thermal pad. It works just as well as fresh thermal paste, but hopefully it won’t drip out if the laptop is running up-right, in a vertical laptop stand.

Side-view of the EXP GDC Beast adapter.

Top-down view.

Backside of the adapter, with the AMD RX 480 attached.

The extra power cable was barely long enough to fit.

Testing

My testing with the GTX 1650 was brief, as the open source nouveau driver crashed on the Wayland desktop¹, the proprietary drivers under Fedora did not seem to work, and under Windows the NVIDIA driver intentionally triggers the error code 43 issue. A tweak exists for the Windows issue, but that didn’t help much either.

The AMD Radeon RX 480 seems to work well out of the box on Windows 11, plug-and-play. However, it wasn’t that stable, but that could be related to the possibly faulty GPU itself or the power supply that can barely drive it under load. Or the fact that Windows itself was running off of a 128GB Samsung USB 3.0 flash drive. Probably the latter, given that the USB stick died as a result of this experiment.

Microsoft does not approve of this setup for at least 3 reasons.

On Fedora Linux 40, I discovered that eGPU-s aren’t really plug and play. Sure, the image is there, but on my ultrawide monitor I saw a whopping 11 frames per second! Turns out that by default GNOME on Wayland doesn’t like to use the eGPU as the main rendering device, even on more legitimate Thunderbolt-based eGPU setups. It’s a common enough problem that there exist multiple solutions to this issue.

I didn’t like any of them², so I disabled the i915 kernel driver responsible for driving the iGPU, which did the trick. The 3440x1440p ultrawide monitor was being rendered by the eGPU and for the most part the experience was very smooth.

It's booting Linux!

The iGPU in the T430 can also run the ultrawide monitor at 60 Hz via the DisplayPort port on the dock, but it can’t run things at 60 FPS at all times because it’s really, really weak.

The eGPU ran things much smoother, which made the whole setup feel more responsive and great to use. It’s not the fastest setup in the world, but the GUI running smoothly certainly makes it feel like one.

I didn’t have a ton of games installed for testing, but I did give Minecraft a try, and it ran just fine. The CPU was struggling when building new parts of the map, but the visuals were doing fine and I could see myself playing on this setup for hours.

There were some things that weren’t running smoothly even with the eGPU. For example, 1080p H.265 video playback was awfully choppy. Using radeontop revealed around 80% GPU usage, so either the decoding engine is crap on this GPU, or the very limited PCIe bandwidth and CPU-GPU data transferring is the culprit.

I also tested LibreELEC to see how it handles the eGPU and I’m happy to report that it renders Kodi on it by default, with the UI running very smoothly.

On Linux I also saw some stability issues and crashes, but those could also be down to the faulty GPU or the PSU being underpowered. I can probably rule out the latter by limiting the GPU clock speeds, and the Arch wiki has some great instructions for that. Those toggles do work, but I noticed that GNOME can be a bit choppier when the GPU switches between lower power states to higher ones.

The physical stability of this setup is questionable, especially if you use a bigger GPU. Most prefabricated cases for this eGPU adapter assume that you’re running it with an ATX/SFX PSU, so those are out of the question for me. There exist some 3D printable designs out there, but they can sometimes be very specific to a particular setup or GPU, so I’d have to design my own.

Work in progress.

I really like the look of the GPU, no need to worry about a case. Unless you have cats, which I do have. Uh-oh.

The final setup.

The power consumption of this setup isn’t great, unfortunately. Using a smart plug I measured the idle power of the eGPU component to be about 20W. Typical desktop usage and video playback results in around 50W and peaks around 120-130W, which is right on the limit of the shoddy PSU I’m using.

If I had the choice to pick any GPU for this setup, then I’d likely opt for a modern AMD GPU that didn’t require additional power via the 6/8 pin power connector as that option would likely be a lot more efficient while yielding similar performance.

I ended up disassembling the setup in the end, partly because I misdiagnosed a Mesa bug to be a GPU issue, and because the RX 480 really was acting weird from time to time, even with LibreELEC.

That can't be good.

Not all experiments end up being successful.

Conclusion

I think that these types of eGPU adapters are great even in 2024, but only for a small number of very specific use cases, and with GPU-s that actually work.

If you already have an older laptop around, and a compatible spare GPU collecting dust, then this setup will make sense for desktop or media playback machines, and perhaps gaming if your demands are not very high.

I don’t suggest building this setup from scratch, it’s probably not worth the money and hassle. Used gaming PC-s that have similar specifications to this eGPU build, but with no funny PCIe bandwidth limitations, go for less than 200 EUR at this point.

nouveau? More like novideo, am I right? ↩︎
not that they’re bad, they are just made for a different type of setup. ↩︎

No HDMI port on the ThinkPad T430? No problem!

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Mon, 08 Jul 2024 13:00:00 +0300

The ThinkPad T430 has a few options for running it with an external display:

VGA port, which is pretty much obsolete at this point
mini DisplayPort connector on the laptop itself
DVI or DisplayPort on a dock

The mini DisplayPort port has annoyed me for as long as I’ve had this machine.

Most places where I’ve had to present something only offer an HDMI cable, which means that I always have to carry a dongle around, and I keep forgetting to bring one everywhere I happen to go.

Until now.

I have a few of these SATA HDD adapters that replace the optical drive on the ThinkPad T430, and I discovered that my mini DisplayPort to HDMI adapter can fit in one without a problem.

It's a snug fit so it doesn't fall out.

You'd never know the adapter was in here.

I don’t currently have a need for a second SSD in my T430, so this “mod” makes perfect sense to me.

I bet there is someone out there who is capable of routing an actual HDMI port in place of this adapter. The existence of FHD mods for this laptop suggests that this is possible. If you’re that person and created that mod, let me know.

About the time my ThinkPad T430 ran with an external GPU

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Wed, 03 Jul 2024 06:00:00 +0300

The ThinkPad T430 is not a remarkable laptop. It’s thick, bulky and built like a tank. I got mine in 2016 when the first university scholarship money dropped, and it’s still my backup laptop of choice.

Around 2017 I did something every reasonable poor computer science student would do: I got an eGPU adapter for it to play some games. I never ended up playing many games, but I loved tinkering with and testing this setup a lot.

These are my notes on the setup that I used to have. The notes used to be on Reddit, but after yet another user-hostile change¹ I deleted my account and all the content associated with it.

The specs

The ThinkPad T430 that I used had a CPU swap at one point, running an Intel i7-3820QM. This upgrade doubled the CPU performance on the laptop compared to the dual-core CPU that was in it before. This change was critical because it opened the door to running lots of modern games at reasonable performance.

The eGPU adapter I ran with was the EXP GDC Beast v8.4, using the ExpressCard34 connector. This adapter turned the ExpressCard34 slot into a PCIe 2.0 x1 slot. It’s not really hotpluggable, but using this connector instead of something like the internal Wi-Fi adapter slot made this setup much more portable. Done gaming? Shut down, disconnect the ExpressCard cable and you’re good to go!

This adapter is also a good example of connector reuse. The other end of the ExpressCard plug is an HDMI port. Video signals require good quality cables, and so does PCI Express.

Close-up of the ExpressCard34 connector.

The eGPU was once powered by a simple ATX PSU with the jumper wire set on the 24-pin connector, but eventually I bought a Dell DA-2 power supply. The adapter has a power connector designed specifically for that pinout and it made the setup a lot cleaner.

The setup with an ATX PSU.

The setup with a Dell DA-2 PSU. Much cleaner, isn't it?

At one point I got a fully metal case for the eGPU. It made the setup a lot cleaner and the risk of the GPU tipping over while turned on was also substantially reduced.

The case had no right to be this good for a janky setup.

I tested a few GPU-s:

AMD Radeon HD 6870
nVidia GTX 950
nVidia GTX 1080
AMD Radeon RX 480

The HD 6870 was an old card back in 2017, and it did not run great at all with this eGPU adapter. It was buggy and the performance sucked.

Loaner GPU-s, such as the RX 480 and GTX 1080 were much better experiences, as long as you output the video signal to an external display right from the GPU.

The nVidia GTX 950 was the card I ran with for the longest time in this setup. It worked well with this setup and had decent gaming performance.

GPU-z info while the GPU is idle. Note that the PCIe link speed is set to gen 1.1 under "Bus Interface".

When the GPU is under load, the PCIe connection switches to PCIe 2.0.

eGPU tech tips

You will want to use an eGPU adapter with an external monitor for the best performance. The PCIe bandwidth is very limited in a setup like this. Using the internal display of the laptop will mean that the final rendered image will have to be sent back from the eGPU to the CPU. 1x PCIe 2.0 lane is about 500 MB/s of bandwidth. Sending 60+ frames in the native resolution (1600x900) will eat it all up. The higher the resolution on the internal display, the fewer frames you’ll get.

I tested out these two scenarios in CSGO. The internal display yielded about 61 FPS while an external display would result in 100-180+ FPS.

The PCIe link bandwidth was not actually that noticeable in a lot of games, such as GTA V, Rocket League and Dirt 3.

In some games, such as Dirt Rally (2015), it did not matter what settings you went with, something about the game made it run poorly no matter what graphics settings you went with. In cases like that you might as well cap the framerate to something tolerable like 30 FPS and crank the graphics settings up.

VFIO on a laptop???

This setup was also around the time when I first learned about the magical world of VFIO and GPU passthrough.

The logical continuation of that discovery was to try it out on my laptop, and to my surprise it actually worked. I could pass the eGPU to a VM and play GTA V on it!

This setup ran on Antergos OS, a flavor of Arch Linux. It’s been so long since then that the flavor of Arch Linux is now discontinued.²

Antergos OS on the host, Windows 10 on the guest VM running Furmark.

The laptop was ideal for this purpose as well. You have the integrated GPU, display, keyboard and mouse for the host system, and you can pass USB devices and the eGPU to the VM.

There were the usual VFIO tricks that I had to try out to avoid the dreaded error code 43 issue, but after that it was smooth sailing.

The performance was very similar to what I saw on the host system.

Here’s a clip of me recording this setup while playing GTA V. Smartphone cameras have come a long way since then.

ExpressCard eGPU-s in 2024?

What got me started with writing down my notes were some YouTube videos that were made on this topic recently.

A pimped-out ThinkPad T430 can still be a perfectly serviceable computer in 2024, and an eGPU will completely sidestep the problem of the weak integrated GPU.

The T430 is not powerful enough to natively run a 3440x1440 ultrawide monitor at 60 Hz. It is even struggling to do it at 30 Hz. An eGPU adapter with a modern low-power GPU, such as the AMD Radeon RX 6400, could solve this issue nicely and give the ThinkPad T430 a new lease of life.

I’d be lying if I wasn’t considering remaking this setup with everything I’ve learned 6 years later. GPU pricing sucks, though.³ They still sell these eGPU kits in various forms, however the price has doubled or even tripled over the years.

If you’re interested in building a similar setup, then do check out egpu.io. With an M.2 or Thunderbolt connector you can build a much more capable eGPU setup.

If you can get a PCI Express signal out of a computer, then you can probably rig a GPU up to it. Unless you’re Jeff Geerling, in which case you might need to recompile the kernel a few times.

guess which one. ↩︎
validates my view that it’s still a good idea to stick to the big distros that these short-lived Linux distros are based off of. ↩︎
I’ve survived two cryptocurrency mining hype cycles, and now we are in an LLM bubble. Great. ↩︎

Back to roots

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Mon, 24 Jun 2024 14:00:00 +0300

This blog is running on a home server again.

I have once again gained access to a competent internet connection¹, and I think I have figured out the IPv6 setup as well², leading to this change.

The IP address is dynamic, there are occasional power outages and I might just mess up my configuration and bring it all down, but I get to brag about this setup so it all balances out.

it’s fiber, baby! ↩︎
the software support for IPv6 is a goddamn mess and I regret learning about it. ↩︎

How to save an old printer from the e-waste pile with a Raspberry Pi

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Wed, 12 Jun 2024 06:00:00 +0300

A family member has a Canon PIXMA MP250 printer, originally released in 2009. It has been a very reliable piece of hardware, especially for a printer.

Then came Windows 10. The printer would not work out of the box with it and the official drivers got stuck during installation. Fiddling with the printer in device manager, trying to install drivers via Windows Update and stars aligning got the printer to work again.

Then came Windows 11. Nothing I did could get it working now, and the printer is not even officially supported by Canon on this version of Windows.

This printer works out of the box on any mainstream Linux distribution, and the family member had been using the printer with Ubuntu for years, but a certain piece of important software in Estonia was not working properly a lot of the time, which triggered a switch to Windows.¹

Quick searches online suggested that I could turn this printer into a network printer, as long as I had some patience and a spare computer.

The idea is simple: take a spare computer, hook it up to the printer, install CUPS on it, configure it and you’re good to go!

I initially tested this setup with a Zimaboard running Fedora Server, but the final solution used an old Raspberry Pi B+ running the latest version of Raspberry Pi OS.

How to set it up

The steps will be similar with other pieces of hardware and Linux distributions, you may just need to adjust a few commands and package names.

The steps are roughly based on these two guides:

If you’re following along with a Raspberry Pi, set up your OS first. A minimal installation with no graphical interface will be just fine.

Next, install CUPS and optionally the right printer drivers for your printer. The Canon PIXMA MP250 requires the package printer-driver-gutenprint.

sudo apt install -y cups printer-driver-gutenprint

Make sure that CUPS is running.

sudo systemctl enable --now cups.service

Next, allow your user to manage printing. In this example, the username is pi.

sudo usermod -a -G lpadmin pi

Next, make sure that CUPS allows connections from other machines in your network, and then restart it.

sudo cupsctl --remote-any
sudo systemctl restart cups

Find out the IP address of the Pi and navigate to it in your browser. It will look something like https://192.168.0.49:631. The browser will likely warn you about a self-signed certificate. It’s safe to proceed.

If you haven’t already connected your printer to the Raspberry Pi via USB, then do so now.

Use the graphical interface to add a new printer under the Administration section. You’ll need to enter the username and password for your user here. It’s the same user we used earlier, so in this example it’s pi.

If you can’t find your printer model listed in the Add printer view, then you are probably missing the proper driver package, or your printer simply isn’t supported.

You should be able to find your printer in this view.

After adding your printer, you can use the CUPS web interface to print a test page.

If that worked, then you should now be able to use any other device on the network to print!

On Linux and Windows, open the settings section that handles printing, and add a new printer. This networked printer should appear automagically, no additional configuration required.

I also did testing using a Fairphone 5 running Android 13, and an iPhone X running iOS 16. In both cases the printing just worked.

Test page printed using Windows 11.

Test page printed using Fedora. Yes, most of the colors in the colored ink cassette have run out.

I’ve used the scanning functionality of this particular printer on Linux and it has worked, however I forgot to test this out using this setup. I’ll try to test that out once I get the chance to try the printer out again.

Hardware tech tips

I first ran this setup with a Zimaboard to test the idea out. It worked very well, but as a simple print server it is a bit overkill.

I then replaced it with a Raspberry Pi B+, and while the printing works, I discovered that a single ARM CPU core is apparently a limiting factor during printing. A simple test page print pegged the CPU at 100%, and that’s with the 700->1000 MHz overclock applied.

Raspberry Pi B+ struggling during printing.

If you don’t mind a slower printing speed, then any Raspberry Pi will be absolutely OK. More performant versions of Raspberry Pi will likely fare even better.

If you already have a small home server running, then you may want to hook the printer up to that and set up the printer there. It’s already running 24/7, so it would be a perfect match.

One useful aspect of this setup is that you can queue up print jobs even when the printer itself is powered off. Once you turn it on, the print jobs will start, assuming that you have enough paper and ink.

Free open source software prolongs the useful life of hardware once again. See y’all in October 2025!

2024-06-15 update

This post got featured on:

And even the leader of OpenPrinting, Till Kamppeter, chimed in! They also held a talk about OpenPrinting at FOSDEM 2024.

Regarding scanning, seems like there’s a separate project for that called SANE - Scanner Access Now Easy.

Find out more about OpenPrinting here.

that might be about to change as there is an unofficial Flatpak version out there. ↩︎

They make USB-C cables with displays now!

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Wed, 05 Jun 2024 06:00:00 +0300

I’ve reached a point in my setup where most of the devices that I use are based around the coveted USB-C port. This meant that I had a valid reason to get a few extra because I didn’t yet have a stockpile of good USB-C cables.

That’s when I found out that there exist cables that have little screens on them that show the power consumption of the connected device. This is a great little addition to my power consumption monitoring addiction. It’s also a simple way to understand if your device is charging at the speed that you expect it to.

The cable I ordered cost 6.72 EUR. It is a bit stiff and hard to work with, probably due to its supposed USB 4 support requiring actually good cabling and shielding. It works well enough for an USB-C dock with a DisplayPort connection that’s running a 3440x1440 display at 60 Hz.

It’s too early to give a definitive answer about the longevity of the cable.

You should be able to find these types of cables with a search query like “USB 4 Cable with LED Display”.

The paranoid side of me suspects that a cable like this one would be an ideal place to hide a malicious chip. That’s the only downside that I can think of.

If you’re interested in going more in depth with measuring the power consumption over USB, then you might want to look at other options. This GitHub repository lists a few examples of these types of measuring devices.

Steam Deck: I like it

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Sun, 02 Jun 2024 06:00:00 +0300

I got a Steam Deck.

Only took me a year or so of contemplating getting one, and trying out HoloISO, the unofficial SteamOS installer finally convinced me to get one.¹

It took another year to actually get down to writing down my thoughts on it.

This post is written from the perspective of a software developer who used to play video games a lot as a teenager, less so as an adult, and as someone who dabbles with Linux as a hobby.

Overview

The Steam Deck OLED exists now, but back when I got my Steam Deck, it came in three flavours, and in the interests of my budget I opted for the cheapest option that came with only 64 GB of storage (eMMC based).

Steam Deck, the packaging, and cat.

I also got a 512 GB microSD card to hold all the games. All in all, this setup cost about 470 EUR. Certainly cheaper than the 512 GB NVMe SSD model that sold for 679 EUR at the time of writing. The shipping estimate to Estonia was about 1-2 weeks, and mine arrived in almost exactly 1 week, which was nice. And in one piece as well!

Half a terabyte of games, all in that small microSD card.

I don’t have the official Steam Deck Dock (try saying that 10 times in a row), but what I do have is a vertical laptop stand and a Dell WD19TB dock. As long as you leave a gap for airflow, then this combination works quite well if you want to use the Steam Deck in a docked mode. The combination of this dock and the Steam Deck also works nicely, the 3440x1440p ultrawide monitor gets picked up fine and even the gigabit Ethernet port works at the rated speed.

However, it seems like the Steam Deck USB-C compatibility can be dependent on your monitor and dock choice. While the Dell WD19TB dock worked great with an ultrawide monitor, a Lenovo Thinkpad USB-C dock would fail to output anything to the same monitor quite often. Turning on the setting to enable safe resolutions for the external display and setting the resolution manually usually fixes it, but it’s still quite annoying.

Steam Deck docked.

It’s booting Linux!

Not long after I set the Steam Deck up, I switched to desktop mode and opened a terminal window to confirm that yes, this thing is running Linux. Arch Linux, in fact, or at least a modified version of it.

btw I run Arch

But in all other aspects, it’s just like a normal computer, with an actual UEFI setup that you can browse. With the Steam Deck, you get the polished experience of a gaming console and the freedom to do whatever you want to with the software and hardware.

You can also boot any x86-based Linux distro, like Fedora Linux.

If all my other computers stopped working, then I could totally work out of a Steam Deck.

Or if SteamOS 3 is too limiting for you, then check out projects like Bazzite. The experience will be very similar, but it’s easier to tweak the setup to your liking. Bazzite also allows you to format storage as btrfs with compression enabled, meaning that you can fit more games on your device with a very slight increase in CPU usage. I’m sticking with the official Steam Deck experience for now, but I’m tempted to try Bazzite for the btrfs aspect alone.

The desktop experience on the Steam Deck is quite nice. You can easily boot into it from the graphical interface, and it runs on KDE Plasma. The design language of KDE is an eyesore, but it makes up for it with the responsive UI.

The touchpads are not that big, but they’re plenty good for clicking around. Manually popping up the on-screen keyboard by pressing STEAM + X is a bit annoying at first, but it gets the job done.

Installing software can be done in the graphical environment via the “Discover” app that ships with the desktop environment. One thing you’ll probably notice is that the selection of software is a bit limited, and that’s mainly down to how the Steam Deck is set up by default. All the apps you install are installed via Flatpak and while the selection of apps that are available as a Flatpak grows with time, not everything is yet present there.

I was surprised to see that the Steam Deck Desktop FAQ actually has useful information for an enthusiast like myself, even mentioning what a flatpak is.

Peeking under the hood

During the first couple of months I went the tinkerers’ path, enabled the SSH server, unlocked the root partition for modifications and automated the setup of my Steam Deck. While this thing is running a flavor of Arch Linux, it’s not really designed for these types of manual modifications. I had to mess around to get the Arch default package manager pacman to work, but at one point messing with pacman and running commands that I didn’t know much about, I blew up the OS by force-installing some packages and throwing out a crucial dependency. Whoops.

Well, at least the Steam Deck OS recovery setup is quite straightforward. Download the OS recovery image, put it on an USB stick, boot it on the Steam Deck and re-image the OS. There are also options to attempt repairs or open a terminal window in case you know how to fix the mess you’ve created.

One downside of the SteamOS recovery setup became apparent when Valve released a buggy software update around summer/fall of 2023. The update was buggy for many users, including myself, with games crashing or running poorly. On my Deck the SD card was also automatically reformatted for some reason so I had to redownload all the games I had on that. When I went to download the recovery image, I discovered that you could only go back to the current, buggy version of Steam Deck OS.

By fiddling with the download link I could navigate to a web directory where older recovery images were present. Those weren’t much help though. Although you could boot into an older version during the initial Steam Deck setup, it would auto-update to the newest version automatically, with no obvious way to bypass it.

The real workaround would have been to manually boot into the older version of the OS that sits on the other partition and not update for a while. I unfortunately discovered that after I had re-imaged the Steam Deck, which formats the whole drive.

I’ve now stopped with the tinkering. I still have an SSH server running on it, but it’s there so that I can install my favourite applications as flatpaks and deploy the backup script, via Ansible. That’s it.

It’s booting… Windows?

Cursed Steam Deck.

Don’t worry, Valve doesn’t even officially support running Windows on this thing, and while you can do it, I don’t really recommend it. You’ll be giving up all the nice things about SteamOS while also taking on the hassle of running Windows, including unexpected Windows updates.

Games are too damn big

512 GB of storage on something as small as a fingernail was something I would have never expected to see 10 years ago, but here we are. There are even bigger microSD cards out there, but those also cost a lot more.

And even with this marvel of technology I ran into issues when installing games. Install GTA V, Forza Horizon 4, and God of War, and just like that about half of all that storage is gone.

The 64 GB internal drive was also quite limiting, especially due to shader cache and compatibility layer (Proton) related files taking up a majority of it. The situation has improved now, but I still went ahead and replaced it with a 1 TB M.2 2230 NVMe SSD that I ordered online for 74 EUR in May 2023, which still makes it a cheaper option than the 512 GB model sold at the time.

The new SSD works well, it’s fast and doesn’t seem to affect battery life negatively. Changing it was quite simple, all you need is an appropriate screwdriver, good fingernails or a plastic spudger to get the case off, and some delicate care when working inside the Steam Deck.

Before I replaced my SSD, I tried all sorts of wild-ass ideas to fit more games on my Steam Deck.

First, I tried Nextcloud. The idea is simple: I’m not going to be playing all my games all the time, so it makes sense to offload some of those to my home server and retrieve them over the fast local network whenever I want to play them. This idea makes even more sense if your internet speeds are not that great.

The results were mixed. The syncing part was working surprisingly well with the transfer speeds capping out at 1 Gbit/s over Ethernet and actually being faster than the microSD card itself at times. However, the usability of this setup is not that great. You can select which folders to sync in the Nextcloud Client application, but Steam will still believe that a game is installed even if the game files are missing as long as the app manifest file is present. This can lead to annoying situations where you try to play a game and discover that the games files are not present on your Steam Deck.

The second solution I tried out was iSCSI. I followed this guide on Reddit. Getting iSCSI to work was tricky even for someone like myself who should be able to do this technical stuff properly, but I ended up getting it working. I formatted the iSCSI device as ext4, mounted it and made it a Steam library folder in Steam desktop mode.

The benefit of this solution is that as long as you have access to the server, the iSCSI storage device will show up on your Steam Deck as any other drive and games will just work. If your network is good enough, you can play games that are stored on your server, which might make a lot of sense for games that you generally play in docked mode. Transferring games to your internal storage is also simple, performant and can be done entirely within the Steam UI. The iSCSI setup is not that great mainly due to the technical complexity of the setup, making it a no-go for the more casual audience. The performance over Wi-Fi was also lousy, and it’s also difficult to justify having a good chunk of your home server storage being permanently held up by Steam games.

After all that hassle, I went with the solution that most people would have gone with already: store games on an external SSD. Duh. It has the same benefits as the iSCSI setup, but it’s faster and you can bypass a lot of the technical complexity. The external SSD is also automatically mounted when you connect it to the Steam Deck if it’s formatted as ext4, making it a very convenient option for docked play.

Not as janky as you'd think.

If you happen to have a fast desktop PC with plenty of storage, or a home server, then consider using the Steam local network game transfers. You won’t fit more games onto the Steam Deck, but redownloading them will be much faster.

We really need to start decoupling high resolution textures and other high quality assets from the base game at some point because the current approach is just silly. For a device like Steam Deck you’re rarely going to run those games with the highest quality settings, meaning that those assets will just sit there on the disk taking up space.

A free optional DLC could work well in theory. In practice I’m sure that game publishers would quickly turn a high quality texture pack into a paid DLC.

The part where I play games

Enough about the part where I tinker with the system. Let’s talk about the games.

"art of rally" is a great game to play on the Steam Deck.

I had some idea about the game compatibility when I tried out HoloISO. I’m happy to report that the game compatibility is even better on the Steam Deck. This fact alone is a minor miracle. Some years ago I was convinced that the only way to properly play games on Linux was to set up a VM with Windows installed in it.

With the Steam Deck, you download the game and run it. Simple as.²

The Steam Deck is not the most powerful machine in the world, but it has a decent CPU in it. The GPU is more akin to integrated GPU-s that ship in modern AMD Ryzen based laptops, it’s still plenty of power to run games at the native 1280x800 resolution.

God of War (2018) did stutter a bit, but in most situations you can expect around 40-60 FPS.

GTA V was running fine as well, although the game is CPU limited at times and can drop to around 40 FPS in certain sections of the map, especially in GTA Online.

Teardown runs alright after you lower the settings, and it’s best played in docked mode due to some fun mods being designed around keyboard controls. It’s really fun, but it will bring the Steam Deck to its knees when you do some heavy destruction. Totally playable though, and I’ve had a lot of fun with this game.

Wreckfest runs well with medium or high settings. I feel the framerate dropping a bit when heading into the first corner and being part of a 24-car pile-up, but in all other situations it’s smooth enough.

Need for Speed Hot Pursuit (2010) runs very well and is an absolute blast on the Steam Deck. Crank the settings to maximum and enjoy the thrilling police chases.

Need for Speed Most Wanted (2012) is also great on the Steam Deck.

Need for Speed Rivals (2013) runs well after you apply the tweak that forces the game to run at 60 FPS. The game is a bit buggy even on Windows, so expect some bugs and crashes.

More modern Need for Speed titles, such as Need for Speed (2015), Need for Speed Payback (2017) and Need for Speed Heat (2019) are playable, but will typically run anywhere between 40-60 FPS. The handy framerate capping feature of the Steam Deck will help a lot in those cases since a stable 40-45 FPS is a much better experience than a wildly fluctuating and unstable framerate.

More lightweight games run like a charm while using very little power. This includes titles like Vampire Survivors³ and Donut County.

What I really like about the Steam Deck is that performance tuning is very accessible. You can toggle mangohud with various levels of detail to narrow down any performance troubles, and you have great control over the CPU and GPU speeds. All of this can be done while the game is running.

It's like MSI Afterburner overlay, but built-in and all set up for you.

The APU in the Steam Deck tries its best to share the limited power that the system can pull between the CPU and GPU, and in most cases it works well, but there are games where the CPU or the GPU demands can fluctuate a lot. In those scenarios it helps if you lock the GPU clock speed to an amount that keeps the GPU usage to around 80% or less. Being able to play with settings like this is one of the reasons why I got Forza Horizon 4 to be playable within minutes of tinkering.

Forza Horizon 4 on the Deck runs surprisingly well after you tweak a few settings.

There are a few games that I like to play but aren’t working well on the Steam Deck, such as GTA III or Vice City, but that is pretty much it. Over the one year that I’ve had the Steam Deck, a lot of games have gained support, either by Valve improving the Proton compatibility layer, or game publishers releasing updates that make the game work on the Steam Deck.

One such example is FlatOut Ultimate Carnage, which seems to have been bought by a new publisher that removed the “Games for Windows Live” mess and got the game “Steam Deck Verified” in the process.

The performance of the Steam Deck was actually quite surprising in a lot of games. For example, I’ve never seen GTA IV run so well on any machine that I’ve owned before.

The best GTA IV experience I've ever had.

If you want to see if your favourite games are compatible with the Steam Deck I recommend you check out ProtonDB for a detailed overview. YouTube is your friend if you want to see some specific games being tested on the Steam Deck.

When it comes to emulating PS1, PS2 and PSP games, the Steam Deck is fantastic. There’s plenty of performance to emulate those consoles well, and for whatever reason playing the childhood classics on the Steam Deck feels so right. With PSP the comparison is even more apt as it’s basically a bigger PSP but with a much better battery life.

Steam Deck and the whole PSP game library is a great match.

More intensive games can discharge the battery in about 2-3 hours, but when playing emulated games for older consoles you can expect up to 7-8 hours of gameplay.

Non-Steam PC gaming

It’s clear from the name that this console is set up around Steam. As long as all your games are purchased on Steam, you’ll have a great time.

I do have some games from other sources, such as GOG.

There exist community-supported launchers, such as HeroicGamesLauncher. They work, but might require a bit of setup and fiddling to get things like cloud synced game saves to work properly. In the case of art of rally on GOG, you’ll also lose out on multiplayer daily/weekly challenges.

If you have some older games on CD-s, or legally acquired archived copies of PC games, then the situation gets tricky. There exist solutions like Lutris or Bottles, however I never quite had a good experience with those, especially once you deviate from the happy path.

In my experience, if you want to have the best chance at playing your old PC games, then you’ll need to obtain a version of the game that has a no-CD patch applied to it already, add it to Steam and set a version of the Proton compatibility layer. Some legally dubious sources for these games label these packages as “direct play” as well.

There can also be a difference when running games via gaming or desktop mode. Need for Speed Most Wanted (2005) did not work when started via gaming mode, but it started up just fine in desktop mode.

Proton, the compatibility layer, is great for what it allows us to do, but I feel like the concept and technical implementation is something that novice users will never quite understand. This also becomes apparent when you look up discussions online where novice users try to apply mods to their favourite games. It works out of the box on Windows, but you need to grok the way Proton handles things for it to work on the Steam Deck.

I consider myself competent with computers and stuff, but even I need to sit down and think about what’s going on when working in this area.

Backups

I rely on Ludusavi to collect all my game saves, and I use rsync to back them up to my home server.

I made a desktop entry for the backup script. By right-clicking it in desktop mode and selecting Add to Steam, it will appear in your Steam library like any other game or application. This means that I don’t have to quit gaming mode to back up the Steam Deck.

The backup script also updates all the applications that I’ve installed as a flatpak, and shuts the device down after it’s done.

It’s big.

The Steam Deck is big.

This does mean that Valve can pack a lot of compute power and cooling into it, but it comes with the trade-off of it not fitting that well into a backpack. The hardcase that it ships with is great for preventing damage to the Steam Deck, but it takes up a lot of space.

I don’t find the size uncomfortable, quite the opposite actually, but for others it might be a problem.

I’ve had the opportunity to briefly play on a Nintendo Switch Lite. There’s something that feels special about that form factor. The performance is not great compared to the Steam Deck, but I feel like the Nintendo Switch is in its own class of portable gaming consoles.

I wonder how a Nintendo Switch sized Steam Deck would perform.

The controls

The controls on the Steam Deck are good. It takes the best aspects of the Nintendo Wii U and Xbox 360 controllers and smashes them into one polished experience.

The Steam Deck has some extra mappable buttons on the back. They sound great in theory, but I rarely use them. When I do, I find that the L4 and R4 buttons are more easily reachable compared to L5 and R5 (the lower ones).

When I do use those extra buttons, they are invaluable. In Need for Speed Rivals (2013), I mapped L4 and R4 to the two power-up slots, which left my thumbs free to focus on driving and looking around.

Using those extra buttons does require some readjustment. It was difficult to reprogram my muscle memory and remember that I have those extra buttons there.

Steam also provides lots of controller templates, both Valve-approved and community ones. If the game you want to play doesn’t have official controller support, then it’s highly likely that someone else has created a controller configuration that emulates keypresses and allows you to play your favourite game on the Steam Deck.

Sometimes that doesn’t quite work, in which case you can create your own layout, or improve an existing one.

There’s also the gyro functionality, but I’ve only used it in Aperture Desk Job, which itself is like a fancy demo of the controller capabilities of the Steam Deck. Never quite found a suitable use case for this input method in other games.

The fun factor

Based on what you read, you might think that I got the Steam Deck for the tinkering aspect. While that’s certainly part of the rationale behind getting one, it wasn’t the main one.

I got one as an excuse to finally spend more time resting. I have had a bad habit of trying to be productive at all times, which can lead to exhaustion and eventually burning out.

With the Steam Deck I’ve been able to actually spend some time playing games and turning my brain off.

I love it.

And it gives my cats a great opportunity to snuggle up while I do some illegal street racing. Win-win.

There’s something different about playing games on the Steam Deck. Perhaps it’s related to the memories I had as a teenager playing on my PlayStation Portable a lot, or knowing that I can replay all my favourite console games without having to set up the physical consoles themselves.

The sad reality of adulthood

I love using the Steam Deck. However, I still can’t seem to make much time for it when life gets busy.

That’s my only big gripe with it.

Conclusion

If you’ve been on the edge about getting a Steam Deck, then I hope that this post has helped you make a decision. It has some rough edges, but when it works, it feels magical.

If you already have a gaming PC with a modern AMD GPU in it, then consider trying out projects like Bazzite, HoloISO or ChimeraOS to get a taste of the Steam Deck experience.

Even if I can’t put in as many hours as I’d like to, I still love my Steam Deck and don’t regret getting it at all.

this makes Valve’s silence on releasing SteamOS 3 to a wider audience puzzling. Do they not want to sell more Steam Decks? ↩︎
assuming that it’s a Steam Deck Verified title. ↩︎
it’s really addicting, be careful. ↩︎

ThinkPad T40: it can still run modern Linux, for now

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Fri, 24 May 2024 06:00:00 +0300

I recently busted out my old ThinkPad T40, the last of the OG IBM ThinkPads.

I picked it up some time around my university days because I liked collecting ThinkPads at the time, and it was a nice complement to my existing ThinkPad T60 and T430.

20 years difference, but they still look similar.

The battery is dead, but everything else still works. Checking a few online listings, I’m surprised that I can still find batteries sold for this model. Probably old stock that’s been sitting around in a warehouse slowly discharging to death, but hey, you might get lucky!

I also learned that I had replaced the internal IDE hard drive with a 16 GB mSATA SSD via an adapter at one point. I don’t think they even sell SSD-s that small nowadays, but the size feels age-appropriate.

I had already installed a flavor of Linux on this, but had completely forgotten the password.¹ This was a good excuse to wipe it and see if I can install a modern mainstream Linux distro on it.

I went with Debian 12 (Bookworm). It’s the newest Debian release, and unlike many other distros, they still offer 32-bit installers.

Modern Linux on an ancient laptop.

It had been a while since I had last installed Debian via the terminal user interface. The experience felt surprisingly snappy.

It's probably been 5+ years since I last installed Debian on a machine. What a throwback!

I went with the good old XFCE desktop environment. It strikes a good balance of being lightweight, simple and usable.

Things were great until I started up Firefox. It works, but man, is it slow. All the years of browser complexity have caught up with this laptop and made the web barely usable.

It does pass the techtipsy test.

Insert joke about recursion.

This will likely be the last time this laptop runs a modern OS. 32-bit CPU-s are losing support both in x86 and ARM ecosystems, and the year 2038 is not that far away any more.

So long, and thanks for all the fish!

note to future self: the username is thinkpad, and the password is thinkpad. ↩︎

OpenWRT, ISP modem and dynamic IP addresses: how to fix connectivity issues without rebooting your router every time

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Mon, 20 May 2024 06:00:00 +0300

My current ISP provides an internet connection over a copper wire. To use it, I have a crappy modem (Technicolor CGA2121, DOCSIS 3.0). It’s running in bridge mode, meaning that all it does is convert the signal running over the coax cable into plain old Ethernet.

My main networking device is a TP-Link Archer C7 v5. It runs OpenWRT. This router/Wi-Fi AP box connects to the modem and handles everything, including getting a public IPv4 address from the ISP.¹

After a power outage or my ISP doing maintenance, the public IP address has usually changed. This wouldn’t be a problem if I just stuck to the ISP-approved modem.²

With my setup, there was a problem. The OpenWRT box would try to operate with the IPv4 address that it was given since the DHCP lease had not yet expired. However, this meant that there was no internet connectivity. A reboot of the OpenWRT box would resolve the issue.

This manual workaround wasn’t good enough for me. It would be quite problematic if this issue happened while I was away from home because I’d still like to access my home server.

After traversing OpenWRT forums and consulting the Slack workspace of my local hackerspace, I found that bringing up the WAN interface again would result in the OpenWRT box getting a new public IPv4 address. Problem solved!

To automate this workaround, I created a single crontab entry in the OpenWRT box. This is also configurable in a graphical user interface as long as you have LuCI installed.

The crontab entry looks like this:

*/5 * * * * /bin/ash -c '/bin/ping -c 3 8.8.8.8 > /dev/null || /sbin/ifup wan'

Every 5 minutes, the router pings Google’s DNS server. If that command succeeds, then the internet connection works and that’s it. If the ping fails, then the other half of the shell command is executed, which brings up the wan interface on my router.

Feel free to use a different IP address to test with. Your WAN network interface might also have a different name.

The downside of this solution is that if the server you’re using to verify your internet connection is down or refuses pings, then you’ll be causing interruptions in your home network every 5 minutes.

Talking to the ISP about this issue was something I considered as well. Then I remembered that it took me 1.5 months of fighting chatbots and repeating the same information to different customer care agents to use my own modem that’s identical to the one the ISP uses. That’s a hell no from me.

I really should get around to drilling those holes in the apartment building to get access to a fiber connection. ↩︎
using the ISP-approved box would introduce a whole other set of problems because they are surprisingly low quality. ↩︎

The hidden media play/pause/stop keys on the Lenovo ThinkPad L390 Yoga

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Tue, 07 May 2024 06:00:00 +0300

ThinkPad keyboards were once well known for their great layouts, feel and functionality. This included the media playback control keys.

Media playback control keys on a ThinkPad T420 keyboard.

On the ThinkPad T430, the new chiclet keyboard layout moved the media keys to the function row. Still there, but less convenient to access.

Media playback control keys on a ThinkPad T430 keyboard.

The ThinkPad L390 Yoga doesn’t have any visible function keys for controlling media playback. However, I found that the play/pause and stop buttons are still functional on the up/down arrow keys. Left/right arrow keys act as Home and End keys.

evtest output for play/pause key combination:

Event: time 1714716211.222697, type 4 (EV_MSC), code 4 (MSC_SCAN), value e3
Event: time 1714716211.222697, type 1 (EV_KEY), code 143 (KEY_WAKEUP), value 1
Event: time 1714716211.222697, -------------- SYN_REPORT ------------
Event: time 1714716212.291293, type 4 (EV_MSC), code 4 (MSC_SCAN), value a2
Event: time 1714716212.291293, type 1 (EV_KEY), code 164 (KEY_PLAYPAUSE), value 1
Event: time 1714716212.291293, -------------- SYN_REPORT ------------
Event: time 1714716212.366568, type 4 (EV_MSC), code 4 (MSC_SCAN), value a2
Event: time 1714716212.366568, type 1 (EV_KEY), code 164 (KEY_PLAYPAUSE), value 0
Event: time 1714716212.366568, -------------- SYN_REPORT ------------
Event: time 1714716214.026847, type 4 (EV_MSC), code 4 (MSC_SCAN), value e3
Event: time 1714716214.026847, type 1 (EV_KEY), code 143 (KEY_WAKEUP), value 0
Event: time 1714716214.026847, -------------- SYN_REPORT ------------

evtest output for stop key combination:

Event: time 1714716254.780584, type 4 (EV_MSC), code 4 (MSC_SCAN), value e3
Event: time 1714716254.780584, type 1 (EV_KEY), code 143 (KEY_WAKEUP), value 1
Event: time 1714716254.780584, -------------- SYN_REPORT ------------
Event: time 1714716255.614775, type 4 (EV_MSC), code 4 (MSC_SCAN), value a4
Event: time 1714716255.614775, type 1 (EV_KEY), code 166 (KEY_STOPCD), value 1
Event: time 1714716255.614775, -------------- SYN_REPORT ------------
Event: time 1714716255.658388, type 4 (EV_MSC), code 4 (MSC_SCAN), value a4
Event: time 1714716255.658388, type 1 (EV_KEY), code 166 (KEY_STOPCD), value 0
Event: time 1714716255.658388, -------------- SYN_REPORT ------------
Event: time 1714716256.961601, type 4 (EV_MSC), code 4 (MSC_SCAN), value e3
Event: time 1714716256.961601, type 1 (EV_KEY), code 143 (KEY_WAKEUP), value 0
Event: time 1714716256.961601, -------------- SYN_REPORT ------------

On a very recent laptop, the ThinkPad P14s gen4, the media playback keys seem to be gone. Left/right arrow keys do still work as Home and End keys, though.

If you also miss your media playback control keys on your ThinkPad, then hit that Fn key and give it a go, you might get lucky!

Monitoring energy usage with smart plugs, Prometheus and Grafana

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Thu, 02 May 2024 06:00:00 +0300

This post isn’t a detailed line-by-line tutorial on how to set up each individual piece of the setup as those types of guides tend to get out of date really easily, but if you know your way around Linux and the command line, then you can definitely replicate this setup on your own.

Over the past few years I’ve been interested in learning about how much energy my computing setup and home appliances use. I’ve used a simple digital energy meter before to get instantaneous readings, but that was not ideal for monitoring how an electrical appliance consumes power over a longer time period.

My friend bought a few smart plugs from athom.tech. After getting confirmation that they don’t suck, I went ahead and ordered some myself.

The smart plugs I bought were the EU style plug V3 variant. The shipping times in EU were reasonably fast, shipping in 9 days (and that included Christmas!).

The plugs ship with Tasmota pre-flashed. The plugs come with a small paper strip explaining the steps to take to connect the plug to your Wi-Fi network, and after that you can manage the plug in your browser.

Quick start instructions that ship with the smart plug.

Updating the firmware to the latest version is easy and doable in the web GUI with just a few clicks.

The web UI is simple and very functional.

The software stack

My initial goal with these plugs was to visually monitor the power consumption of a few devices, such as my home server, router, workstation setup and the electric water heater.

The power meter data is collected to an existing Prometheus instance on my home server. If you’re not sure what Prometheus is, then think of it as a tool that periodically reads metrics from different sources, saves them to disk and allows you to later query and manipulate that information.

I run multiple instances of this tasmota-power-exporter solution on my server, one per plug, which get scraped once per second by the Prometheus instance. It is possible to also make the plugs themselves export these metrics, but I didn’t fancy building Tasmota firmware myself, yet.¹

I already had a Grafana instance running on my home server, so I reused that to show a few basic graphs for the power meter setup.

Free tech tip: make sure to change the min step setting to 1 second to get the most detailed data points on your graphs.

It's this one.

Observations and findings

So, what have I learned after running this setup for almost 4 months?

Water heater

It shouldn’t come as a surprise that the electric water heater uses up the most power. The one we have is a 30L one, just enough for a quick shower or two or for washing a large load of dishes.

Typical power usage: 4.51 kWh/day.

Minimum observed: 0.56 kWh/day, happens usually when nobody is at home

Maximum observed: 11.1 kWh/day, a lot of washing and showering happened on that day

Typical power consumption pattern of a water heater.

Home server setup

I run all my home server workloads off of a Zimaboard. One of its big selling points for me was its super low power consumption. When idling, the Zimaboard can use about 2 W, typical power usage with all my services running was about 7 W and the maximum power consumption was around 15 W.

The Zimaboard was actually using less power than my internet modem/router box (which is hot garbage by the way). The ISP-provided box used 12-14 W at all times, regardless if it was operating in router or bridge mode.

Zimaboard + modem/router box power consumption.

The ISP modem/router box is a horribly inefficient device for what it does.

At one point I temporarily switched my home server setup back to the ASRock Deskmini X300, mainly because I added some latency-sensitive workloads onto my home server setup and the Zimaboard was a bit too weak for those.

One thing that became really obvious from the power meter is that the Deskmini idle power consumption is horrible in comparison to the Zimaboard, coming in at around 15-20 W. This might be in part because my Deskmini doesn’t seem to expose any lower CPU power states than C3 while the Zimaboard exposed C-states all the way down to C10. I could not find a way to expose lower C-states in UEFI settings or the Linux kernel. Based on a quick remark in this video I believe that it should be possible to drop ASRock a message and receive a custom BIOS that enables some extra features, but that seems even more unlikely for a consumer-grade device such as this one.

ASRock Deskmini X300 + ISP box power consumption.

I’ve added a panel to my usual Prometheus node exporter Grafana view that shows the server setup power consumption alongside other metrics. Higher CPU activity is clearly visible on the power consumption graphs.

A moderate jump in CPU usage can result in a big jump in power consumption.

Voltage

The Tasmota plugs also report current voltage values. I graphed them just for fun, but noticed that there is a sort of seasonality to the values. During the usual peak power consumption hours the voltage drops across the board.

Seasonality as expressed through voltage readings.

This seems like a candidate for testing anomaly detection, as shown in this great GitLab article. I’ve used this concept at work and it works reasonably well for some metrics.

In other cases the voltage drops were caused by running appliances that use a lot of power, such as the water heater, electric kettle, electric stove or the microwave.

Water heater causing the voltage to drop slightly for all plugs.

During cooking with an electric stove, the on-off cycles were also noticeable on the voltage reading on all of the plugs.

The impact of an electric stove on the voltage.

Workstation

I have one plug that reports numbers for everything that’s connected to my home office setup: monitor, USB-C dock, monitor light bar and anything that might be charging on the table.

The power consumption of this setup varies a lot. Sometimes I do a longer home office stint. Sometimes I recharge various devices.

The typical power consumption of the whole setup while doing something on my computer is around 45-60 W, with peaks near 90-110 W. For an ultrawide monitor, USB-C dock and my laptop I think this is a pretty good result. I’ve had desktop PC-s that have used just as much power at idle, and in the olden days there were incandescent light bulbs that used up more electricity than this!

Typical power consumption of my work desk when working from home.

Typical power usage: 0.95 kWh/day.

Minimum observed: 0.07 kWh/day

Maximum observed: 1.52 kWh/day

Charging

These plugs are also great for observing the charging patterns of various devices.

A laptop or a power bank charges faster at the beginning, but the speed drops off as the battery gets more full. At one point trickle charging seems to kick in until the device is fully charged.

A power bank charges quickly in the beginning and starts slowing down once it gets close to being fully charged.

This pattern is similar on most devices that I charge. The battery on my e-bike seems to be an exception to the rule, with a slighly increasing power consumption throughout, and a faster drop at the end of the charging cycle.

E-bike battery charging pattern, from nearly empty to full.

0.5kWh of power consumed for 60km of range with the “Turbo” preset on the bike doesn’t sound bad at all. Half a cent per kilometer!

Stability

The stability of the smart plugs is usually fine, but there are frequent cases where certain plugs don’t report back in time with the statistics. This may be a Wi-Fi access point issue, but annoying nevertheless.

I’ve had to completely power cycle two plugs a few times because they dropped off from the network completely and would not come back. Again, could be the fault of my Wi-Fi AP.

Future ideas

The plugs also provide a way to turn the devices on and off over different API-s (even over HTTP!), which is something I’d like to utilize either using something like Home Assistant, or a simple script that toggles certain devices on and off based on the current electricity price.

I haven’t gotten around to this part yet, but from my research it seems that the necessary integrations exist for Home Assistant.

Closing thoughts

Overall, I’m very happy with this setup. I can get reliable measurements for all sorts of computing setups from now on, which will make judging the power efficiency of the devices I use much easier. No more guesswork!

If you enjoyed this post, then I highly recommend giving this FOSDEM 2024 talk by Florian Quèze a listen. The talk also goes into details about measuring the power consumption of various devices in the speakers’ home and observations that they’ve made.

This was pointed out to me by a fellow reader. Thank you! ↩︎

Lenovo ThinkPad P14s gen 4 (AMD): it doesn't suck under Linux

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Fri, 12 Apr 2024 06:00:00 +0300

I’ve had the opportunity to try out another new laptop at work. I’ve used a brand new laptop recently, and it was horrible.

But this time I’m pleasantly surprised.

The Lenovo ThinkPad P14s gen 4 has great specs:

CPU: AMD Ryzen 7 PRO 7840U (8 cores, 16 threads, up to 5.1 GHz)
GPU: AMD Radeon 780M (integrated)
RAM: 32GB DDR5, soldered
SSD: 1 TB NVMe
Display: 1920x1200 resolution
Two USB-C ports
Two USB-A ports
HDMI out
A full-sized Ethernet port!

I have used it for about a month at work, and so far I have not noticed anything outright broken about it. The Fedora Linux 39 installation that I copied over from the previous laptop (Dell Latitude 5411) simply works, no modifications required. The CPU is blazing fast. The GPU driver does not crash. No issues with Wi-Fi or Bluetooth so far. USB-C docks and monitors that I have just work. The webcam is also functional and has an acceptable image quality for meetings. CPU and GPU temperature data is present with lm-sensors.

The build quality is what you’d expect from a modern ThinkPad: better than consumer-grade crap, but not as solid as a ThinkPad T430. The case has a soft-feeling light gray finish to it, which will likely look horrible after a few years of use.

The battery life is not great: don’t expect to last a full 8-hour work day on battery with this thing. It’s been years since Apple has released Apple Silicon laptops with great battery life and somehow the rest of the industry still hasn’t caught up.

The fan has a really annoying high-pitched whine to it. You only hear it when pushing the CPU hard, like you’d do when compiling a legacy Java monolith, but in a quiet room it’s really annoying. I’m not even sure that the fan has to be run that hard, the AMD Ryzen CPU inside is very efficient and can regulate its power consumption based on the current temperature. You can tweak the fan curve on this laptop with a basic script.

I haven’t tried functionality that I don’t use myself, such as the fingerprint reader, so I cannot comment on that.

This laptop can also play games. I took an external SSD, installed Bazzite on it (GNOME + gamemode version), set everything up, and installed Forza Horizon 4 on it. It runs, and for an integrated GPU surprisingly well. I could push the game to run with medium-high settings on the internal 1920x1200 display at a solid 60 FPS, with very rare dips below it. That’s even better than what I got with the Steam Deck, and the CPU running near the 5 GHz mark certainly contributed to it. This makes the laptop a great candidate for a LAN party after work, unless you favor more demanding games, in which case you might want to temper your expectations a little bit.

It’s the first time that I’ve had a brand new laptop that just works on Linux. Hats off to the FOSS community for making this happen, AMD for getting their act together, and Lenovo for not completely fucking it up this time.

How to take down production with a single Helm command

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Thu, 04 Apr 2024 07:00:00 +0300

You’re Cletus Kubernetus: a software developer, and a proud Fedora Linux user.¹

You know Kubernetes, especially after the time you migrated some services to it.

Everything is calm.

Your pods are running. Your service is up. Business as usual.

You release some minor changes to production. Everything is still working. Great!

But then you receive a message from a colleague. Oh no, something has gone wrong with a particular piece of functionality!

No worries. You’re using Helm. You can roll this change back safely. You ask your colleague. “Oh yeah, helm rollback should work.”

helm rollback it is.

Cool, cool, new pod is starting up. Seems like it is indeed working.

Wait, where did all the pods go?

After a hectic troubleshooting session with the team, you redeploy the service and start investigating. A colleague uses the staging environment to do a helm rollback and it works as expected, the previous version of the service is successfully deployed.

You investigate logs. The helm rollback call worked as expected, and then it began deleting every entity related to the deployment. Pods, secrets, ingresses, everything related to the service was gone, and your name was present on each deletion.

The troubleshooting was on standby for a few days since you had no further leads and had to get other work done. But you couldn’t really move on from this issue mentally, could you?

One day you continue the investigation by opening the Helm GitHub repository, looking at the open issues and throwing in some keywords that might be relevant, such as “rollback”.

What the fuck.

It wasn’t an issue with Helm, or the way you ran it. Apparently the version of Helm packaged in Fedora Linux included a patch that introduced this issue. You then use the staging environment to reproduce the issue. Everything was gone, again, but this time in a safer environment.

You promptly run dnf remove -y helm.

After this and the xz backdoor, the idea of living in the countryside and learning beekeeping doesn’t sound that bad, does it?

Credit for that name goes to my colleagues, I wish I was that funny. ↩︎

Fairphone 5: a biased long-term review (281 days and counting)

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Wed, 06 Mar 2024 12:00:00 +0200

Changelog:

2024-03-06: 58 days, initial post
2024-07-28: 202 days, still going strong!
2024-10-15: 281 days

After months of contemplating I finally pulled the trigger and got myself a Fairphone 5. The fact that iPhone X stopped receiving major iOS updates certainly helped make that decision.

“But why? My Xiaomi/Oneplus/Samsung/other glued-together device is like so much cheaper and faster and makes better photos and the software is good after I completely format it and install a custom ROM! And LinusTechTips said that it is trash!”

Well, it’s simple: Fairphone is the one of the few companies out there that has a good track record of long-lasting software support and the phones they produce are easily repairable.

The competition is catching on regarding the software support side, but at this time those claims have not been proven yet, unlike Fairphone.

They also make an active effort to make sure that the people behind the production of their phones are treated well.

These efforts need to be supported so that we don’t end up replacing our phones every few years because the manufacturer stopped releasing software updates or your battery died and you cannot easily replace it.

This post is not sponsored, I bought this phone with my own money. The iPhone X that I used before the Fairphone 5 got passed to someone else as an upgrade so this was not a vanity purchase.

tl;dr it’s a perfectly cromulent phone that comes with some intentional trade-offs that improve the repairability and longevity while reducing the environmental impact and human suffering.

For those who saw the LTT video: Fairphone made a response video where they address most concerns and explain some design choices and trade-offs that they made with the Fairphone 5.

The mission

I really admire what Fairphone is trying to achieve.

If you’re not sure what makes Fairphone special and different from others, then give their website a read. They also held a talk at FOSDEM 2024 which goes into more technical detail, describing their journey, goals, and challenges faced when building phones and trying to support existing ones for as long as possible (Qualcomm sucks in this regard).

In short: they build sustainable phones that you can use and repair for a very long time, made by people that get paid and treated fairly.

One of the not-so-fun facts from their FOSDEM 2024 talk was that it only takes a few extra dollars per phone to make sure that the factory workers get paid a living wage. Makes you think: if that is all it takes, then why aren’t other companies doing it?

My use case for a phone

It’s important to recognize that everyone uses their phone differently. Some expect great performance, especially if they play games. Some just want a good camera. Some just want to chat with their friends and family and share dank memes.

My expectations are very modest: a phone should make adequate pictures, be reasonably fast and get regular software updates. This post is written from my perspective and will likely conflict with your views on what a phone should be, keep that in mind.

The Fairphone 5 is the first one that I actually considered buying because it resembles a modern smartphone with relatively few compromises. This is in part thanks to the rate at which smartphones innovate slowing down, both in performance and the amount of new features that people actually care about.

The cool factor

I got the transparent edition, meaning that the backside of the phone is made out of semi-transparent plastic. I can at a moments notice flip the phone over and see the components, including the battery, SIM card and the microSD card. Absolutely unpractical, still cool though.

What took me almost 2 hours on an iPhone X took me about 30 seconds on a Fairphone 5.

I wouldn’t do the “I can replace the battery in 30 seconds” party trick too much because I feel like the plastic clips holding the back cover in are quite fragile and can break if too much force is used. You can easily buy replacement covers, though.

The software

My first proper smartphone was a Samsung Galaxy Gio back in 2011. It was also my first introduction to custom ROMs and over the years I ended up running all sorts of phones with various flavours of Cyanogen and LineageOS. At one point certain banking and e-ID related apps began blocking users who decided to run custom ROMs, so I went to the Apple side for a few years.

Why this preface? Well, it’s because coming back to the Android side of things felt so refreshing and freeing.

I can:

finally run a real version of Firefox on my phone with an actual ad blocker.
- this is an absolute gamechanger for me and something I sorely missed on iOS
install any app that I want.
- there are also a lot more FOSS apps available.
rearrange shortcuts on my home screen as I want to.
- or hell, run a different launcher altogether
seamlessly back up photos from my phone to PC using Syncthing!
work with actual files!
cram in a 512GB microSD card.

The Fairphone 5 ships with Android 13, with Android 14 upgrade currently scheduled to summer 2024. The base OS is quite clean, the only preinstalled app (other than the Google stuff) is the “My Fairphone” app that shows information about your device. That same app can also be used to verify that the functionality on your phone works, and yes, the speaker testing ones get quite loud.

If you decide that the stock Android experience isn’t for you, then know that this phone can easily run alternative operating systems. At FOSDEM 2024 almost every alternative OS booth had one or more Fairphone 5 running, including Ubuntu Touch and CalyxOS. It’s clear that the FOSS community has embraced the Fairphone.

I was a bit sad to see that a lot of open source apps that I used to run or that come up in searches are unsupported or out of date. In some cases modern forks exist, but overall the picture looks very discouraging, especially if I was new to the scene.

Performance

The performance isn’t great, but it’s good for most activities. I can feel it being sluggish from time to time or with certain animations.

This is due to an intentional trade-off that Fairphone made: to be able to properly support this device for a very long time, they had to go with a Qualcomm SoC that gets long-term support, and that ruled out the ones that provide the best performance. They ended up putting in a chip that was designed for IoT applications, funnily enough. If having a slightly slower phone is what is needed to run this phone for 5+ years, then it’s a worthwhile trade-off in my view.

Fairphone highlighted the OEM support issue in their FOSDEM 2024 talk in case you’re curious about why this trade-off needed to be made. My opinionated tl;dw: we have to get OEM-s to either open source the proprietary parts or hit them with some regulations and enforcement so that they do support the chips they produce for longer.

Battery life

The battery life is what you would expect from a typical smartphone: 1-2 days of normal use, but I once stretched it to almost 4 days during a time when I used my phone very little. Do note that this is highly dependent on your usage patterns, if you do constant video calls then you can’t expect the battery to last much.

If you want to extend the longevity of your battery, then there is an option to limit charging to 80% in battery settings. I’ve recently enabled it because it doesn’t impact my use of the phone much. Let’s see how it fares in a few years.

Size and feeling

This phone is big. Really big. Biggest one I’ve had yet.

I actually prefer smaller phones, something like the iPhone 13 mini size would be ideal (Apple stopped making smaller phones that aren’t iPhone 7 reskins after that so that’s not an option any longer). When I was in a store that sold iPhones, I compared my phone to the latest iPhone 15 offerings and was surprised to see that my phone was about the same size as an iPhone 15 Plus. Hopefully we’ll see smaller Fairphones in the future, all while not introducing too many compromises.

Some people are surprised at the thickness of the Fairphone 5. Yes, it’s thicker than most of the competitors, but note that with other phones you’ll likely buy a separate case for the phone, so you’ll end up with something just as thick anyway. At least with Fairphone 5 you don’t have a glass backside that easily shatters if you drop it.

Camera

When I bought the phone, I expected only one thing from the Fairphone 5: its camera should be better than the one on the iPhone X. GSMArena comparisons and reviews by others looked promising

The camera on the Fairphone is acceptable. I don’t have proper photography education, but in most cases the Fairphone makes OK pictures. In low light situations it seems to be inferior to flagship phones from other companies, which may often end up with smudged pictures. At least it has a low light mode, something that my previous phone (iPhone X) lacked. The colors look funky sometimes and are oversaturated.

This is a selection of photos made with the Fairphone 5. Thumbnails are compressed to avoid unnecessary bandwidth. Click on the image to view the original, full-sized version.

Click on the image to see the full version.

The headphone jack

I feel like I’m obliged to mention that this Fairphone doesn’t have a headphone jack. As someone who spent a few years on iPhones, this one doesn’t bother me too much.

USB-C port

I plugged it into an USB-C dock and it actually output to an external display in portrait mode. Not sure how practical, but really cool.

Oh, and the USB-C port is easily replaceable.

Price

If you only focus on the spec sheet and compare the Fairphone 5 to the competition, then it will strike you as an expensive phone, coming in at 699 EUR.

It is expensive, but that’s a result of a lot of small things. Fairphone tries to do their best to source the raw materials from ethical sources and that narrows down your selection by a lot. Add to that living wages to the factory workers, more repairable components, declining to add sponsored apps/ads into the OS and that’s the result.

If you take the price and divide it by the number of years that you expect to get out of this phone (5-10 years), then the price is likely much more comparable to the competition.

If a higher purchase price results in less human suffering and a longer lifetime of the device, then so be it.

Schematics

They provide schematics for the phone (direct link here).

My hours of watching Louis Rossmann do board level repairs have taught me that this is a very good thing.

It’s not perfect

Every phone has issues.

iPhone 15 had overheating problems.

That one iPhone SE 2020 that I ran for a few months had a hard to reproduce system board issue that made the phone inoperable for 5+ minutes in random situations.

Here are some issues that I’ve noticed while using the Fairphone 5

SIM card

After I set up the phone, the SIM card would stop functioning properly after a few hours of use. Reseating the SIM card did not work, and I was quite frustrated because I was expecting to send the phone back just after I set it all up. Luckily it seem to be a known issue and there is an official support article that covers it. My SIM card is much older than the specified 2 years, but I couldn’t be arsed with replacing it yet so I used pure ethanol to clean the contacts on the SIM card, popped it back in and it has worked with absolutely no issues ever since. I’m glad that I got this easily fixed, but for a normal customer this would be a very frustrating experience so hopefully there is something that Fairphone can do to fix and/or avoid this in the future.

The display

The display of the Fairphone 5 supports 90 Hz refresh rate, but ships with it set to 60 Hz. I’ve heard that most other phones do something similar with the likely reason being the negative impact that high refresh rate displays have on the battery life. At the time of writing, however, there is a known issue with the 90 Hz mode which can actually result in more lag and a worse user experience. I’ve set the display back to 60 Hz for this reason. Luckily this is something that can probably be fixed with a software update.

The automatic brightness setting is quite bad, especially when compared to any iPhone. The brightness jumps around a lot in darker conditions and it’s very irritating when driving because random street lights and cars seem to trigger the sensor often. Again, probably something that can be fixed in software.

Software stability

Overall the software is stable, but there was this one time when the phone crashed and rebooted unexpectedly. It was triggered while playing music and opening a Snapchat notification. Nothing serious and has only happened once in an almost two-month period, but still noteworthy.

The Google crapware

This isn’t really a Fairphone issue, but more of an Android gripe. I still have to mention it though. The Android 13 experience has started resembling the Windows experience. I had to toggle off countless “features” to turn off most of the Google crapware that ships with the phone, and even then I still have to stare at the unremovable Google Search bar at the home screen. That thing takes up so much useful screen real estate and is absolutely useless to me. It used to be removable according to some discussions I found online, but I guess Google noticed that feedback and made it a permanent fixture on the home screen with an update. Absolutely disgusting. You can work around this by using a launcher that’s not affected by Google. Kvaesitso is a FOSS one that seems to work well enough for me.

Final thoughts

Fairphone 5 is what you get when you focus on delivering a phone with long-term support and repairability in mind, all while doing your best to treat your workers well. This does mean that you’ll be making some trade-offs.

You won’t get the best performance, the best camera, or the smallest size. What you get in return is the ease of repairability, years of software updates with a proven track record, and the knowledge that at least those workers who assembled your phone get paid a living wage.

If you don’t expect much from a phone and are willing to pay more for a phone that you can use for a lot longer, then the Fairphone 5 is a good option.

Can I make it to 2031 with this phone? I’ll definitely give it a try.

2024-07-28 update: 202 days

My Fairphone 5 is still going strong. Here are some notes after 202 days of use.

Android 14

The promised Android 14 update is here and on time. The upgrade took a while to install but was boring, in a positive way. Nothing I relied on broke and the interface looks more-or-less the same.

The only aspect I’m not happy about with the Android 14 update is the USB webcam mode feature not being present. It’s possible that it’s an optional feature of Android 14, and none of the Fairphone 5 materials indicate it being included.

Video playback issues

I’ve noticed some oddities regarding video playback. The video playback can be choppy on higher resolutions, especially when I run them at 2x speed. Lower resolutions are fine. Sometimes the issues only occur with certain encodings, like H.265, or audio tracks where I can only seem to hear certain audio channels.

Camera in action

The camera doesn’t seem to be the best option for point-and-shoot situations where there is a lot of movement or where the lighting conditions are bad. I’ve taken my fair share of blurry photos as a result. The included microphone also seems lacking in real life use, especially when I accidentally cover them with my fingers.

If I could pick one improvement for the next Fairphone, then it will have to be the camera. A lot of people care about camera quality and might rule out getting a Fairphone as a result.

At least with the pro mode you have a lot of control over the camera, which is great for more stationary shots.

Durability

I have nothing bad to say about the durability here. I have dropped it a few times already and as a result I have two small dings on the aluminium frame.

The backside has also held firm. After one fall a couple of clips came loose, but I could easily pop them back in.

Various oils and finger grease have an interesting effect with the transparent case, resulting in the see-through effect being better. The grip has gotten better with time and negates any need for a separate case. The Fairphone 5 is still a big phone and not that easy to handle one-handed, but the grippy backside makes up for part of it.

The display glass has a few scratches, but nothing too bad. I can’t seem to find any serious scratches on the backside.

Battery

The battery has held up fine. I turned off the battery saving feature because of how simple it is to replace the battery, enabled fast charging because it’s really handy, and stopped showing the battery percentage on the notification bar to reduce battery anxiety.

5G

I switched mobile service providers during this time and could finally test the 5G capabilities of this phone. Since I went with the cheapest option at the time, then the signal strength will not be as good as with other ones, but at least I know it works in Estonia.

Maximum speeds I’ve seen are around 250 Mbit/s. Could be limited by Fairphone 5, or the service provider (more likely option).

Display brightness

At some point I got an update that seems to have improved the automatic brightness feature. It’s not as jarring as before, especially at low light levels. I’m not sure which update fixed it, but I haven’t noticed any annoyances with it for a long time.

2024-10-15 update: 281 days

I have one gripe with the software support on Fairphone 5.

The security updates are lagging behind way too much.

They’ve always come with a small delay of a few weeks, but it took over a month to get the 5th of September security patches. These are supposed to be monthly.

I finally received the 5th of September security patches today, 15th of October. There’s already a new security bulletin out with the October patches, which I don’t have on my phone.

It’s better than no updates at all, but this does not bode well for future long-term software support.

Next update coming whenever!

My very first career day

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Thu, 22 Feb 2024 08:00:00 +0200

This post is a short overview of my experience at a career day in Valga, Estonia, hosted with the help of GreenDice.

Kind of felt like a teacher at one point.

I’ve never spoken at a career day before nor attended one as a student, which is why I instantly agreed to going to one when GreenDice reached out to me.

Why? I never had opportunities like that as a student myself, which is why I try to do my part in making sure that future generations have them.

After agreeing to speak at the event, I had to come up with an idea for a short workshop. Coming up with an idea was surprisingly tricky due to the set of requirements and assumptions that I could rely on. What I could rely on were Windows 10 PC-s that had a browser, reasonably fast internet access and 45 minutes per group. The target audience would be students from 9th grade with all sorts of backgrounds, so the workshop would have to be accessible to everyone.

I also asked my colleagues for ideas and got useful feedback: whatever you do, make sure that the students have the chance to find and fix problems themselves.

The workshop ended up as a simple website hosted on CodePen, an in-browser HTML/CSS/JS editor. It runs well in a browser and allows students to see the results of their changes visually. The web is also something most of us rely on daily, so working on something relatable is likely a good idea. Hopefully this exercise would also help demystify how websites work.

Software development solves problems using software as the tool, so the idea was to have students take an existing but slightly broken software solution and fix it together. I would give a 60 second long intro that was mainly about what HTML, CSS and JS do on a webpage and how to recover from any issues, and then we would get going.

Once everyone had opened up the editor, I asked the students what they thought about the webpage. Was it working correctly? Did it look good? Based on the feedback we’d tackle these issues one by one. The solutions also had to come from the students, so in a way I was there to just listen and try out fixes that the students came up with. At the end of the session we’d have a solution that functioned properly and looked good.

The issues I planted into the website were as follows:

the background image might have been cute, but not suitable for the site as it made reading the text difficult
the previous developer left a visible comment in the middle of the page
the page title is too small
there is a photo of two cute cats present when you load the page
- students convinced me that it was a feature, so we kept that in
when you hover over the button, it goes blank
images of the food options are upside down
one image would not load properly
one food choice would have the wrong image
one food choice was missing an image

The masterpiece that the students started working on.

This was just the right amount of work to fit into 45 minutes and in general the students were following along nicely.

If I had to do something similar the next time, I’d definitely have someone else assisting at the other end of the classroom. Although I tried to encourage everyone to make mistakes and refresh the page in case things went really wrong, some students were a bit hesitant with making changes to the website.

I’m also thinking about doing something else with the students in the future if there are enough resources and time. Perhaps something related to game development, robotics or AR/VR would probably be pretty cool to show off.

If you have organized career days about IT yourself and have ideas to share, then please do reach out to me, I’d be happy to hear about your ideas and experiences!

Oops, I published my drafts!

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Tue, 13 Feb 2024 08:00:00 +0200

Those of you who follow my blog using the RSS feed might have seen that new, incomplete posts popped up around the time I published my FOSDEM 2024 post.

Oops.

I recently tried looking for an alternative to writing blog posts in IntelliJ and out of all the options I stuck to MarkText. Well, it does things a bit differently and what was once a front-matter containing all sorts of metadata, including the one that prevents publishing a post if it is a draft, turned into a code block, and Hugo happily served that up.

``` vs ---, there’s a difference!

Anyway, sorry about that, and thanks to those who noticed and let me know.

Free tech tip: if you have a website with an RSS feed, do follow it yourself, you’ll catch these sorts of issues very quickly!

FOSDEM 2024: my experience, some notes and tech tips

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Mon, 12 Feb 2024 08:00:00 +0200

I finally went to FOSDEM. I’m sleep-deprived, completely exhausted, but incredibly excited about the whole experience.

FOSDEM in one picture.

I’ve split this post into three separate sections.

overall notes on the conference and the city
- less technical, but sheds light on the FOSDEM experience
the hallway track and stands
- the thing you probably came to FOSDEM for
my notes on the sessions I attended
- plus sessions that I plan to watch in the future

The conference

FOSDEM (Free and Open source Software Developers’ European Meeting) is a huge non-profit, volunteer-organized conference that’s free of charge to attend thanks to sponsors and donations. It took place in Brussels, Belgium at the ULB campus.

I heard about this conference from a friend who went there years ago, and after hearing about the experience and the content there I knew I had to be there at least once in my life. But then a pandemic happened, so it was sort of in the backlog for a while.

My current employer provides a perk that allows developers to attend technical conferences. After the FOSDEM 2024 schedule was available I put together an initial set of talks to attend, presented it to my employer, and got a green light to attend (and travelling/accommodation paid for).

The city

To make sure that I’m well-rested, I arrived the night before FOSDEM with my friend. After dropping my stuff off at the hotel, we went to get something to eat and continued at the Delirium Beer Garden to sample some of the finest Belgian beers. It was very crowded, but after the first beers and some walking around the many floors we found a place to chill.

We ended up with sampling a meter of beer which was a selection of ten small beers, laid out on a plank. It was a nice way to try different tastes and honestly all of them were either good or great. Cherry, raspberry and the green cactus option were the sweetest, like some good lemonade that has the potential to cause hangovers.

Meter of beer. It didn't come pre-sampled by someone else, don't worry.

This one was called "Cactus". Didn't taste like one, but certainly looked the part and was absolutely delicious.

Free tech tip: don’t underestimate the strength of Belgian beer. You won’t taste it, but that 8-9% alcohol per volume really hits in the morning.

Brussels is allegedly very nice, however I’m not a big fan of the way trash is handled, there are bags everywhere and some of them are broken, leaving small pieces of trash everywhere. You could also smell the lack of public restrooms when walking down the streets in the city center.

There are some glimpses of good bicycle infrastructure, but they often don’t connect well or are simply lines on the ground which doesn’t protect that well from cars. Based on the honks and the number of times they did not stop at crosswalks, I can only assume that drivers in Brussels are both very aggressive and very incompetent at driving. In a way it felt like home, especially since my hometown of Tallinn also uses red paint on some bicycle paths.

The dos and don’ts of FOSDEM

I went to the ULB campus using a tram. If you’re not sure which one to take, then take the one with the most developers headed towards it. Knowing when to get off is also easy as you can simply follow the crowd.

To FOSDEM!

I arrived roughly 15 minutes before the keynote session. At that time there was plenty of room in the auditorium. By the time the keynote started there were no seats available, so you might want to do the same and arrive a bit early.

15 minutes before the keynote. The place was packed at the start of the keynote, with more people still coming in.

There were a lot of people attending FOSDEM. Keep a map handy, because your next session might take place in a room that might be tricky to find, especially as a first-timer. The FOSDEM Companion app is great for that, click on the session you’re interested in and the room name, and you’ll have a good idea on where it’s taking place.

This FOSDEM was the 24th FOSDEM to take place, which is important because the 25th one is happening next year. If you’ve been to any previous ones and are attending next year, then bring the oldest FOSDEM shirt with you, there will be a special photoshoot taking place.

If you’re going to attend FOSDEM in-person, then definitely read the FOSDEM survival guide. All the points there are still relevant and helped me out tremendously during my trip.

In short:

take an earlier bus/tram to get to FOSDEM to avoid crowds and get there on time
if you want to guarantee getting a seat in a session, consider joining an earlier session in the same room
- you might also learn something about an area you had no clue about, which is a great bonus!
don’t underestimate the strength of Belgian beer
- trust me on that one
food lines can get very long, bring some light snacks and plenty of water with you
- some places had water bottle refilling stations, such as the U building, but I had a hard time finding one in others
- there was plenty of Club-Mate for sale and based on the number of empty bottles it was a crowd favourite

There were 879 events taking place at FOSDEM 2024. That is a lot of talks to go through. My recommendation is to take an evening at least a week from the event and go through the list of talks starting with main tracks and lightning talks, and following it up with devrooms that align with your interests.

If you do it in the FOSDEM Companion app, then you can bookmark them and view your initial schedule. The app will show you where there are overlapping sessions by coloring the start/end times red. The app can also show you if a room is too full to attend. Don’t forget that switching rooms might take about 15 minutes in some cases due to the size of the campus, so leave some room for walking in your schedule.

In my experience, the most enjoyable FOSDEM experience is one where you attend at most 2-3 tracks during the day and leave plenty of time for eating, resting and visiting all the stands. It’s going to be exhausting either way, so you might as well make the most of the FOSDEM live experience. Talks are recorded and will be published soon after the event so you’re not going to miss much if you take things easy (unless there are issues capturing the session).

Here are a few simple guidelines that I recommend following so that listening to talks in FOSDEM live is a great experience for both you and those around you:

try to get to your next talk about 15 minutes before it starts, that allows you to queue up early and not miss your session
don’t talk or whisper during the talk and the Q&A session with the speakers
- there were so many cases where the speaker could not even hear the question being asked, which is disrespectful to the speaker and the audience
if you really need to leave early, then be as quiet as possible
- the old desks at the seats make a lot of noise if you carelessly put them back up-right
if you really didn’t have time for a snack, then don’t bring something that makes a lot of noise or smells strongly to the auditorium

If you want to get a FOSDEM t-shirt or hoodie in your favourite size, then the first half of the first day is the ideal opportunity for that. At FOSDEM 2024 the t-shirts cost 25 EUR, hoodies cost 50 EUR.

Most payments you’ll make will likely be with contactless payments via your card or phone, but definitely bring some cash with you in smaller denominations as there were a few cases at the start of the conference where the payment systems were not working properly, likely due to network issues. Somewhere between 100-200 EUR will likely cover anything you’d need at the conference itself.

With the food trucks I made an observation: the lines were longer for the more expensive and higher quality food options. The cheaper ones are okay and are generally quite fast to get. The more “premium” trucks were not much slower though so waiting in line might pay off.

During the keynote it was emphasized that some of the attendees will get sick as a result of this conference. They call it the FOSDEM flu. I guess it’s a thing that happens at many big conferences, but in any case if you do visit FOSDEM then please don’t do it while sick. If you fall sick at the event, then rest at the hotel and watch it on a livestream instead. There’s always next year!

One pattern I noticed is that quite a few talks were used as a great marketing tool for highlighting issues that the speaker would like some help with solving. What better way to improve FOSS than to involve people who might not even be aware of the issues but have the technical know-how to help?

The stands (the hallway track)

One of the highlights of the FOSDEM experience are the various stands in different buildings in the campus. You’ll find all sorts of different projects being represented, ranging from big, established ones (Fedora, Debian, Nextcloud, PostgreSQL etc.) all the way down to small projects and organizations (postmarketOS, Ubuntu Touch, CalyxOS and many more).

Most stands offer free merch, such as stickers. A lot of stands sell merch, most accept card payments, some accept payments via online credit card payments, and a few take cash only.

I visited most booths at the event and spoke briefly with many of the people present there. If you’re just curious about what’s on display and have no background knowledge, then don’t be afraid to ask questions, no matter how simple you think they are!

Here are my notes on the stands that I found the most interesting

GNU Radio

This stand had a simple and effective demo of a device monitoring the 2.4 GHz spectrum, which is also where older Wi-Fi standards operate in. The person hosting the booth was happy to explain what was going on, even bringing out details such as signals bouncing around the room and how they show up on the spectrum visualizer.

Turns out that you can achieve this by putting together two blocks in a visual editor in GNU Radio, and this is why it’s on my todo list now.

There’s also a video clip of the visualization running.

GNU Radio showcasing its visualization prowess.

Minetest

Minetest is an open source game engine and had a booth present at the event. They had the game running on a laptop. A Steam Deck with the official dock was also present. I’ve previously heard about this project many years ago, at that time it was a simple Minecraft-like game but a bit too basic in my opinion. The demo at the booth reminded me of something more polished and yet familiar enough to remind me of Minecraft 1.8 Beta days.

Turns out that over the years the game engine has seen a lot of development and there are many creators who have mades games based on this engine. I was also delighted and surprised to hear that Minetest has been used in educational projects.

Minetest running on a laptop, with a booklet showcasing some games made on it in front.

This was a nice and colorful corner of the event, and now I’m interested in giving Minetest a try again.

Ubuntu Touch

Ah, Ubuntu Touch. I believe the first time I heard about Ubuntu Touch was when they introduced the convergence idea. Your phone is the only computer you have, use it as a phone when out and about, and connect it to a dock when at home to get a desktop experience.

The booth had a wide variety of devices present, including Fairphone 4 and 5 (this isn’t the first time you’ll see these models mentioned). At the booth were people who actually daily drive Ubuntu Touch, and one person I talked to expressed their interest in porting Ubuntu Touch to a Samsung foldable phone.

Ubuntu Touch isn’t probably something I’d run full-time, but perhaps it’s time to give it a go again. My Nexus 5 still has the boot logo that Ubuntu Touch installer creates, so that will serve as a reminder for me to check it out.

Ubuntu Touch mascot looking over the phones.

SailfishOS

I knew of SailfishOS, but had never tried a device out with that OS. At FOSDEM I finally had the chance to do it, and out of all the custom OS options that aren’t Android-based, this one was the one with the smoothest and most polished experience. When speaking to the person hosting the booth, I also learned that it is possible to even run Android apps on it, although after doing some research on their website it seems like it’s a feature that you need to pay for.

The phones laid out on the booth were various Sony Xperia devices. I’ve not used or seen those devices, but they felt really great in hand and were super small compared to modern phones. Seems like these models are also the ones that have got the most polish and support, based on the materials on the website and my two minutes of fiddling with them.

Out of all the non-Android alternative operating systems, this one has the most potential to be a daily driver. The person at the booth uses theirs as a daily driver so there must be potential on it. The corporate backing is likely what allows the OS to feel usable and hopefully that will be enough to keep this project alive.

Various Sony Xperia phones running SailfishOS.

Droidian and postmarketOS

There were multiple booths where you could try out various versions of Linux-based phones that aren’t Android.

The phones were all sorts of older and newer devices that originally shipped with Android. The experience and support was a combination of the hardware and the state of the mobile Linux distribution and UI. Some were quite okay, some were rough, and some were just slow due to the age of the phone itself.

Plain old Linux? On my phone? It's more likely than you think!

I really appreciate these types of efforts to give new life to old devices. Most Android devices lose software support after a few years, but with projects that basically put Linux on a phone you can still find ways to make use of the devices. They might not be stable enough yet to be considered a reliable daily driver as a smartphone, but you can do all sorts of cool things with them, like running web servers or hosting encrypted remote backups of critical data.

One of the booths also had a Pinephone in a keyboard dock, which was neat as I’ve never seen a Pinephone in real life, nor had the opportunity to try one with a keyboard. It wasn’t ergonomic trying to do it with two hands, but maybe I was holding it wrong.

It was interesting to see GNOME or KDE-based user interfaces on phones. The gestures and navigating the UI was a bit rough since there were slight differences on each OS. Still cool though!

Oh, and there was a visitor who had some cool gear with them. Other than a GPD portable computer, they also had a mechanical keyboard with a wide display attached to it. They tried connecting it to a phone that was on display at the booth, and it recognized it as a second display on the first try. The model of the keyboard is Ficihp K2 in case you’re interested in similar gear.

How to take notes in the coolest way possible.

CalyxOS

I had never heard of this Android-based OS before, but there they were, showcasing it on a freaking Fairphone 5 (and also some other phones)!

The people at the booth were very friendly and helpful, answering any of the questions I had about app compatibility and the OS itself.

The project seems to be backed by an US-based non-profit, the Calyx Institute.

I’m a bit wary about using this on my daily driver Fairphone 5 running Android 13 as I just got it, but if I had a spare one for testing I’d definitely check out how well it fares as a privacy-focused OS. The potential is there and hopefully CalyxOS will have a chance to prove itself as a long-term and stable project.

Totally forgot to take a photo of phones running CalyxOS as I was too busy giving the OS a try, but here's a hat!

Side note: I’m planning on covering my Fairphone 5 experience as a separate post in the future, but one takeaway from FOSDEM is that I’ve accidentally picked a tinkerer’s dream phone.

Automotive Grade Linux

Like with many projects, I had never heard about this one before FOSDEM.

This booth was displaying various interfaces designed for cars. There was a demo vehicle dashboard which was simulating a moving car. Next to it were two bigger displays that had infotainment and control panel functionality with all sorts of buttons on it.

Yup, that's RISC-V!

Between these two setups was a small control board with 9 keys and two dials. Some of the buttons popped up warning lights on the digital dashboard, such as passenger airbag failure. Other buttons had various stickers on them: HTML5, Qt and Flutter. Turns out that the intotainment display demo was built on top of containers and pressing a key would stop the running one and start up a different one based on whichever button you clicked.

Unlike with my actual car, with this one you could turn the warning lights off without having to spend a lot of money!

One of the infotainment displays was running off of a Raspberry Pi, which was cool. The rest of the displays were running off of a stack of various bits of hardware.

Dashboard and infotainment setup, with the container switching buttons between them.

Based on the marketing materials at the booth it seems like automotive grade Linux is also collaborating with some big names in the auto industry, which is neat!

Pine64

I’ve occasionally checked out Pine64 progress and devices online, and at FOSDEM I finally got around to play around with some of them.

So much tech to play with!

I first checked out the smartwatch named PineTime. It comes in multiple versions, one for daily use (the sealed model), and one which is bulkier but easier to open and flash in case you mess up your firmware. I’m currently using a Casio F105W and haven’t been a big fan of smartwatches, but the hardware on the sealed model felt premium and something I’d pay good money for. The functionality was basic, but it had everything you’d want from a basic watch, plus a heart rate monitor and even a Pong-like game. I’m seriously considering getting one now.

Do you know what time it is?

The PineTab-V, a RISC-V based tablet, was also present and running KDE Plasma. The experience was unstable at the time as I had the settings app crash on me once, but that’s likely down to driver support not being quite there yet. That’s OK, since they advertise the device as an experimental device right now.

Something I wasn’t even aware of was the PineNote, a Linux tablet with an e-ink display. It was running an actual GNOME desktop on it so the interface was instantly familiar to me. As a test case I opened up my blog on it and it worked just fine. Scrolling the page to read the blog was a bit clunky due to the low refresh rate that an e-ink display has, that’s just a limitation of the technology. Here’s a clip demonstrating it.

Shameless plug, I know.

While I was checking out the hardware, there was also a demonstration going on of the capabilities of the Pinecil, a small soldering iron that’s really cool and gets quite hot. I’m a bit surprised that it didn’t trigger the smoke alarm that was a few meters away from the booth.

Bytenight 2024 @ Hackerspace Brussels (HSBXL)

One of the many FOSDEM fringe events was a party hosted at Hackerspace Brussels, a hackerspace located in an old industrial building. The event had DJ-s playing music, a t-shirt making booth, a fancy projector setup, a room with SuperTuxKart on a big screen and a small Raspberry Pi-based mini arcade box, and the room where I assume work gets done since it had all the electronics equipment and workspaces.

The projector setup used in the party area was simple but elegant. It was positioned in a way that allowed various surfaces to reflect the video at different levels, making it all feel three-dimensional.

I loved how one of the hackerspace members was excited to show off the insides of the mini arcade cabinet and explain the build process of it, including the design of the physical components. It’s cool seeing people get enthusiastic about something they’ve built with their own hands.

At one point me and my friend ended up speaking to a group of people, one of whom was showcasing their Tamagochi and offering it to others to interact with. I learned from them that there exists a self-hostable Snapchat-like app called Piqchat. My friend gave it a go on their instance and after fiddling with the sign-up process (we were at least two beers in at this point) he got it working.

We arrived around 20:45ish at the location, and at that point the dancefloor was a bit empty, but after 23:00ish it looked like a proper party. I wasn’t in the mood to party much because there was a whole second day of FOSDEM left and I was already exhausted.

Interesting location, interesting vibe and probably a great place for raves.

And the rest…

Here are some quick notes on other booths that I need to throw out there so that I don’t end up forgetting them.

KDE: other than stickers or merch, they had small handmade plushies of the project mascot Konqi. I’m still a bit sad that I didn’t get one at the first opportunity because they were all sold out during the first day.

Loved the fact that they were next to the GNOME booth. :)

PostgreSQL: I got to play in a game of Kahoot and while I crashed and burned in the final standings, I learned a lot about the PostgreSQL project details in the process. They also had cute hand-made (in Germany of all places!) elephant plushes in PostgreSQL-blue so of course I bought one. Fun fact: the ones sold there were version 2 of the plush which has the nice feature of not falling over on a flat surface. See, making PostgreSQL stable isn’t that hard!

It's clear that I don't know how to take selfies, and the poor low-light performance of the Fairphone 5 camera certainly didn't help.

Firefox cookies: Mozilla handed out free cookies from a pink food truck. No cookie banner included, but I guess it’s a nice way to remind everyone that Mozilla Firefox still exists. It blocks ads on desktop PC-s and Android, use it or lose it!

First-party cookies!

ISRG and LetsEncrypt: I went to FOSDEM and all I got was this free cert. The person at the booth was also handing them out in large quantities, just like Let’s Encrypt servers.

Brilliant!

Feeel and wger: I know that it doesn’t look like it (yet) but I do go to the gym regularly and my current workflow involves keeping notes in a Google Sheets file. I never knew that a FOSS solution exists for tracking my workouts. I need to try it.

FOSSASIA: these lovely people were selling LED badges that you can program easily with an Android app called Badge Magic. This was 100% nerd bait and ranked high on the “I don’t need it but it’s so cool that I need to get one”. Simple. Elegant. Awesome! Oh, and due to the transfers being Bluetooth Low-Energy powered, I think someone accidentally sent their badge contents to my device. Whoops. Here’s mine in action!

A lot of people wore these over the campus, including myself.

Jenkins: they had Roundernetes, a Kubernetes cluster made of lots of Raspberry Pi-s, in a round frame. I’d love to build something similar one day, just because it’s neat!

They certainly know how to nerd-snipe someone!

Checkmk: it was the first time I heard about this monitoring solution, but after hearing about their background and their approach to building Checkmk that involved a lot of cooperation with customers, I’ll need to give this a proper look. Oh, and they had an air quality sensor hooked up to the booth, the results were OK when I visited the booth.

The talks

You’re probably coming to FOSDEM for the vibe, the people and all the fun stands, but there are also hundreds of talks that you can choose to attend.

Hey, it's that guy from curl!

There were many talks covering interesting topics, but I could only attend a relatively small number of those live. There’s simply too much interesting content out there.

Here are my notes on the ones I attended live.

“Where the !?*! are the packets going?”

This was one of those sessions I decided to sit in on to catch one I really wanted to attend and it was a good decision as the room was packed.

This talk gave a quick overview of a modified version of traceroute by Catchpoint and the enhancements they made to it, involving QUIC support, functionality that bypasses firewalls and some shenanigans in Linux setups on Azure (the Microsoft cloud services platform).

techtipsy rating: 6/10, it was alright.

Broom not included: curling the modern way

This talk was originally meant for a 40 minute session but due to limitations of the FOSDEM schedule, 20 minutes was all he got.

This session was a rapid-fire presentation of curl development and all sorts of things that you could do with it. I’ve used curl so far for simple requests, but after this session I know that you can do all sorts of shenanigans, including parallel transfers and a lot of JSON mangling. JSON, JSON, JSON!

I also learned about trurl which is handy for working with URL-s. Parse them, modify them, do whatever you desire without re-implementing all that complexity yourself. Neat!

techtipsy rating: 10/10, short, amusing and yet useful listen about curl, looking forward to a longer version in the future!

Improving IPv6-only experience on Linux

I initially didn’t plan on sitting in for this one, but since one of the WiFi networks hosted by FOSDEM itself was IPv6 only and I’m plagued by my ISP only working over IPv4 (old-man-yells-at-Elisa.jpg), I decided to listen to this one.

The speaker gave an overview of the situation with IPv6, which operating systems handle it well and which do not, and how you can improve the IPv6 compatibility on your own machine.

In short: Android works, Linux/Windows mostly work, but all the Internet of Things garbage (smart everything) is something you should simply avoid at all costs since those hastily thrown together pieces of crap only work over IPv4.

The rest of the session covered possible solutions for making sure that IPv4 and IPv6 both work nicely on your machine. Turns out that there really isn’t one perfect solution for this problem, especially since you can’t just ignore that part of the Internet that doesn’t support IPv6, yet.

This talk was also a way to ask for help from the FOSS community to help come up with a solution. If you’re into this topic, then go ahead and reach out!

techtipsy rating: 8/10, learned a lot about IPv6 support and what’s going on in this area.

An engineer’s guide to Linux Kernel upgrades

Great presentation that included useful recommendations for rolling out Linux kernel upgrades and why you don’t want to wait a long time to upgrade, backed up with data and nice visualisations, and memes.

Although this talk was about Linux kernel upgrades, the main idea is very much applicable in software development as well. If you don’t release software as often as possible but keep making changes at the same pace, then your change delta (or the number of pending changes) increases, and so does the risk of something going wrong, either by releasing a buggy change that’s now harder to troubleshoot due to the large number of changes deployed, or not releasing new versions of software with security patches.

This talk also explained the Linux kernel development workflow, the meaning behind version numbers (it’s not semantic versioning!), and how security patches get merged into “long-term support” versions.

In short: update often, deploy often, measure things.

techtipsy rating: 9/10, great listening for software developers and Linux newbies alike.

Soft Reboot: keep your containers running while your image-based Linux host gets updated

My day job uses Kubernetes for running our services. Have some pods (containers) running on a host that you need to update? No problem, Kubernetes starts new pods on another host, directs traffic to those and you can now update that host.

This talk shows another way of updating your host operating system and apparently this functionality is already present on most modern Linux systems that run systemd.

In short, if your containers meet certain criteria, then you can utilize this soft reboot functionality to update and reboot your base system without any noticeable interruptions to your containers. Your kernel will still be the same version, so keep that in mind. And yes, the irony of this solution is not lost to me after listening to the talk that emphasized the importance of updating your kernel as often as possible.

Although the customizations behind it are quite simple if you follow the guidelines, this talk still felt like magic to me. I guess that only means that I should look into it further.

techtipsy rating: 9/10, interesting concept that has real world usages at big tech companies and yet felt like something I could understand with my small brain.

Juggling with UIDs and GIDs: rootless container deployment with Ansible

Short but sweet talk about how the speaker manages their home server, including a neat workaround that most people who have worked with containers have faced: permission issues between containers and the host.

I’m actually struggling with a similar issue right now in my home server setup and I’ll definitely have to reference this talk once I get around to trying to fix it.

It was kind of odd how relatable this talk was to me: it’s related to self-hosting, the speaker has a home server running, they mentioned linuxserver.io images, and they care about running containers rootless. Even my friend made a note about the similarities.

techtipsy rating: 10/10, incredibly relatable and useful to me, a self-hosting enthusiast.

Orchestrating eBPF Applications in Kubernetes and Fedora

This was another session that I decided to attend because hey, I’ve already got a good seat and the session after that is in the same room.

I had little knowledge about eBPF before or what it does, but after listening to this one I have a rough idea about it. In short: small programs in the kernel that are used for networking, security and monitoring purposes.

This talk shed some light on deploying eBPF applications with bpfman and nuances around it.

techtipsy rating: 6/10, flew a bit over my head due to my knowledge gaps in eBPF, but it did raise my awareness about this area.

Lift and shift: Modernising a legacy LAMP application with systemd-nspawn

During my self-hosting adventure I’ve used various setups, including running containers with systemd-nspawn. If you don’t know what that involves, then it’s pretty much a container that acts more like a virtual machine when you interact with it. I have also worked with out of date software projects so this one felt like it could be relevant to my interests.

This talk was a quick overview of the troubles that the speaker faced with a horrifically out of date software stack that was still actively used at a school.

The steps taken involved securing the system against basic attacks, then working out how to containerize it and run it, including ripping apart old Debian Docker images and having to work around such limitations as “there’s no systemd”.

Great reference for anyone facing similar situations. Wish the speaker had more time to go into the whole other category of issues related to the software stack itself, including the MySQL related issues they faced during this migration.

techtipsy rating: 8/10, great topic and useful tips, would have loved more horror stories and pictures.

Beyond Joins and Indexes

This session was a sequel to a previous talk that the presenter gave so I was worried a bit about having knowledge gaps, but luckily that wasn’t that big of a problem.

This talk covered some common PostgreSQL queries and various operations that go on behind the scenes, supported by illustrations on how exactly PostgreSQL performs those operations.

There were a lot of examples shown and it was genuinely interesting. At one point it became more difficult to follow along, but I’ll mark that down to this being the first session of the second day of FOSDEM. I think my brain quit on parallel hash join part. In a way I felt like I was back in school.

Given the background of the speaker, it’s no surprise that they have lots of presentations and resources on PostgreSQL. Will definitely have to check those out.

techtipsy rating: 9/10, made my head hurt but I’m now aware of what to Google when dealing with PostgreSQL, and I know which materials to reference to refresh my memory on this topic.

Isolation Levels and MVCC in SQL Databases: A Technical Comparative Study

This talk started out well: forget what you know about SQL standards because those are apparently not correct in modern SQL databases. Way ahead of you there, Mr. Pachot!

Turns out that life before MVCC (multi-version concurrency control) was rough and reading data involved locking it. Unintentionally locking tables is something that’s a common problem at my day job with writes, so I can’t imagine what type of hell life would be if that was still the reality.

This talk shed some light on how databases handle transactions, reads and writes. Good resource to refer back to.

Made me awfully paranoid about databases and data consistency now. Can I even trust databases any more? Or my knowledge on this topic?

Sidenote on YugabyteDB: as someone who has been surprised by Amazon Aurora quirks at my day job (broken indices after a minor version upgrade, anyone?), I find it interesting that there are other companies that provide a similar service involving a PostgreSQL-compatible database with a custom data storage solution below it.

Author also has a blog on all things SQL, might be worth checking out.

techtipsy rating: 8/10, made me feel even more insecure about my database knowledge, but at least I know how to fix that.

Linux load average and other silly metrics

Quick overview of why Linux load average is not an useful metric to follow, and can be incredibly misleading in some scenarios, especially if you compare sync vs async I/O. I’ve been guilty of tracking that metric religiously myself, so… whoops.

Even the Linux kernel source code calls it a silly metric!

Look at Pressure Stall Information instead to understand where your system is bottlenecking.

techtipsy rating: 7/10, good topic but the live demo part could have used some polish.

Private clouds do not need to be legacy!

This one was a run through various high-level tech tips to take into account when faced with the task of moving to a on-premise cloud setup.

During my own professional career that officially begun in 2016, there has always been a push to move workloads into the cloud, for all sorts of reasons, legitimate or misguided. Turns out that on-prem isn’t dead and some organizations are moving back in that direction due to legal or technical requirements.

Here are my takeaways:

avoid third party software because that’s usually a source of lock-in
if faced with buy vs build decisions, prefer build
keep things super simple
if faced with build-time or run-time complexity, prefer the former, otherwise more technical layers in run-time result in more difficult troubleshooting due to added complexity
consider using Kubernetes (and an open source distribution of it)
version control your infrastructure
if you automate something, automate it 100% end-to-end, don’t half-ass it or include manual middle steps

A lot of ideas mentioned here are also applicable to those working with cloud platforms in any large company. If you have competent engineers, then you’ll probably have no major problems if you move to a private cloud setup as a lot of the tooling and knowledge should transfer over.

techtipsy rating: 7/10, would have loved to see case studies of public -> private cloud migrations to illustrate the process and highlight the pain points.

Firefox power profiling: a powerful visualization of web sustainability

This was a fun one for me because I care about how much power my tech consumes, and the speaker put a new twist on it. Ever wondered how much energy is spent loading your website? Now you can find that out!

This talk ties in with Mozilla’s vision for a better internet and their stated sustainability goals. Building new computing hardware is very resource and energy intensive, and shoddy websites motivate people to upgrade sooner than necessary.

The steps to take to measure the performance and power consumption of your webpage are quite simple. The power measurement support is hardware and platform dependent, but even without that you can get a pretty good idea of how well your website performs. The whole process was easy enough to do that I could follow it along on my laptop during the talk.

The talk was simple to follow and yet covered a lot of details and nuance around power profiling on various platforms. This should be mandatory for anyone working in web development, especially those that add animated full-page backgrounds on websites or load in megabytes of JavaScript.

Give this talk a listen and start measuring website performance, I highly encourage it!

techtipsy rating: 10/10, I hate poorly performing websites and love my machines running as efficiently as possible, and this talk brings those two topics together in a very nice way.

Postgres vs. Linux filesystems

My experience with hosting PostgreSQL has mainly been limited to one instance that is used by Nextcloud, and during my professional career I’ve only observed migrations away from ZFS-based self-hosted PostgreSQL instances, so I was really interested in seeing how different filesystems affect PostgreSQL performance.

This talk laid out some general guidelines for anyone running PostgreSQL, such as keeping your kernel up to date to get the best performance and avoid old kernel bugs, and presented lots of graphs detailing the behaviour of different filesystems under heavy workloads simulated using pgbench.

I recommend that you view the results, they might surprise you. As a btrfs user, they certainly made me think about choosing it if I were to ever build a dedicated database server on raw hardware.

Benchmarking is very difficult to get right so I suspect that there might be other nuances at play here with the results. Interesting presentation regardless.

Sidenote: the presenter was running QubesOS, which I’ve briefly played around with in the past. I never expected to see anyone actually using it as their daily driver. Very impressive!

techtipsy rating: 9/10, I now have an urge to do my own benchmarks to verify these results (but my TODO list is long enough already).

Are Project Tests Enough for Automated Dependency Updates? A Case Study of 262 Java Projects on Github

I picked this one mainly due to being annoyed with Dependabot spam and the state of tests at my day job.

This talk covered the promise of automated dependency upgrades and ways you can avoid seemingly innocent upgrades breaking things in production, covering tools like PIT and Uppdatera. If your tests suck, then adding mutation testing and/or static analysis to your project might help catch problems before they make it to prod, find unused dependencies and tests that do not properly verify that the service works.

The speaker also briefly mentions the age-old problem of balancing code reuse vs increasing operational costs. Maybe don’t bring in thousands of dependencies so that you can avoid writing a few lines of code?

There’s a whole research paper on this topic as well (check presentation for more details).

techtipsy rating: 8/10, maybe a project where tests are in shambles can be improved after all…

The ones I watched later

Due to scheduling conflicts, room capacity limitations and the fact that I can’t teleport, I could not attend everything live.

Here are the ones that I watched later from a recording.

FOSDEM infrastructure review

Quick overview of the infrastructure that makes FOSDEM tick. Especially loved the rendering rack made out of laptops and lessons learned from doing FOSDEM over the years. Includes challenges faced by the organizing team and the (funny) workarounds they employed.

techtipsy rating: 9/10, recording quality is good and I appreciate the look behind the scenes.

Where Did All the Fun Go? And How to Bring it Back with FOSS!

Certainly one of the more unorthodox talks at FOSDEM 2024, but it does carry an important message. The AI and heavy metal parts can feel a bit jarring at first, but in the end it all makes sense, trust me.

Made me miss something that I realized we don’t have much of in tech lately: fun! Makes me appreciate creators like “Programmers are also human” and KRAZAM so much more now as they carry on the spirit that was covered in the talk.

This is one of those talks where the recording is a bit off, the slides are missing for about half the session so I recommend opening the slides in another tab (muted) and try to follow along.

techtipsy rating: 8/10, great content that lifts the spirit, but recording hampered by technical issues.

DIY Private Container Registry

This one I feel was hampered by issues with the recording and the short time slot. I felt like the session could go more in depth on the topic and the custom solution that the speaker was involved in building.

Never knew that the authentication and authorization part of handling private container images isn’t actually standardized and relies mostly on cloud providers’ features and/or HTTP basic authentication.

techtipsy rating: 5/10, there is potential for a good talk here if given more time and if the speaker goes in depth.

Open Source for Sustainable and Long lasting Phones

I discovered this talk randomly while scrolling through the FOSDEM schedule and checking the main track talks. I think I skipped over this one initially because I had no idea that this was all about Fairphone!

If you’ve never heard about Fairphone or why they exist, then I 100% encourage you to give this one a watch. You’ll learn everything you’d need to know about Fairphone, including their evolution and how they have approached software and hardware longevity in their phones.

I was surprised to hear how many obstacles Fairphone faced when trying to keep phones running on up-to-date software.

Yes, I do think that LinusTechTips missed the point of Fairphone.

If I hadn’t bought a Fairphone 5 before coming to FOSDEM, then this talk would have totally made me get one. That, and the fact that most of the custom OS booths had one on display. And Fairphone supports them!

techtipsy rating: 10/10, this talk gives me hope that we can change the world, one phone at a time.

You too could have made curl!

Inspirational overview from the guy behind curl, covering the journey of curl and the various challenges over the years. Daniel is very supportive and encouraging in this one, absolutely no gate keeping here and I love it!

If you had doubts about if you could ever be a successful maintainer of an open source project, then this talk covers most things that you’d need to know about the journey, what to keep doing and what to avoid.

There are more general recommendations as well, such as the time management aspect. Daniel intentionally leaves out playing video games and uses every available time slot that he has. I’ve found that if I just start doing things no matter the timeslots and pick carefully what to focus on, then I can get a lot done, so this strategy does work!

techtipsy rating: 10/10, the microphone quality isn’t great on the recording, but the good content makes up for it.

How to Build an Open Source School Cloud for 5 Million Users

This one was mainly about how ownCloud Infinite Scale was used to roll out a solution used by over a million students.

Brief and a bit light on the technical specifics, but still useful. I heard about ownCloud and their Infinite Scale offering before and was curious about it due to the alleged performance improvements, but never gave it much thought because the most of the community backing seems to be behind Nextcloud, and that matters a lot to me. Now I know how they achieve that scale and it sounds interesting, might need to give it a go.

If you’re wondering on a high level what components it would take to roll out such a big project, then this talk covered that fairly well. The tooling looks a lot like what I work with at my day job.

The recording glitched out near the end of the talk but returned to normal by the time they got around to the Q&A part. Good questions from the audience!

techtipsy rating: 8/10, would love to see a more in depth talk on this topic that also deals with operational struggles and the project management challenges.

Using code generated by AI: issues, misconceptions and solutions

A short and good intro into the world of AI and how it interacts with copyright law. Comparing audio file formats to AI models from the copyright perspective was a brilliant comparison to make.

I’m pretty sure now that most AI models are trained on copyrighted works. Hope the markets don’t notice that.

techtipsy rating: 9/10, easy listening, well-spoken speaker and I learned something new from an area that I don’t usually follow that closely.

Streamlining Developer Experience: The Power of CI/CD Standardization and Interoperability

Speaker starts with a mention of having used AI to generate some work in the slides, which is a fun point to make when the last talk I watched was related to AI and copyright. They were a bit of a distraction though as my mind automatically wandered to detecting AI anomalies in those images (there were plenty of those).

This one covers a lot of what is needed to provide a good developer experience so that your developers can better focus on their core objectives. It was more of a high-level overview of the principles and examples of tooling that support these. Most of what was covered sounded familiar so I guess that part of my day job isn’t in shambles.

Never knew that Backstage could integrate with CI/CD pipelines.

Some minor audio glitches in recording. Not a dealbreaker.

techtipsy rating: 6/10, AI-generated imagery was more of a distraction for me; would have loved to hear more about the technical specifics or case studies where these types of improvements have been made.

Stopping all the attacks before they start: Building a security-first API

Simple overview of what it takes to build an API that’s reasonably secure.

Unsurprisingly, the focus is on the very basics: validate your input, add authentication that works, check if the user is authorized to perform the actions and monitor and log your system. Speaker also emphasizes the importance of having a threat model because how else would you know who and what to protect against?

My career has shown that implementing the basics will get you very far towards having a secure system. If you ignore the basics, you will pay the price (and don’t forget about the accrued technical debt!).

techtipsy rating: 8/10, covers basic topics, but that’s the whole point since our industry can’t even get those right.

Observations on a DNSSEC incident: the russian TLD

My interest in this one was mainly about the actual real-life reported Internet outages in Russia that occurred shortly before FOSDEM. There was a lot of speculation and this talk covered the whole event really nicely, calling out the baseless claims by random people on the Internet. The speaker then proceeded to show what actually happened and showcased DNSViz, which was very helpful in visually showing what went wrong with the .ru top level domain.

techtipsy rating: 9/10, the session was short but sweet, the speaker cracked a few laughs from the audience and a technical topic got presented in an understandable way.

Open Source in 2024: boundaries, burnout, business

Great talk that covers the human side of open source. While this talk is more focused on the open source perspective, a lot of the recommendations here are also applicable to software developers in general. A lot of focus is put on setting boundaries, which is great. This talk complements this one by Daniel Stenberg of curl fame so give that one a watch as well!

I like how the speaker added links to their blog to various slides that cover the topic in question in depth. My reading list got a bit longer again.

During the Q&A there was an interesting idea mentioned: the human user guide, which as far as I understood is a document that covers everything you’d need to know about the person you’re working with. Things like “you should know X about me”, “it makes me happy when my colleagues do Y”, “it makes me sad when someone does Z”. It sounds like something that can backfire if implemented poorly, but it’s an interesting concept.

The slides seem to be missing from the recording so I recommend that you open them up in a separate window.

techtipsy rating: 10/10, great speaker covering important topics that many software developers unfortunately learn by first-hand experience, myself included.

Power profiling my entire house with the Firefox Profiler

A nice story about a software developer from Mozilla deciding to measure all things electrical in their home, and why they decided to use the Firefox Profiler to visualize it.

This one is relevant to my interests because I run some smart plugs at home and collect all that data for display in Grafana. A lot of the observations that the speaker made about devices consuming power sound similar to the ones I’ve made over the past month or so.

This one also contains a few tech tips for new parents who are also software developers. Unexpected, but a welcome addition.

Some small audio disruptions in the recording. Not a dealbreaker.

techtipsy rating: 10/10, interesting topic, slides are structured well, talk contains actionable advice, and it makes me want to measure absolutely everything at my home even more.

So you think you know Git

If you know git basics, then this one will be really useful as it covers lots of git features and functionality that you might not even be aware of.

I mainly use git via IntelliJ GUI nowadays due to the ease of use, but I now know what it is probably doing in the background to achieve those results, especially when it comes to checking who wrote that awful piece of crap soluti- oh, it was me back in 2021.

techtipsy rating: 10/10, speaker is very enthusiastic, lots of good tech tips and I learned something!

Magic and Software

Not even going to spoil this one, just go watch it.

techtipsy rating: 10/10, magic magic magic.

Closing thoughts

I loved FOSDEM 2024.

My expectations were high based on what I heard before, and it still exceeded them! The occasional technical issues weren’t a dealbreaker, there was generally plenty of room in the sessions (at least those I went to), and both the content and the people were great.

I felt at home, surrounded by people with whom I have similar interests. It’s not an environment I find myself in often, even at work, which is why FOSDEM felt so special.

Next time I’ll try to take part in more community events if possible, talk to people and leave some time for sightseeing in Brussels.

I’d like to thank my employer Concise Systems for paying for travel and accommodation.

Special thanks go to my friend Arti who introduced me to FOSDEM in the first place and helped make my first FOSDEM a memorable one.

FOSDEM gang.

My cat water fountain comes with a spicy USB power adapter

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Tue, 19 Dec 2023 14:00:00 +0200

It turns out that you can’t trust any USB type A power adapter to be within spec.

I have a Catit Flower Fountain for my two adorable cats. The idea of a water fountain for cats may sound odd, but having one really helps with cats staying hydrated and that alone avoids all sorts of health issues.

At one point I wanted to see if I could create a sort of a DIY UPS for the water fountain. It would be quite bad if I was at work and a power outage results in cats not being able to drink water (they don’t really care for normal water bowls after getting the fountain). I had some battery banks available for testing, and I noticed that the pump for the water fountain is powered over a USB type A cable.

Should be easy, right?

Apparently not.

I tried multiple different power banks between the water fountain and the USB power adapter that came with it, and all of them would work for a bit and turn off after some time. I didn’t think much of it back then, but I did notice that two of the power banks I used started glitching out during normal use elsewhere.

Months later, I attached an IKEA power strip to the side of my work desk to make charging various things easier. It also has two USB type A ports and the water fountain was near the desk temporarily, so I plugged it in there. It worked, but I noticed that the water fountain was quieter now, the “hum” that it makes was almost gone. That made me curious, so I used the original adapter again and the “hum” was there again.

I took a look at the original power adapter specs to see if there’s a difference in the amount of current that these two different USB power sources provide. What I discovered instead was that the power adapter that comes with the fountain outputs a solid 7.5V. I measured the voltage with my multimeter as well, and it reports 7.71V. USB type A ports typically provide about 5V, with a maximum of 5.25V from my observations in the real world.

Yikes.

7.5V over USB type A is probably not safe with other devices, especially since a normal person only sees a USB port on the adapter and thinks that it is perfectly safe to use it to charge their phone or other devices. Yes, properly implemented USB type C ports can negotiate all sorts of voltages, but this is not one of them.

Probably explains why my power banks are acting odd now and glitching out.

This is why I have trust issues.

2023-12-19 update

By popular demand, here are the two adorable cats.

Tux and Põssa. Põssa can be roughly translated to "Piggy" in English. I'll let you guess how he got that name.

The simplicity of the modulo operator: how I scaled an inefficient solution on a legacy system

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Mon, 11 Dec 2023 06:00:00 +0200

Your service cannot process events fast enough during peak hours.

There is no obvious quick and dirty fix.

Refactoring would take ages.

People have been unhappy for a while now.

What the hell do you do?

Rough representation of the situation we were facing. Not pictured: honking and yelling.

Background

I had the pleasure of working with a legacy backend system recently. It had plenty of ongoing problems, but one of those was more acute compared to others: the service could not process certain very important events fast enough during peak hours and that was problematic for everyone that relied on those events.

The processing of events was very basic:

load 1000 entities
for each entity, do the following sequentially
- do some processing
- send a JMS message via ActiveMQ
- wait for confirmation

The processing was triggered by the scheduled jobs system that the service had been relying on since its inception more than a decade ago. That system only allowed to run one instance of a scheduled job at any time.

This processing worked well for almost a decade at this point, but started causing more and more issues as the service was the backbone of a rapidly-growing business. At least it was a nice problem to have.

I was the lucky guy who got assigned to solving this issue, and in hindsight I’m really glad I got it because tackling this one was fun.

The slowness of the process was down to the part where events had to be sent via ActiveMQ. It’s possible to send events without waiting for any confirmation, but our service was going with the slower approach of waiting for a confirmation that the message was successfully sent. After consulting with my colleagues, going through the depths of git blame and doing some Jira archeology I soon learned that this slowness was there for a reason: some events could fail to be processed if the service suddenly died, which wasn’t that rare of an occurrence. Sure, the processing was quick, but you had the risk of losing events, and that was even worse than sending events with a big delay.

When looking at the behaviour of this code path I learned that sending each event took about 20 milliseconds. That is not a lot of time for a single message, but if you have queued up 1000 events, then that results in 20+ seconds required to send all of those messages. Take into account the fact that you are doing this processing sequentially and the fact that during peak hours you had to process thousands of events within one minute, and you can see where this becomes problematic.

To give some additional context: we’re dealing with a service that has at least 2 or more instances running at any time in a Kubernetes cluster. We had other high priority work ongoing as well and refactoring this part of the system was not feasible in a short time window. Spending weeks or even months on this issue was out of the question.

The solution

The first idea I tried was pretty basic: try to see if we could process each event in parallel using Java parallel streams. Hibernate and ActiveMQ put brakes on that idea pretty quick due to objects related to them not being thread-safe.

The second idea I tried was to find a way to bulk-send events since I suspected that the ActiveMQ connection setup and teardown was being done separately for each processed entity. That, however, was not the case.

The third idea was the one I went with. From my early programming days I learned about the modulo operation. I also had vague knowledge of Kafka and its way to split work via partitions. Didn’t take my mind long to connect the dots, and a TechTipsy certified quickfix™ was born.

I took the existing job, created multiple instances of it and gave each instance a number from 0 to 4, let’s call it a “worker ID”. Those modified scheduled jobs ran the same database query, but with a small adjustment: in addition to other criteria, each scheduled job only picked entities where the modulo 5 result of its numerical identifier matched the worker ID.

This means that the service could process 5 times more entries at once without having to commit to a big and risky rewrite.

I imagine this is what the "just one more lane, bro!" crowd thinks happens in the real world.

The PostgreSQL modulo operator helped facilitate this process and was the key to avoiding loading all of the entities into memory and filtering that list down using the modulo operator in application code.

All-in-all, this took a few days of work to implement, roll out in staging and production, observing and refactoring to make the solution maintainable. For a service that’s part of the critical path in the tech stack and notorious for its slow deployment cycle, it was pretty fast.

Example

Here’s an example to help illustrate the process. The database holds 7 entities with the following ID-s: 1001, 1002, 1003, 1004, 1005, 1006, 1007.

Worker 0 starts up and selects all entities where the result of ID mod 5 = 0.

1001 mod 5 = 1
1002 mod 5 = 2
1003 mod 5 = 3
1004 mod 5 = 4
1005 mod 5 = 0 <– this one
1006 mod 5 = 1
1007 mod 5 = 2

Worker 0 selects entity 1005.

Worker 1 selects entities 1001, 1006.

Worker 2 selects entities 1002, 1007.

Worker 3 selects entity 1003.

Worker 4 selects entity 1004.

All the workers operate in parallel.

Caveats

This solution isn’t perfect and has some caveats related to certain implementation details.

Adding additional workers is relatively simple, but requires lots of small but well-documented steps. This will only be a problem if the system sees additional growth that it cannot handle, but that should be years from now. I like to think of it as job security for future generations.

Due to the way the scheduled jobs solution is built, it is possible for one instance of the service to run more than one worker at once, which could be a problem for compute-heavy or memory-hungry processing work. However, for this use case it’s not a problem.

All-in-all, I’m happy with the solution, the team was happy after I made the TechTipsy certified quickfix™ more maintainable, and everyone relying on this solution to work properly were also happy.

Control - how to make a game enjoyable for casual audiences

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Fri, 01 Dec 2023 10:00:00 +0200

I’ve decided to intentionally take more time to play video games this year, since it’s a relatively healthy way to escape from the real world once in a while. A friend recommended one game in particular: Control: Ultimate Edition. During the Steam summer sale of 2023, I went ahead and bought it.

yeet

I have liked it more than I expected to.

And I'm not even done with the game!

What prompted me to cover this game wasn’t the captivating story, the sheer amount of content available in the Ultimate Edition of the game, or the wild action that you can find yourself in, it’s much more simple than that.

In short, Control provides you with “Assist mode”, a set of options that allow you to adjust the high difficulty of the game. This includes settings that help lock on to enemies, which is great for players using controllers. There are also sliders to adjust the recovery rate of various resources and even a one-hit kill mode.

My favourite of all of these is the immortality toggle.

Assist mode options in Control.

I don’t take time to play video games to add more stress to my life. I work with legacy software that falls over in novel ways almost weekly at this point, there’s too much stress in my life already. I like a good challenge once in a while, but in most cases I play games to have fun. I’m also a fan of great stories, and many great stories are told via the medium of video games. I’d hate to miss out on those experiences due to the game being too difficult or making me feel frustrated.

In the case of Control, the immortality toggle was what allowed me to relax, follow the story with great interest, and feel like a god that kept yeeting exploding forklifts at various enemies with explosive gun shots sprinkled inbetween. There were still some challenging parts in the game even with immortality in place, but they did not cause me to feel stressed out or get frustrated with the game.

This one simple toggle is probably the sole reason why this game has become one of my all-time favourites.

I’m sure that there are people out there who disagree with the view that games should always be easy or enjoyable. I don’t care. I play games for myself, not to be validated by some bozo on the internet.

While writing this post, I looked around the internet to see what others think of Control and its difficulty, and what caught my eye were numerous results discussing the difficulty of the game. I was also happy to see that there are others out there who agree with me (or more precisely I agree with them). I found it surprising that the “Assist mode” was added to the game a year after its release. Guess it pays off to be a patient gamer.

Perhaps all of this explains why I had such a blast playing GTA San Andreas as a teenager. Cheats like HESOYAM and AEZAKMI are still hardwired in my brain since I used them a lot, and they allowed me to wreak havoc in-game and not have to worry about getting stuck at an annoyingly difficult mission.

Hats off to Remedy Entertainment for adding the assist mode, I really appreciate it.

Oh, and the game works well on the Steam Deck. Great job!

The optimization treadmill: why I keep changing my computing setup all the time

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Wed, 25 Oct 2023 10:00:00 +0300

I have a bad habit of changing my computing setups all the time. I tend to see new gear, then I get some new ideas, and then I obsessively think about it for weeks and months until I just buy it. And then the cycle repeats.

I’ve had time to think about why that keeps happening and I think I’ve got it.

I keep changing the goals, constantly, and with that I kept optimizing my setup in a different direction every time.

Oh my, that's a lot of money spent over the years.

This post also serves as a way to explain my thought process behind my last post about the Zimaboard, since my writing may not have been clear enough on that point, or some of you just can’t read.

The treadmill

It all started with the first step in my self-hosting adventure. I had a 1TB WD external hard drive that was a single copy of all my important files, which is less than ideal if you want to preserve your data. My goal became to have another redundant copy of the storage, so I built a server from old PC parts and used hard drives, all running ZFS, and hosted in a room where loud PC-s were more of a norm.

First actual server setup that I had.

After that setup moved to my rented apartment, I discovered that having a buzzing tower of 8 hard drives (that was an upgrade I later made) consumes a lot of power and is also unbearable when you’re trying to sleep in the same room. My goal became to reduce the noise and the power consumption, so I upgraded to a Ryzen 2000-series based setup and at least two WD Red 4TB hard drives. It was quieter, faster and more power efficient compared to the previous setup.

Fun fact: one of the WD Red drives died and had to be replaced through warranty.

Then I discovered that laptops use even less power than that. A ThinkPad X230 I had around used less than 10 watts when idling, and it just so happened that I had one to play around with. My goal became to reduce the power consumption of my home server setup even more. I ended up putting my storage on two 4TB WD Red hard drives connected over USB 3.0 and running my home services off of that ThinkPad.

Soon after that, I discovered that having a cat around USB-connected storage running btrfs is a really bad idea. There was data loss, but luckily nothing I could not replace. My goal became to have a low power setup that was more cat-proof and did not rely on USB-based storage. That’s how I ended up with a TerraMaster F2-221.

Meet Tux, the data loss queen. Hopefully makes up for the fact that I don't have a picture of those two setups.

Damn, these hard drives sure are loud in that NAS! I also discovered that some external 2.5" hard drive enclosures can be opened up to reveal 2.5" 15mm SATA hard drives in up to 5TB capacities. And you can plop those onto a PCIe card, for some reason. And I really liked the Thermaltake Core V1 and SSD-s for certain types of storage. My goal became to build a really cool and performant server that housed both hard drives and SSD-s, because why not?

RGB-powered server.

Oh, wait, no, that thing uses a lot of power and a ThinkPad T430 became available in my computing fleet. I probably got a good deal on some 8TB hard drives around this time as well. My goal became having a lot of storage around while also using less power, and having it all as cat-proof as possible. That’s how a ThinkPad T430 ended up being a home server.

Embrace the jank.

Oh, crap, that ThinkPad kept overheating because the CPU in it was a bit too much at higher loads! My goal became to find a suitable replacement that did not use a lot of power. Luckily I heard about the concept of TinyMiniMicro and got myself a Lenovo ThinkCentre M73 Tiny.

The original TinyMiniMicro setup.

Damn, those hard drives sure are noisy, apparently due to WD running those at 7200rpm. My goal became to silence those hard drives as much as possible. I got a lot of acoustic foam, and what ended up working was to stack the drives horizontally, and add some foam between the pieces, that got rid of the humming. I also added a fan to keep the drives from overheating.

The TinyMiniMicro setup with some foam. I added the fan later.

Hold on, the Raspberry Pi 4 is allegedly usable as a home server now? And it uses even less power when idling? My goal then became to have a server setup that has a single-digit idle power consumption, meaning less than 10 watts.

The Raspberry Pi 4 that I did testing with.

Unfortunately I could not get the software part running right, the setup would freeze due to unknown reasons. At least I wrote a post about troubleshooting ZFS issues as part of this run, and Google keeps recommending it to people according to Google Search Console data.

If Google keeps recommending it, then I guess it's useful to people.

Alright, I can’t make the setup more efficient. Hold on, I also have a desktop PC, and that also consumes power when idling. That seems wasteful. My goal became to do my computing all on one machine, filling the roles of a server, workstation and gaming PC all at the same time. And I did it, and wrote a post about it as well!

The all-in-one setup.

Spoiler alert: it was in fact not the end-game of my setup, because I soon discovered that having everything on the same PC requires quite a special setup, and a reboot on the host OS affects all the other use cases. If you care about uptime, then it’s probably better to have separate machines for all of that. I didn’t want to give up on the idea of virtualized setups just yet, and thus my goal became to have one PC for my server and gaming needs. Games need storage, lots of it. My server has lots of storage. Makes sense, right?

The heart of the new setup: ASRock X570M Pro4 motherboard.

After using that setup, I discovered that VR and virtualized gaming don’t mix that well. Damnit. Oh, hey, I have all these SSD-s now from that experiment, and if I shrink my data needs a bit, I could fit it all on a mirrored setup of two 4TB SSD-s, and they don’t use much power as well. That opened up a lot of possibilities regarding my next setup because a lot of small PC-s can handle two SATA SSD-s just fine. My goal became to build a silent and efficient home server. This resulted in the ASRock Deskmini X300 setup.

Look how tiny it is!

I’m hoping that the current DeskMini-based server configuration is the one that will last me a while.

That aged well.

I had a ThinkPad T430 lying around, and now that my storage needs were served by two 4TB SSD-s, I could finally do that one stupid idea from 2020. I also dislike knowing that I have perfectly good hardware sitting in a box and collecting dust. My goal then became to use my ThinkPad T430 as a low-power, silent and efficient home server. It was successful. Or at least mostly successful.

The ThinkPad T430 as a server.

But the search for that elusive combination of low power consumption, adequate performance, stability, resilience and silence was not over. My blog gained traction from time to time and it would be really embarrassing if an overheated laptop or a longer power outage resulted in an outage. You see, the ThinkPad was good as a server, but it still overheated under specific conditions and once the battery runs out, the “Power on with AC attach” functionality does not work, which is quite bad for a server. My goal became to find that elusive solution.

I tried the LattePanda V1 since I got it for a good price. Didn’t work out for my use case, sadly. The website situation got solved with moving that portion of my self-hosting setup to k-space hosting, leaving me to do further experiments without affecting my lovely website that much.

The LattePanda test setup.

After keeping my eyes open for various options, the Zimaboard was the one that finally won. Silent. Meets my storage, networking and performance requirements. Small. Super low power usage. x86 based so I know my stuff runs on it well. It’s not perfect, but it did all of those things.

Hopefully this one sticks.

And there you have it, the reason why I have written over half of these blog posts, and why my setup just keeps changing and depleting my wallet at the same time.

Let’s be honest, I’m sure that as soon as I find something exciting in the tech world to play around with, I’m probably going to try it out. Some call it passion, I call it lack of fiscal responsibility.

Free tech tip to those starting out in self-hosting: experimentation is really fun, but it’s easy to lose focus and get carried away with changing your setup all the time. You can do a lot with much less hardware than you think, so try to approach building a minimum viable home server setup first and upgrade only if absolutely necessary. In my experience it’s just as fun and probably much cheaper as well.

Zimaboard: the closest thing to my dream home server setup

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Mon, 09 Oct 2023 14:00:00 +0300

I stumbled upon this Hardware Haven video about the Zimaboard recently.

I liked it a lot.

I finally bought one.

In short, Zimaboard is a small single-board computer that is relatively affordable and comes with an interesting selection of ports, which includes an exposed PCI Express port.

Before we get down to the build, here’s a list of aspects that I want to see in my dream home server:

low power usage (2-15W typical power usage)
8GB of RAM or better
enough performance to run my workloads, most of which are containerized
2x SATA or NVMe SSD slots, plus option for a third drive for the OS
passively cooled and completely silent
compact size
gigabit Ethernet or better

You might be thinking, “Wait, that’s your dream setup? No clusters of machines, Threadrippers, 10 Gigabit networking, crazy number of disks?”. Well, yes. After years of trying all sorts of setups and learning about my home server usage patterns, this is the set of requirements that finds a balance between performance, efficiency and silence.

Basics

I recommend checking the product page to see the exact specifications. The configuration I bought was the 832 model: 8GB of RAM, 32GB of eMMC storage and a quad-core Intel Celeron N3450 CPU. It’s not the most powerful setup or even a recent one with the CPU being from 2016, but it’s just powerful enough to fit my needs.

This variant of the board costs 200 USD, but other configurations cost much less than that, I just needed the extra memory to be on the safe side. If you don’t care about the noise and size aspect of home servers, then you can get a better deal on the used market (see the TinyMiniMicro project for inspiration), but as you know by this point, I care about those aspects a lot.

The package

What sets the Zimaboard apart from other single board computers is how polished the product feels. Unlike a board like the Raspberry Pi, this one comes with a case and a cooling setup already attached to it. I suspect that a similarily configured Raspberry Pi 4/5 with all the accessories added on top would result in a price that’s quite similar to the cost of a top-of-the-line Zimaboard.

The heatsink looks great and is practical at the same time. Under the most torturous loads I could only see the CPU being around 72°C and due to it being passively cooled it made absolutely no noise. With the case being present, I do not have to worry about placing the board on my desk and scratching the table or shorting something out.

Just plop it wherever.

The board does not seem to have a power button, but by default it’s configured to power on as soon as you connect the power supply, which is great if you’re going to use this as a home server.

The box that the board was shipped survived and overall I’d say that the packaging is good. The board comes with some stickers and a single SATA data+power cable. The included power adapter comes with EU, US and UK plugs all included.

Shipping to Estonia was quite fast, taking just 10 days. The shipping costs were 18 USD.

The box did its job.

Storage

The included 32GB eMMC storage is fine for hosting your operating system. The read speeds cap out at around 175 MB/s and the typical write speeds I observed were around 50-100 MB/s. The storage is identified as mmc-BJTD4R_0xc7d04e40 under /dev/disk/by-id/, and searching online suggests that it’s a Samsung chip.

Because this board offers two SATA ports, I also added a SATA Y-cable to my order (4 USD) so that I can take my existing Samsung 870 QVO 4TB SATA drives and move my home server setup to this board.

SATA Y-cable in action.

Allegedly this cable can also be used to drive two 3.5" hard drives powered by the board itself according to the shop page for the Y-cable, but I suspect that at that point you’ll be pushing the limits of the 12V/3A power adapter.

There is also a white drive activity LED placed near the SATA power connector on the board. Perhaps not for everyone, but I like the aesthetic and the sight of the server doing server things.

The performance of the SATA ports is what you would expect. When performing read operations on both SSD-s I saw the maximum total transfer rates hover around 900-950MB/s, which is pretty close to the SATA III transfer speed limit.

There is no native way to mount the two SATA drives to the Zimaboard. The creators of the board do sell a metal bracket, but it doesn’t seem to integrate that well to the board. However, there exists a 3D printable design that houses two 2.5" drives, even 15mm ones, and that’s how I ended up using a 3D printer for the first time in my life. The print was done using a Voron v2 Afterburner printer hosted at k-space and it came out pretty well.

Print in progress.

The final product

Another angle.

Some 3D printing related inconsistencies, but nothing serious.

The SSD mounting holes were a bit finicky due to slight printing errors, and during my first installation attempt I forgot to put in the plastic middle layer of the case that’s between the PCB and the stock bottom cover, but other than that the installation was a breeze.

Installation of the caddy requires removing the stock backplate and splitting it.

The finished result.

The drives never hit above 64°C, which is absolutely okay for these SSD-s. I’m not sure how actual spinning hard drives might fare in this environment, but I don’t think that it’s going to be that serious because 2.5" hard drives should not generate as much heat in the first place.

Power consumption

What I love about this board is the power usage. At this point my home server is not even the most power hungry component of my setup, that honor now goes to the ISP-provided router/modem combo box that always draws at least 12W, even when it’s in bridge mode.

Here are my power usage measurements (measured with a simple power meter):

idle: 2.5W
no drives attached, CPU stress test (stress -c 4): 9.0W
one Samsung 870 QVO 4TB SSD attached, CPU stress test: 13.2W
2 SSD-s attached, typical power draw in my setup (~10-40% CPU usage): ~8-9W
2 SSD-s attached, max load on SSD-s and CPU: ~14W

The Zimaboard, ISP modem/router box, my TP-Link router/Wi-Fi AP and my CyberPower UPS all together use at most around 34W, all combined. That’s even less than what my Dell monitor uses at reasonably low brightness levels!

These tests are not scientifically accurate, but they should give you an idea on what power consumption numbers to expect when running this setup.

Performance

I use a simple Prometheus Node Exporter + Grafana setup to view how much resources my various servers use. My home server has lately either been an ASRock Deskmini X300 based setup or an old ThinkPad T430, and something that both had in common was that the CPU usage was generally very low, mostly at or below the 10% mark. There would be bursty loads from time to time and backup processes running that bump that up, but not significantly.

Typical CPU performance on a ThinkPad T430 acting as a home server.

Memory usage of my setup was also quite low, with all my services and containers fitting into 2GB during typical usage. With this information and some CPU performance comparisons done, I knew that the Zimaboard will likely be able to handle my home server tasks.

This board is not very powerful, but if you mainly rely on containerized workloads and can rely on Intel QuickSync to accelerate media transcodes, then you’ll be just fine.

Zimaboard CPU usage, which includes migrating filesystems, creating new multi-TB backups from scratch and a lot of Jellyfin playback and GPU-accelerated transcoding. Pretty much the worst case scenario for this board.

I was actually impressed with how well the GPU-accelerated transcode worked on this machine once you set everything up properly. If you’re running Linux and Jellyfin, run vainfo to get a list of supported codecs and make sure that you have hardware decoding selected for those in Jellyfin settings. Also enable hardware encoding. I checked with intel_gpu_top to see if work was offloaded to the GPU and saw activity there, which means that hardware acceleration for Jellyfin worked out great! Check the Jellyfin hardware acceleration for more details on other requirements that have to be met for all of this to work.

The Jellyfin QuickSync hardware acceleration config that works on my Zimaboard. May not be 100% correct but so far have not encountered issues.

intel_gpu_top during Jellyfin transcoded media playback.

Caveats

The Zimaboard does ship with a Linux-based OS called CasaOS. However, I had no intention of using it as I know what my requirements are and my infrastructure is already decently documented and set up via Ansible, which is why I can’t say how good that experience is. I also did not test Windows 10 or 11.

During testing, I noticed that the USB 3 ports can act a bit weird. The Fedora Server USB stick would not boot properly if connected to one of the USB ports, and when doing other tests with external USB storage I noticed hiccups from time to time. Not sure what might be the cause, but I’m writing it down nevertheless. It’s not a dealbreaker for me luckily since I don’t rely much on the USB ports in typical use anyway.

I did not test the PCI Express port, simply because I don’t need it, yet. It’s a great addition, though, and opens up plenty of modding capabilities in the future.

Fedora Server and btrfs

The Zimaboard gave me an opportunity to start fresh with my home server.

I made a leap: I’ve ditched ZFS and am now running Fedora Server with my storage being on a btrfs RAID1 setup, snapshotted and backed up using btrbk. It was a bit tricky to migrate and set everything up regarding backups and snapshotting, but I got it working.

ZFS is great, but it has always felt like an unwanted guest in the Linux ecosystem. The kernel developers don’t care much for maintaining compatibility with ZFS since it’s not in the kernel due to licensing issues, and Ubuntu has been one of the few distros that actually ships a kernel that includes ZFS built in. I didn’t want to be tied to Ubuntu forever, especially because of how they try to make snap a thing. ZFS DKMS builds are generally okay on distros like Debian, but on others you might find yourself not being able to access your data after a reboot because of a kernel update.

btrfs has had some issues in the past, especially with the RAID5/6 setup, but in my single and dual disk setups it has been solid for years, except for that one time around 2018-2019 when I ran btrfs RAID1 over USB storage. To be fair to btrfs, that was a pretty stupid setup.

Before committing to btrfs, I used two USB sticks to create a RAID1 setup and created real torture test scenarios. Tests looked something like this:

write a file to the filesystem
use md5sum to calculate a hash of it for verification purposes
completely wipe one USB drive with dd
run md5sum to calculate the hash again (it matched every time)
run btrfs scrub on the filesystem to fix all errors
rinse and repeat with variations to this setup

After doing all that, I was quite confident that this was going to work.

One thing to note with btrfs: if you’re running any type of multi-disk and redundant setup like RAID1 or RAID10, then make sure to include the mount option degraded in /etc/fstab so that you can still mount your filesystem if one or more of your drives fail. If you lose too many drives then it’s still probably going to fail to boot.

Here’s my /etc/fstab setup to serve as an example:

LABEL=turbo /turbo      btrfs subvol=turbo,compress-force=zstd:1,ssd,degraded,nofail            0 0

When doing the migration I kept all the paths the same, and each ZFS dataset was recreated using btrfs subvolumes, which is why this filesystem is mounted on a top-level folder.

Alternatives

My notes for the next dream home server setup included a few candidates:

the Zimaboard
Asustor Flashstor 6 or Flashstor 12
a TinyMiniMicro style machine with at least two NVMe SSD slots

The first two fall nicely into my requirements for the dream home server. I picked the Zimaboard mainly because it’s much cheaper and I was already running two SATA SSD-s for my home server storage, so migrating would be really easy and I would not have to buy any new drives.

If large capacity SATA SSD-s start becoming less common and NVMe SSD-s become even cheaper than they already are, then I’ll have to look into something like a Flashstor.

Conclusion

Overall, I’m very happy with this purchase. The performance is just enough for my services to work reasonably fast, the board uses very little power and it’s silent, and it looks good on my wall.

Oh it's a setup it's a setup it's a setup 🎶

Managing the coax cable like this feels so wrong, and yet it works fine.

My setup is also quite flexible, so in case I need more resources, I can get more Zimaboards and make them serve different purposes. However, it’s more likely that I’m going to discover a new toy to experiment and play with by the time I run out of resources on this one.

2024-08-20 update

I ran the Zimaboard for over half a year, but decided to switch back to the ASRock Deskmini X300.

I still love the low power consumption and the expandability, but I ended up putting more load on the system than I initially anticipated CPU-wise (running over 20 different Docker containers, some heavy). Should’ve guessed that it would happen eventually.

If it came with an embedded AMD Ryzen APU, then that would probably change things up quite a bit.

Zimaboard: still the closest thing to my dream home server, but it’s that last little bit of performance being missing that gets me. If your needs are moderate or low, then it’s still a fantastic option, if the price is right.

My eyes are now on the Asustor Flashstor Gen 2 machines. Up to 12 NVMe SSD-s plus 4 cores of AMD Ryzen goodness sounds too good to be true. Let’s see what the reviews will say once it’s out.

My trip to the Communication and Laptop Museum in Estonia

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Tue, 19 Sep 2023 06:00:00 +0300

While on vacation I went on a small road-trip across Estonia. During the second half of the trip I ended up being in Võrumaa, and while driving I suddenly remembered a random fact that some people mentioned in a hackerspace Slack channel: there’s a new museum around here!

The museum was officially opened on 10th of June 2023 (more news coverage from ERR) and when me and my wife got there, we were visitors number 96 and 97. By the time we arrived there, other visitors were wrapping up and we were greeted by the owner of the museum.

The bulk of the museum lives in a small wooden house. While the building might be small, there was plenty to look at on the inside. Old servers, mobile phones, rotary dial phones, telecommunications equipment, promotional phone company materials from 90s and 00s, it had it all. Some of the phones were even connected together and you could make actual calls between them.

Promotional materials and actual customer support desks that used to belong to EMT (Eesti Mobiiltelefon), a telecom company that operated since 1991.

Ads used to be wild.

Lots of older mobile phones. Most of them were relevant during various parts of my childhood.

Big mobile phones.

The biggest mobile phone: promotional display monitor shaped as one of the first iPhones.

Next to the building was an actual 2G/3G tower. Not as high as the real thing, but all the equipment was there and set up. It wasn’t connected or operating for obvious reasons, but still, pretty cool to look at. So cool that I forgot to take a picture of it.

There were a bunch of old payphones lined up near the museum building. If I remember correctly, one of those was connected up to the other phones inside. Now these were a blast from the past, I remember seeing these payphones all over Tallinn during 2000s, didn’t use them much myself though. The payphones also contained examples of plastic cards that you had to use to be able to use the payphone. Those had all sorts of wild designs so there was plenty of opportunity for personalization.

Payphone with various pay cards present behind a plastic shield.

Party like it's 1999.

The yard contains a selection of signs from telecom companies over the years. Those used to live on the outside of buildings or within shopping centers.

You can see when the rebranding started and evolved.

The signs are bigger than you realize when you get closer to them.

The highlight of the museum for me was the owner and maintainer, Marko Koemets. It’s clear from the way he speaks that he is passionate about communications equipment and technology in general. With each exhibition he always had interesting stories to share, either about the equipment on display or the history of Estonian telecommunications starting from the 90s. I won’t spoil any of those here, you’ll have to visit and hear them first-hand.

He also shared some future plans with the museum. If there are enough donations, then the museum is going to expand with a second building dedicated to old laptops. The owner has 600+ laptops waiting to be shown off.

If you ever find yourself in south-east Estonia and you’re interested in this type of tech, then do give this place a visit. Details about opening times, prices and contact information can be found on this lovely appropriately themed website.

If you can’t make it there physically but are interested in supporting the museum, then you can make donations to the bank account listed on the website. You’ll also get a mention on the website! There’s also merch that you can buy on-site to support the museum.

Ultra limited edition "Side ja laptopi muuseum" mug. Inside are some pens, two of them belonging to a telecommunications company EMT.

I recommend checking the website and details about the opening times before planning a visit. At the time of writing you need to call ahead to set up a visit. It’s definitely worth the effort and after the visit you can explore south Estonia, there’s plenty to see and do there.

Steam local network game transfers are a game-changer

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Mon, 11 Sep 2023 06:00:00 +0300

Steam recently launched a new feature: local network game transfers.

The idea is simple: if you have a game downloaded on another PC and you’re both on the same local network, then Steam can download game data from that PC, avoiding the need to download the game over public internet. Using this method you can reduce your internet usage and enjoy faster download times.

I think this feature is absolutely brilliant. There are many homes out there, even in developed countries, where the internet connection sucks. The speed might be capped at something very slow, such as 10 Mbit/s down, or you might have a bandwidth cap in place. Perhaps you have multiple gaming PC-s and you want to avoid downloading 100+ gigabytes worth of files over the internet to avoid hitting your bandwidth cap. In those situations being able to download your games to a PC that has a bunch of storage in it makes a lot of sense.

This isn’t the only way to achieve the same goal as projects like LanCache exist. However, with Steam the setup is so simple that anyone who knows how to install Steam could achieve a similar result. The only caveat is that you’re limited to games installed through Steam, LanCache supports many other services.

Here are some ideas on how you can set up your own Steam cache.

Using an old laptop

If you have an older laptop around and it has a gigabit Ethernet connection, then that can be a good candidate for setting up a Steam cache.

The hardware I picked for this test is a ThinkPad T430. It has a 4 core CPU in it, 16 GB of RAM and plenty of ways to attach storage to it. With storage I tried two approaches: using an older 1TB 2.5" SATA HDD, and using two Samsung 870 EVO 1TB SATA SSD-s in RAID0.

It's a Steam cache. See, it says so right on the lid!

The hard drive based solution isn’t something I recommend, unless you plan on running two or more in a RAID-like setup. The Steam cache use case seems to have hit the limits of the hard drive and that limited the transfer speeds significantly.

The SSD-based setup was great! You might want to run something other than a RAID0 setup if you want to avoid a drive failure taking your whole Steam library with you, I ran with this setup mainly to see where the limits are. The Steam library is also something you can easily redownload, should things go really wrong.

One benefit of setting up an older laptop as a Steam cache is that you can take it with you to a place that has great internet connectivity, download your games there and bring it back home with you. That’s exactly what I did when performing testing, I went to my local hackerspace, started the downloads, and came back half a day later to find that over a terabyte worth of games had been downloaded. Doing the same on a slower home network would have taken days. I guess the idea of IP over avian carriers, or over humans, isn’t dead yet.

When downloading the games, I noticed that Steam was hitting the CPU hard. Even on a decent 4-core Intel i7-3820QM the CPU was often running at 100%. This meant that download speeds were usually around 500-700 Mbit/s, which is still good, but nowhere near saturating the gigabit link that the laptop had.

The CPU is really struggling here.

Now that I had all the games downloaded, it was time to test the performance. The maximum transfer speeds were around 400-500 Mbit/s over my local network. I used my Steam Deck over an USB-C dock to start the downloads and picked the internal NVMe SSD as the download target to avoid any storage speed bottlenecks. I would have liked to see speeds close to 1 Gbit/s, but these speeds are still a massive improvement if your network speeds are something like 50 Mbit/s down, that’s a 10x improvement!

Typical transfer speeds observed with this setup.

If you’re worried about the fact that Windows 10 is going to lose support in October 2025 and your Steam cache PC is not supported by Windows 11, then don’t worry, you can do the same thing over Linux.

Using a virtual machine

If you already have a NAS or a home server running, then you can create a new VM, add a bunch of storage to it, setup the OS and Steam and you’re good to go!

I did most of the testing on a Windows 10 VM and a 1TB HDD attached over USB 3.0. I setup the VM using a bog-standard QEMU/KVM setup and set everything up through virt-manager in Linux. The only noteworthy part is that I set up the network interface as a macvtap device, resulting in the VM showing up on my local home network as if it was a separate PC. That step was taken to avoid any issues with the Steam instances on different PC-s not being able to transfer data directly between each other.

My server runs on the ASRock Deskmini X300. It has plenty of CPU performance, but during testing I ran into performance issues with the hard drive again. When I added SSD-based storage to the VM, I found that the performance was great and quite similar to what I saw on the laptop.

The downside with the Windows 10 VM setup was the CPU usage. While the VM was idling, my server saw a noticeable increase in idle CPU usage, utilizing about 10-20% of my CPU at all times. During transfers the VM was using most of my CPU cores.

I did testing on a Linux VM to see if those are more efficient compared to a Windows 10 VM. The answer is obviously “yes”, but there is one issue: I could not get Steam to work. At the time of testing, Steam had recently released a bigger visual overhaul and I suspect that it might have something to do with those issues. The VM has no GPU acceleration and opening Steam would result in “Loading user data” popup showing up, but nothing else happening. I tried Ubuntu Desktop 22.04, Fedora Linux 38, and even Flatpak installations of Steam. All of them ran into the same exact issue. There are probably ways to set Steam to not require GPU acceleration without opening its settings view graphically, but I did not bother with that.

If you like to think in virtual machines, then this is a good option. However, it’s not something I stuck with due to the inefficient resource usage that a Windows 10 VM exhibited.

Using containers

This is the solution that I am currently running.

I recently found the LinuxServer.io webtop Docker images. These containers allow you to run various Linux desktop environments and distros on the same machine. You can connect to them using the integrated KasmVNC solution over a browser. You can even use your GPU on the container host machine to improve the video rendering performance in the container!

Fedora XFCE desktop running in a container.

I picked the fedora-xfce option because Fedora is great, and the XFCE desktop environment is not that resource intensive. I also utilized the linuxserver.io package install mod to install Steam on container startup. For networking, I created a separate Docker network using the macvlan driver to achieve something similar to the VM setup, resulting in the container showing up as a separate machine on the local network. Inside the container I set up Steam to start on startup using standard GUI tools that XFCE provides for this purpose.

I really like this setup. I can easily point the Steam downloads to a larger storage pool and make use of all the CPU power that Steam needs to perform the transfers.

During testing the limitation seems to have been my Steam Deck. I could still not saturate the 1 Gbit/s link.

The KasmVNC solution is quite nice and performant. The fact that I could use it in the browser makes it very convenient to use as well.

The CPU usage during idle is very small, except for when you accidentally leave the Steam window open with animated content being displayed. During file transfers the CPU gets a beating, but it’s nothing the AMD Ryzen 7 5700G can’t handle.

For storage I eventually ended up using a 1TB Lexar NVMe SSD. That’s not enough to download all the games I have in my Steam library, but at least I can download those games that take 20+GB of space and not have to worry about downloading those over a slower internet connection.

I have not ended up using this setup too much yet, but I imagine that with games growing in size this is going to become much more relevant for my Steam Deck. And if I were to move to the countryside where the only internet connection available is a mobile 4G connection capped at 10 Mbit/s down, then this setup would be fantastic to have.

Tech tips

If you intend to run a similar setup, I recommend enabling “Scheduled updates” to run during the night. I set mine up to auto-update games between 00:00 and 06:00, which is where I’m likely going to sleep and not notice that game downloads are taking place. Steam is notorious for hogging all of the bandwidth on the network and this nifty feature helps avoid disturbing other users on the network.

If you’re performing the initial mass download of your games but would like for it to take place during scheduled times, start downloading all of the games and then restart your machine. When Steam starts up again, you’ll notice that all your games are queued up nicely to download during your scheduled updates time slot.

Caveats

This setup is not ideal and the Steam support page highlights it well. For example, this setup does not work in fully offline situations as internet connectivity is required for the initial setup of the local network game transfer. I recommend giving that page a read to understand the requirements and technical specifics of this setup.

The IKEA-powered homelab on a wall

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Thu, 07 Sep 2023 06:00:00 +0300

I recently learned about the IKEA SKÅDIS series, which is a pegboard that supports a variety of extras. During my self-hosting journey having to figure out the best place for putting all my compute stuff to has always been at the back of my mind, especially due to limited floor space at my home. This pegboard gave me an idea.

Behold: the homelab on a wall!

Note that this idea also applies to other types of pegboards that you can find, I just went with the IKEA one for aesthetics (and the wife approval factor).

The layout was improvised during the assembly process, but in the end I’m quite happy with how this turned out. Everything is held in place with zip ties of varying sizes. The biggest ones I used are 360x4.5mm, which fit nicely due to the slot width of the pegboard being 5mm. Some of the zip tie ends have been cut off for aesthetic reasons, but most of them are intact in case I want to perform maintenance on this setup.

There are some improvements that I might implement to a revision of this board. The USB cables connecting the external SSD-s seem to be just small enough to be routable behind the pegboard. Other cables that are visible can probably be hidden on the other side as well, giving the setup a much cleaner look.

I also installed a second pegboard to have access to most commonly used cables and other stuff that I usually store on my table in a chaotic fashion. It’s a work in progress, but it looks very promising already.

The misalignment is a representation of my struggle to install the damn thing properly. I'll fix it eventually.

Maintenance is not going to be very easy on this setup. Luckily the UPS should be maintenance-free for the next 2-4 years or whenever the battery dies, and with the ASRock Deskmini X300 I can unscrew the 4 screws on the back and slide the thing out without having to take off the whole pegboard. I hope that the external SSD-s don’t die on me soon, but if they do, then sliding them out of the zip ties should not be that difficult.

Those more familiar with my content might see echoes of the seedbox on a wall in this post.

Why you might not want to publicly self-host a Wikipedia clone

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Tue, 29 Aug 2023 06:00:00 +0300

A while ago I wrote about how easy it is to download an archive of Wikipedia and host it anywhere you want using Kiwix.

Here are a few things you might want to consider before doing so yourself.

The spam

I have a specific e-mail address set up so that readers of my blog can reach out to me no matter where they see my post. I knew about the risk of spam, but receiving genuine feedback and questions is something I appreciate a lot and I’m willing to mark everything else as spam if needed.

What I did not expect was the amount of spam that originates from my self-hosted copy of Wikipedia, or that they’d use the e-mail that is present on the main ounapuu.ee domain.

The scheme seems to work like this:

crawl the web
find links to genuine sources that have broken (HTTP 404 errors)
spam the owner of the site with a politely worded e-mail template that asks you to fix those broken links with these new ones that link to some marketing garbage or SEO spam
go to step 0

And the worst part is that this probably works for a lot of smaller sites with non-technical owners.

I’ve deleted most of the spam but have some recent examples (click on the image to see the higher quality image).

Wikipedia has a lot of broken links and since my copy of it will always be a little bit out of date, I will probably keep receiving spam like this until the end of time.

Security

Turns out that the version of Kiwix that you can sudo apt install in Ubuntu Server 22.04 LTS is really out of date. I learned about that when CERT-EE reached out to me about an XSS vulnerability that Kiwix had.

If you’re running a Kiwix version older than 12.0.0, then congratulations, you have this vulnerability!

If your distro ships the latest version (Arch Linux, Fedora Linux etc.), then you’re good to go!

If your distro doesn’t, then feel free to use the Kiwix Docker images to get around that limitation.

A very basic example command that should get you started:

docker run --rm  \
    --name=kiwix \
    -v /path/to/content/:/data \
    ghcr.io/kiwix/kiwix-serve:latest \
    /data/*.zim

2024-09-05 update: LLM bots!

Well, it turns out that Kiwix is horribly inefficient at disk usage when LLM data scrapers are hitting it at 50+ requests per second.

The following is the result of running Kiwix for 4 days. Over 8 TB of disk reads is incredibly wasteful.

CONTAINER ID   NAME                          CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O         PIDS
734d8e1b4385   kiwix                         50.28%    10.36GiB / 27.27GiB   38.00%    7.73GB / 94.7GB   8.45TB / 12.9GB   7

I’ve decided to finally take down my instance of kiwix.ounapuu.ee as a result.

My experience at the k-space hackerspace hackathon

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Thu, 03 Aug 2023 06:00:00 +0300

k-space is a hackerspace in Tallinn, Estonia, and they organized a hackathon recently. Unlike in most hackathons, in this one you could whatever you like with no expectations about building a business or coming up with an MVP, just come on down to the space and start working on your passion projects!

I had some ideas on what I could do, but a few of them would have required access to workshop tools, for which I did not have a key. I could always get one, but was too lazy to bother with it. I looked at the list of ideas that had been written down by other hackerspace members and picked the one marked as [easy]: Wire up electronic lock for workshop.

If I achieve that goal, then I don’t have to worry about getting another key, and I could also get access to all the workshop tooling! And since it was marked as easy, I figured that I could do it and have some time left over for playing around with things like the laser cutter.

Impressions

Before I get to the technical part, I’d like to say a few words about the hackathon and the organization itself.

The highlight of the hackathon was the immense sense of accomplishment that I felt. I work as a software developer at my day job, which often involves dealing with incidents, migrating services from virtual machines to Kubernetes, and sometimes from Kubernetes to Kubernetes, and anything else that comes my way. The issue with most software teams is that the products that you build and maintain are often disconnected from the real world. You don’t see the physical impact of the work that you do, and that can leave you feeling empty. Sure, you spent days switching a service to Kubernetes, but you don’t feel like you did anything at all.

During this hackathon I had the opportunity to build things with my own hands, and the door controller software doing its thing resulted in real-life actions. You swipe the card, it beeps, and you can hear the electronic lock being opened. That, and the opportunity to work on a project involving a Raspberry Pi and mounting it so neatly was very satisfying.

Software developer at work. Photo by Arti Zirk.

The hackathon crowd itself was not that big, hovering around 15 or so people. We even had guests from Germany who were really helpful, great to talk with and they provided their expertise to help out wherever possible. I loved how the collaboration worked for the rewrite of the door controller software to the Go programming language. One guy knew all about the original software and how it works but was just learning Go, and the other guy was just visiting the space and had extensive Go experience.

Although this post is mainly about my own experience and the stuff that I worked on, there were plenty of other cool projects going on during the hackathon as well. One that I really liked due to how practical it was is “the kegerator” that provided cold carbonated mate tea throughout the hackathon. It lives in an old fridge and I think the execution of it was really neat! Here’s the livestream recording where other projects are also covered.

There were plenty of refreshments available and I took full advantage of that because if there’s one thing that can get my brain working on overdrive, it’s copious amounts of sugar and caffeinated beverages. Oh, and we got pancakes as well! Peetri Pizza really blew it with the pizzas though, never seen pizzas that are this sad.

Vibes on day 0.

There were some small issues, such as the streaming setup crapping out at the last day during the part where we presented our work, but overall I was happy with how the hackathon turned out.

For future hackathons we probably need to do a bit more marketing and think about the dates more so that we can get more enthusiasts on board. Having our German friends around really added a lot to the event and I would like to see more knowledge-sharing between communities in the future.

Build log

Here’s a rundown of what I did during the hackathon.

There’s also a video where I talk about my hackathon work!

Whenever you see the name Arti, then that refers to my good friend who helped out a lot during the hackathon.

Day 0

The hackathon began on 27th of July at 18:00.

Everyone presented their plans and then the work begun.

I used the first evening to put together a rough plan and figure out what needs to be done. In short, the plan was like this:

buy some quick splice terminals
assemble the Raspberry Pi with the custom HAT that is used to interact with the card reader and electronic door lock
install the card reader on the door
run the networking and power cables
connect all the cables to the Pi
set up the card reader software
test in production

After taking some pictures of the door lock test setup that another participant assembled, I decided to go and rest for what’s to come.

Door controller test setup. Photo by Arti Zirk.

Day 1

This is the part where all the action started.

I began by taking stuff apart.

After making some additional holes I ran the cable that connects to the electronic door switch. That was easy enough.

Then I began thinking about where the Raspberry Pi door controller setup would go. The workshop is a very dusty place and it would probably need some kind of protection from it. The first idea was to custom design a 3D printed case or customise an existing design, but after a short discussion and brainstorming we realized that getting a plastic box big enough to house the Pi and all the wiring makes more sense.

Me and Arti set off to find something like that in the hackerspace. After going through lots of boxes and shelves, we stumbled upon two metal boxes. Those boxes used to house some kind of security equipment and had inside circuitry that converts 220V AC to 13.6V DC, which was ideal for our use case. It also had leads to connect an UPS battery, which was even better!

Oh, and we also found this fun little thing.

Anyway, I went ahead and drilled some mounting holes for the Pi. We had M2.5 nylon screws and standoffs available at the space, and with some assistance from Arti I learned about the wonders of countersink drill bits.

I really liked how the mounted Pi looked. I’ve seen a ton of projects on YouTube and elsewhere with Raspberry Pi-s being modded and mounted in all sorts of ways, but doing it myself felt really good, especially since I was reusing an old metal box that had no other purpose.

I set up the microSD card with Raspberry Pi OS and figured that I could find it on the k-space network. However, I then remembered that the internal network was /16, which made the process much longer. I did not have access to the network switch, but I did end up getting acces to a Pi, it just wasn’t the one I just set up. That was a fun discovery.

I decided to go with the easy route and connect the Pi to an actual monitor and keyboard. That process ended up being faster than scanning the network for Pi-s.

I also added an UPS battery to the setup. It barely fit the enclosure and it had to be raised a bit so that the lid on the box would close properly. But it worked out really well!

By the end of the day we also had the card reader installed on the door. We contemplated putting the reader on the other side of the glass because the reader can read cards up to 9cm away, and quick tests showed that it would actually work, however we had concerns about mounting it that way, and it would look a bit too janky even by k-space standards.

Day 2

Day 2 started with me trying to figure out how to run the cables. Making a suitably sized CAT5e cable was easy and it worked on the first try and drilling a hole through plywood was also simple, but when trying to figure out how to run the power I ran into a bit of trouble. There was very little room in the junction point that I intended to use, and when my brain shorted out I asked Arti for some help (again). He figured out a way to run the additional cable while not interfering with the power cables that ran to the light fixtures.

In hindsight, that first aid kit didn't really have anything that would help with an electricity-related accident.

Due to the way we decided to run the card reader cables, we had to extend the cable that comes from the card reader. For that I used a CAT5e cable and some quick splices. I later learned that using your thumb power alone is not enough to splice cables, you actually need to push them all the way in with plyers. Once that was figured out, me and Arti installed a very professional looking cable cover involving a random piece of plastic tube, zipties and screws. Not that I needed the help, Arti was just working on porting some Python code to Golang and wanted a break. He ended up making a lot of breaks, not sure what that says about the learning curve of Golang.

Now came the difficult part. I had to wire up everything to the Raspberry Pi without breaking anything. I had the photos I took on day 0, but in hindsight they were not that great since I had trouble figuring out what went where. Luckily another door controller in the space had a similar setup and I could use that as a reference point.

The Raspberry Pi HAT gets 12V input power, some of that is passed to the door controller, and then we have two leads that are connected to both the door controller and the electronic lock. If someone swipes their card and they have access to that particular door, the card reader light goes green and 12V is applied to the electronic door lock, making it open it. There’s software that controls it, but I did not have to worry about that too much because two other hackathon participants were working on that.

I had Arti check the cabling before the initial test and after a hesitant go-ahead from him we powered it on. The Pi powered on, but the card reader was not doing anything. It was supposed to do a few beeps and show a red light. Uh-oh. That’s when Arti reviewed the quick splices that I used and pointed out that they were not pushed all the way in. After using brute force (pliers), we powered it on again and it worked!

The rest of the evening was spent testing out the card reader software that was being ported. That revealed a few issues, but we did get a successful door opening!

At this point I was very happy and I think half the hackerspace heard that.

Day 3

At this point I was quite tired, but I felt really accomplished. I assembled the setup without burning everything down, and I only shorted out the UPS battery once after absent-mindedly connecting the same alligator clip to both power leads. Whoops.

I spent that half-day cleaning everything up and mounting the power and networking cables to the wall. I also labeled the box so that we know what that mysterious white box is and where it gets its power.

At 14:00 the hackathon timer ran to zero and we were wondering what would happen with the countdown. Well…

I presented my work, listened to what others were up to, and then we got some creative awards. Arti, the guy working on the Golang port of the door controller software, got “K-SPACE Certified Go Developer”, and I am now a “K-SPACE Certified Licensed Electrician”. Whoo!

I’d like to thank everyone who organized the event, and Arti Zirk for helping out with the project on many occasions!

That was pretty much it!

Left: happy hackathon participant. Right: the guy responsible for organizing it. Photo by Arti Zirk.

How I blew up my backup server (Valve pls fix)

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Sat, 10 Jun 2023 12:00:00 +0300

It all started with me getting a Steam Deck.

RIP the Ubuntu Server installation on my LattePanda V1.

Background

After getting familiar with the Steam Deck and how the Proton compatibility layer works, I decided to write a backup script that would back up everything in the home folder, excluding the Steam games themselves due to the sheer girth of modern games (how the hell has GTA V ballooned up to 100+GB???).

Among the folders that I backed up was compatdata, a folder that contains files that Proton needs to make Windows games run. If you browse the folder, the contents look like a mini-Windows installation, and among those files you can also find your savegames. It made sense to me to back up this folder, and I was quite happy to know that no matter what happens, my game saves would be safe as long as I made a copy of this folder.

Fast forward a few weeks: I’m doing some changes to my self-hosting infrastructure and decide to redeploy changes to a backup server that I have. Nothing fancy, just Ansible roles that make sure that the backup server has some configuration present and that the backup folder has the right permissions.

The step looks something like this:

- name: Fix permissions
  ansible.builtin.file:
    name: /path/to/backups
    state: directory
    owner: user
    group: user
    recurse: yes

Everything was okay until I saw this.

Uh-oh.

After trying to recover the backup server and giving up as soon as I found that I cannot log in as root or use sudo, I gave up on doing that over the network and decided to get physical acccess to the server. One reinstall and Ansible run later, the server is okay again.

Investigation

How did we end up here though?

The backups from the Steam Deck are made to my home server using rsync. The Steam Deck is just a Linux machine, after all, and it made the most sense to me as I could make a backup that preserves all the permissions and links. Should I ever screw something up, I can run the same script in reverse and have everything working as it used to. I would hate setting up all my games and customizations again.

The script looked something like this.

rsync -aAXzv --delete-before /home/deck/ backupuser@myserver:/path/to/steamdeck/ \
  --exclude .local/share/Steam/steamapps/common \
  --exclude .local/share/Steam/steamapps/downloading

sudo shutdown now

Free tech tip: you can create a desktop entry for any script, add it to Steam, and run it within the Steam Deck UI when you’re done playing. You’ll have a backup and the system will shut down automatically once it’s done!

Nothing suspicious about the script, right?

There’s one problem with it: rsync also sends over links. These come in various types (symlinks, hard links) and act as a pointer to another file or folder. Turns out that Proton (or Wine) loves using symlinks. Most usages I saw were pointing towards common distributions of Proton, which makes perfect sense since it saves disk space.

There are also links that point to the root folder /. The use case for these seems also makes sense: present the game with a drive like Z:, and point it to / on Linux side, and now the user can easily navigate their whole file system within the context of the game they’re running. Maybe you want to install the game or its add-ons into a different folder, or maybe you want to load a save game that you have somewhere on your Steam Deck.

Just one problem with this: what happens when you have this symlink on another machine, such as my backup server, and you use Ansible to set the permissions for a folder containing this symlink, recursively? And what if the follow setting in Ansible is on by default since Ansible 2.5?

Turns out that the answer is that you’re going to mess up the file permissions on the whole machine, and most things stop working at that point. I could still login to one user on that box, but I could not do anything that would help recover the state of the machine at that point.

The fix (?)

There are many options for avoiding this problem, or at least working around it. I’m not sure what the perfect fix looks like. In case you know one, let me know.

One thing I added to my backup script was --no-links, which instructs rsync to not copy symlinks. That setting has to appear after the first batch of arguments (rsync -aAXzv) because otherwise it will be overridden by the -a parameter.

I also updated my Ansible setup to avoid setting the permissions for the Steam Deck backup folder as a precaution.

The major downside with my tweaks is that I don’t really have a “full” backup of my Steam Deck anymore. I do have backups of my game saves, but recovering from the backup will become a bit of a hassle due to me having to find and copy the game saves manually to the new compatdata folders since the backed up ones are missing all the symlinks to Proton/Wine-managed dependencies.

Alternatively I can consider using a backup tool like restic that should preserve symlinks. I’ll just need to also test recovering from that backup method.

Either way I can’t really blame Valve, Wine or Proton developers for my own fuck-up. They used symlinks in a way that allowed them to save disk space and give the user easier access to their files in-game. It’s just unfortunate that I learned about this setup the hard way.

Completely unrelated to my issue, but I can’t help but remember that one person whose machine got wiped by Steam.

I held a talk about my self-hosting adventure!

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Fri, 26 May 2023 06:00:00 +0300

There are two reasons why I haven’t written much lately, and one of them is that I was cooking up a talk about self-hosting, more specifically my own self-hosting adventure (the other one is the Steam Deck, more about that in the future).

That's me!

The talk took place on May 25th 2023 at k-space, a hackerspace in Tallinn, Estonia.

This talk was livestreamed and you can watch the recording by clicking on this link.

Dell Latitude 5411: the Linux compatibility sweet spot

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Fri, 28 Apr 2023 06:00:00 +0300

Oh boy, here I go testing a new laptop again!

Well, it was new back in 2020.

You might remember my article on why I went back to a ThinkPad T430 in 2022. Or that other time when I got a new HP laptop for testing and got so frustrated that I wrote about it.

Well, today I’m writing about the Dell Latitude 5411.

It’s not the newest laptop in the world, but I decided to give it a go because of a few reasons:

nobody wanted one at work due to the reputation of this laptop (sounds like a jet engine)
it’s almost 3 years old at this point, which is the Linux compatibility sweet spot (in theory)
there were some changes coming to the office, which meant more flexible seating arrangements and everything being designed around USB-C, and I wanted to continue my employment at the same company which might have been tricky with a ThinkPad T430

And that’s how I ended up with one.

The Linux compatibility sweet spot

If you have spent any time using new hardware on Linux, then you probably know how painful it can be. You might get lucky and receive hardware that has support right from the get-go, but in most cases there are issues around that may or may not get fixed with time.

If you take the same device a couple of years later, it’s highly likely that the Linux support on it will be much better. The price of the device will also be much lower than before, especially if you get it on the second hand market.

Any machine that falls into this time window is what I like to call “the Linux compatibility sweet spot”. This Dell happens to be one such machine.

Here’s a quick rundown of some specifications:

CPU: Intel Core i7-10850H 6C/12T, 2.7 GHz/ 5.1 GHz
RAM: 32GB DDR4
SSD: 512 GB NVMe
OS: Fedora Linux 38
Thunderbolt support
Wi-Fi, Ethernet based on Intel chips

The Dell Latitude 5411 is powerful, and the components in it seem to be working just fine under Linux. The USB-C port has not had major issues in a work environment, and it also works tolerably with a Dell WD19TB Thunderbolt 3 dock. My ultrawide monitor works, the Ethernet port on the dock can push gigabit speeds, and USB 3 ports work as expected.

Honestly, that’s pretty much it regarding compatibility. It just works most of the time, and that’s why I like this machine.

Oh god, the noise

Not everything about this machine is ideal, though.

One of the first things I noticed with this machine is some kind of electrical noise. Some call it “coil whine”, but all I know is that it’s this high-pitched squealing that’s coming from the laptop. Based on some searching and personal testing I can confirm that the noise is related to the CPU switching between idle and really high turbo boost speeds very quickly. If you remember the spec sheet, you’ll know that this CPU can reach 5.1 GHz. The noise is apparent whenever there’s a load on the CPU, and it’s probably the sound of the power delivery components screaming in pain to deliver all this power that this furnace of a CPU requires.

There is a workaround to the electrical noise: try running the CPU at a lower speed. You can do that by disabling Intel Turbo Boost, for example. There’s an UEFI setting for that, but on Fedora Linux 38 there’s an even easier option in the form of the “Power Profiles” setting. When running the CPU slower, the noise is much less severe.

Power Profiles in GNOME 44.

This machine got the reputation of being a jet engine at work, and I agree with those people as this thing can be damn loud even when it’s seemingly not doing much. This is a result of the CPU demanding a lot of power and the cooling solution not being able to catch up. I feel that the stock fan curve is quite aggressive to prevent any reliability issues after long-term use.

If you have the same laptop and are looking for a fix to that issue, then you’re in luck! There are plenty of solutions out there and I am personally a fan of this Arch Wiki section. I use dell-bios-fan-control to be able to control the fan speed myself, and a simple script to fine-tune the behaviour of the system. The script looks something like this:

#!/bin/bash

set -e

echo "Finding device path."
device_path=$(echo /sys/devices/platform/dell_smm_hwmon/hwmon/hwmon*)
echo "Device path: $device_path"

echo "Testing existence of paths."
cat "$device_path"/temp1_input
cat "$device_path"/pwm1
echo "Testing existence of paths done."

echo "Disabling Dell stock BIOS fan control."
/root/.local/bin/dell-bios-fan-control 0
echo "Done!"

while true; do

  temp=$(cat "$device_path"/temp1_input)

  # Only two fan speed levels seem to exist, anything above 200 is basically
  # equivalent to full speed.
  if ((temp > 50000)); then
    echo 64 >"$device_path"/pwm1
  else
    echo 0 >"$device_path"/pwm1
  fi

  sleep 15
done

The CPU is already temperature limited even in the stock configuration and will throttle under load, which should mean that running the machine at a lower fan speed is just fine. Let’s hope that I’m right.

The damn USB-C port

While the USB-C port generally works well, I do still get annoyed with it. When connecting it to my Dell WD19TB Thunderbolt 3 dock, there’s a good chance that it will do nothing but charge. Restarting the machine with the dock connected will usually work, but it does take away from the portability of the setup.

I’m convinced at this point that USB-C is a cursed standard and will never “just work”. If you disagree, then please look up any article that tries to explain how USB-C works and all the things you have to keep in mind if you want to buy a damn cable for your device that does what you want. And then add Thunderbolt to the mix.

If this aspect starts becoming very annoying, then it might become a dealbreaker for me, but right now it happens infrequently enough that I can ignore it.

Oh, wait, it happened again while writing this article. Damn it all.

Other notes

Here’s a rundown of some observations that I’ve made while using this machine:

you can actually boot from the microSD card, which is nice
if you dual boot Fedora (two completely separate installations), then you’ll get two Fedora entries in the boot options list, and although the names are identical, you can set a preferred boot order just fine
keyboard and touchpad are not the best things out there, but they are at the very least tolerable
cannot say the same about the clit mouse, it’s not that good
battery life is what I expected from a laptop that’s not an Apple M1/M2 based laptop: 2-4 hours of use when doing any real work, and up to 8 hours if your work involves staring at the screen with the brightness set to low and Wi-Fi/Bluetooth turned off
lowest power draw I observed with upower was around 6 watts
this machine has enough power to be a solid desktop replacement for software development
the above is true even when you switch turbo boost off, you still have 6 cores running at 2.7 GHz!
never managed to overheat the machine, even if I accidentally forgot to turn the laptop fan back on while running heavy loads (CPU throttled to around 2.2-2.4 GHz, which is impressive!), or when I accidentally put it in my backpack while it was still running
“power on with USB-C connected” or any other UEFI setting with a similar purpose should be avoided, you will end up putting it into your backpack while it’s running
this laptop has a full sized Ethernet port, which is incredibly handy for many IT professionals
it even has the old Dell barrel plug charging port, which is great for situations where you’ve damaged the only USB-C port on the laptop
I don’t have too much faith in the construction of the machine, some parts feel very plasticky and might not hold up well 5-10 years from now
I was the one who actually made the recommendation to purchase this particular laptop model at my current job back in 2020 due to the fleet of ThinkPads being compatibility nightmares with the monitors and docks available, and in that regard it was an improvement. Shame about the fan and electrical noise.

Conclusion

Will I end up switching to a modern laptop now? Maybe. I’ve done so for my day job already, but the jury is still out on whether I might end up with this particular model for my personal use (especially because of the damn Thunderbolt dock issue). There are alternative laptops out there with similar pricing that are also in this Linux compatibility sweet spot.

Should you get one? If the price is right and you don’t mind applying the workarounds for the issues I mentioned, then go for it!

I have restored some faith in modern laptops now. They all suck in different ways, but this one is not the worst one, and that’s good enough for me. Probably. I’ll keep my ThinkPad T430 in an easy to reach spot, just in case.

Life is maintenance, maintenance is life

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Thu, 20 Apr 2023 06:00:00 +0300

Over my relatively short career (6+ years), I’ve noticed a change in the way I approach building things. When I was still an inexperienced junior developer who barely survived operating in a Linux environment and saw backend development as a black box, I was happy to get things working at all.

Nowadays, no matter what I do, I have to take maintenance into account.

In software development, having to account for maintenance means being picky about what dependencies to include.

Do the developers of the dependency have a good track record?

Does it still receive regular new releases?

How often does it break compatibility with new major releases?

Would it be possible to avoid including the dependency by going with another solution, such as writing a small amount of code yourself?

Is this project maintained by a single person?

Or solely by a VC-backed company that is burning through cash fast?

What if that company has layoffs or goes under?

If you ask yourself these questions, you’re probably going to make a choice that’s at least somewhat informed, and you might be able to avoid some pain later on.

It’s a gamble

But sometimes you just can’t win.

When working as a junior developer, I was happy to add yet another dependency to the project because it solved that one particular problem I was working on without me having to worry about implementing the functionality myself. I really didn’t give much thought to what would happen to that dependency years from now.

After working in multiple software development teams, I know how painful it is to get a neglected software development project back on track (hi, T3!). Choices made in the past (and not necessarily by you) can interrupt your team and put brakes on any other development that you were planning on.

One relatively recent example comes to mind. I wasn’t working on this change myself, but saw how it affected the team. There was a frontend project based on React, Redux and a bunch of other pieces that were thrown together. One of those dependencies was a library that was used to render all sorts of tables. There was just one little issue with it: that dependency was now unmaintained and blocking updates to other dependencies in the project, like React. One abandoned library that was utilized in a good chunk of the project meant that another solution had to be chosen and all existing usages had to be ripped out. Given the poor technical state of the project in general (and that’s putting it nicely), that was no easy task, and many weeks were spent on trying to get the house in order. Eventually the team succeeded, but the cost was high, all because a choice made in the past did not pay off.

You might do some basic research, consider the ups and downs of all the choices, and still make a choice that will haunt you or the next developer. Sometimes it’s simply a coin toss. Life happens, and developers are not immune to it. The developer might be working on that one library as a passion project, but decide to quit after burning out. Or they get other priorities, like starting a family, or they decide to switch careers entirely. Or they might have simply passed away. Such is life.

The issue with tutorials

If you’re looking to solve a particular problem, you’re probably going to go straight to your search engine of choice and throwing in some related keywords.

Looking at what’s out there in terms of tech often results in browsing through search results and a bunch of tutorials. Those tutorials are often focused on the bare minimum: here’s what we’re setting up, here are some commands that you need to run to get there, and et voilà, you have something that works!

When was the last time you saw a tutorial that also focused on the maintenance of that solution?

If you’re new to this industry, maintenance is not something that you necessarily think about as you’re most likely focused on getting things to work in the first place. Building things, exposing them to the world and then ignoring maintenance is one of the many reasons why the software landscape is in shambles. Best case scenario, the world around your little service updates and introduces outages that you could have avoided with a little love and care. Worst case scenario, you get in the crosshairs of a bot that’s scanning the whole Internet for vulnerable services, and your little service might be following orders from adversarial countries and taking part in cyberattacks.

If you’re setting something up, think about what the maintenance will look like, and try to estimate the amount of time you’ll have to spend on it. If you can automate a big part of maintenance, then that’s even better.

You should still build new things, but you can’t ignore maintenance.

If you’re someone who just realized that they have this one web service running on a VM somewhere in the cloud with uptime measured in years and no automatic updates being applied, then I guess you now know what you’ll be working on this week.

Not Just Software

Maintenance applies elsewhere as well.

You decide to buy a smartphone, and you care about using the hardware for a long time, perhaps 5+ years. Perhaps due to environmental concerns, or due to smartphones being ridiculously expensive in 2023.

This means that you probably want one where you can replace the battery with a reasonable cost, since that’s considered to be a consumable item in electronics and it will eventually stop holding a charge.

Which option would you go with?

Apple iPhone 14, battery replacement takes 1-2 hours and specialized tools
Google Pixel 7, again, 1-2 hours and specialized tools needed
Fairphone 4, 30-60 seconds to replace the battery, no tools needed if you can pry the back cover up with your fingernail

From this perspective the Fairphone 4 is the obvious choice. You will have to consider other aspects as well when picking a smartphone, such as the camera quality, performance and any other features you expect from it, so it’s not always about maintenance, but it can still play a big part in the decision-making process.

Or let’s consider a laptop instead. You can cheap out and get one with the lowest price from the store, but it’s unlikely to last more than a few years before it has issues, and repairing it might not make sense due to lack of spare parts or the repairability of the device being a nightmare. If you care about longevity, you can instead opt for an used business-class laptop that has better physical construction and lots of affordable spare parts available for quite a long time.

If you can forego having a car, then that’s a load off your back, since those hunks of metal tend to attract all sorts of problems and require expensive maintenance. Trust me, I have one made by a certain German car manufacturer notorious for having expensive repairs, and it’s not fun.

When choosing a place to live, consider that a bigger apartment or house will have a larger surface area for problems to exist. More rooms, more area, more things that need attention and repairs.

If you start looking at the choices you make through the lens of maintenance, you can reduce the amount of time and money that you’ll later regret having to spend due to choices made in the past. Any new object you acquire will likely need maintenance for it to last for a long time. Are you willing to put in that effort? You’ll be paying either way, with your money or your own time, or both.

Free tech tip to developers

If you’re having trouble explaining the need to perform regular maintenance in your software development project to the business side, then try using an analogy. Most people probably would not want to work in an office with broken windows, a leaky roof and the heating system malfunctioning, so why should your software get treated differently?

Conclusion

Maintenance is something you can’t ignore. It will catch up with you eventually, and it will not let you choose the place or time for it. If you make conscious decisions, you can reduce the burden and avoid some of the pain later on.

Tiered storage: use the right tool for the job

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Sun, 26 Mar 2023 06:00:00 +0300

Hard drives are still the default choice for many homelab and data hoarding enthusiasts. They still hold the gigabytes per dollar advantage over SSD-s (for now), and if you buy the big external drives and take the drives out of the enclosures, you can get a pretty good deal.

Spinning rust has one obvious downside: it’s slow, both in maximum transfer speeds and in latency. For most use cases this is fine, but if you run a service that depends on a database or if you have a lot of clients relying on that storage, you’ll find that hard drives just won’t cut it.

At this point you probably don’t want to splurge and build an all-SSD NAS, so you look up caching solutions. Depending on your platform, you’ll have all sorts of options.

If you run ZFS, you’ll likely learn about L2ARC and why you probably don’t want to use it (hint: check your ZFS ARC hit/miss ratio first). Or SLOG, or the special metadata device. And in most cases the top recommendation is to add as much RAM as you can to the system so that you can benefit from a bigger filesystem cache. You’ll see some improvements, but not all workloads will benefit from this and you might up being disappointed after all that work.

I cannot speak for options on other platforms, but in most cases the idea is similar: buy a separate sacrificial SSD and use it as a cache drive.

Cache is (not) king

Take a moment and think about what type of data you’re storing.

Here’s an overview based on my own setup:

archived YouTube videos (bunch of big files)
cat pictures (bunch of small and big files)
services that utilize a database (bunch of smaller files)
various installation media (bunch of big files)
backups of physical disks (bunch of big files)
a copy of my Steam library (bunch of big and small files)
a web server (bunch of small files)

Most of the data I have is accessed relatively infrequently, and when it is, the performance requirements are not that high. Any hard drive based array will be able to handle streaming video or copying big bulky files over the network.

Data that is more latency sensitive, such as Nextcloud, Jellyfin (and its .sqlite DB) and PostgreSQL, take up a relatively small part of the overall storage.

In this case (and assuming that the setup allows for it physically), the solution is simple: add a smaller flash-based storage pool to the setup and use it for data that benefits from it.

I experimented with a similar setup in the past with two 8 TB hard drives holding all the big files and a pair of 250 GB SSD-s handling Nextcloud and other services. Now I had the best of both worlds.

In short, you don’t need to mess around with a fancy caching solution if your storage usage patterns are simple and predictable. If you do a little bit of investigation once, you can bypass all the logic that is usually built into various caching solutions and get a better end result as no caching solution is 100% perfect.

I looked at this site on the Wayback Machine

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Mon, 06 Mar 2023 18:00:00 +0200

Yesterday was election day in Estonia and the results are in, which is why I remembered that one time I looked at my website on the Wayback Machine.

This domain used to point to a Wordpress site that hosted content for a candidate in the local elections, at least around 2011-2014.

For international readers: “Õunapuu” is a relatively common last name in Estonia. Special characters in domains weren’t that widely adopted back then, which is how you end up using “Ounapuu”.

The site is political in nature and a quick glance at the writing reveals that the text is… interesting, to put it nicely.

After 2014, things seem to have changed as the blog did not get any new posts and at some point the domain was not renewed. Around 2017 I registered the domain because a friend recommended that I get myself a domain, because personal domains are cool. Best move I’ve made in my career, especially as I later heard from a colleague that someone else with the same last name was asking about this domain name and wanted to register it in their name.

The site has gone through some changes. It used to host a super basic page that referred to my work, and it also linked to a blog that never quite took off. It did eventually, but not in that form.

The setup that you’re looking at right now was established in 2020. I ended up taking a ready-made Hugo theme and tweaking it to my liking.

And that’s pretty much it. I didn’t expect this domain to have much history, but thanks to the Internet Archive we have records of it.

Here’s to hoping that I don’t forget to renew the domain!

TOMO M4: probably the coolest power bank I've owned

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Fri, 03 Mar 2023 06:00:00 +0200

A friend once showed me a power bank that was so cool that I had to get one myself. Ended up getting two, and here’s why.

This post is not sponsored.

Removable batteries

Let’s start with the feature that I appreciate the most: removable batteries! The TOMO M4 accepts up to 4 18650 lithium ion battery cells, which are very common nowadays and can be purchased from lots of retailers. You can also mix and match cells and even run the power bank with 1-3 cells in it.

TOMO M4 running on two battery cells.

If you’re feeling adventurous and you know what you’re doing, you could in theory take an used battery pack from a device that utilises 18650 lithium ion cells, disassemble it completely and use the healthier cells inside power banks like this one. They might not be healthy enough to power bigger devices, such as e-scooters, but they might be just enough to charge your phone. This is something I want to try out eventually with things like old ThinkPad laptop batteries since the risk of burning the whole building down is smaller with those.

I’ve previously bought cheap “10000 mAh” power banks from brands like Xiaomi and have had mixed experiences. Yeah, they usually get the job done, but should the battery die in it, then you’re probably just going to throw the thing away because the battery is not easily replaceable. That’s really wasteful.

With the TOMO M4 it’s a matter of sliding the plastic case open and just replacing the cells, similar to what you’d do for a TV remote with AA batteries. It’s so easy that even your grandma could do it!

I don’t understand why it’s acceptable to enclose consumable items like batteries in devices that are difficult or impossible to open up (the real reason is probably profits, right?). The battery will die at some point, no matter how well you take care of it.

If the rest of the device holds up, then the TOMO M4 could realistically be the last power bank I buy.

The cool factor

Now that we’re done with the most practical aspect, let’s look at the display. Once you power it on, you’ll see the green-lit LCD screen light up. The screen shows the charge level of the individual cells during standby. While charging, you’ll see a nice little animation.

But that’s not the coolest part. If you connect a device to one of the USB ports, it will show the voltage and current that the power bank is outputting. Multiply those two numbers and you can actually see the wattage that a device is pulling.

The display does exhibit ghosting, which is most noticeable during charging.

Not everyone probably appreciates this as much as I do, but I just find it to be really cool.

The downsides

There are some things I’ve noticed about this power bank that are less than ideal.

The charging port is micro-USB, which is not that great in 2023. Out of all the IT gear that I own, this power bank is the only regularly used item that charges via that port. I’d love to see this replaced with USB-C port, just for the sake of uniformity. It’s probably possible to modify this power bank and replace the charging port with an USB-C port, but I have not got around to doing that yet.

The display backlight can be turned off with a single click of the power button. However, if you’re charging a device that is close to being fully charged and starts trickle-charging, the display backlight will turn back on. This is less than ideal in situations where the backlight illumination bothers you, such as during night.

The detailed analysis that I mention in the next paragraph mentioned that this power bank “Can be used as UPS”. I put this to the test with two devices: Raspberry Pi model B+ and a LattePanda V1. The Raspberry Pi struggled a lot and the red power LED flickered constantly, indicating issues with power delivery. The LattePanda V1 booted up and ran nicely for a while, but under a bigger load it threw some errors into kernel logs and eventually locked up completely. On a functional level this power bank can probably act as a UPS, but in practice it did not work out for me. Other factors, such as the USB cable used and/or the lithium ion cells that I used in the power bank, might have also influenced the results of this quick test.

LattePanda V1, powered by the TOMO M4.

The technical details, in detail

If you’re interested in all the technical details for this particular power bank, then definitely check out this review on lygte-info.dk. It’s a very detailed overview of the capabilities of this power bank, its behaviour under different circumstances and a bunch of measurements and explanations to back it all up.

I also like the way they phrased the verdict: “I will rate it as acceptable.” Short and concise.

The author also reviews lots of other products and has the reviewing process documented in detail.

Closing thoughts

In general, I’m very happy with this power bank. Although I’d probably not recommend it as an UPS for low-power computers, I’d happily recommend this to someone who needs a simple power bank that they won’t have to throw away after a few years.

LattePanda V1 - my experience with a Raspberry Pi alternative

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Tue, 28 Feb 2023 09:00:00 +0200

As with many homelab experiments around 2022/2023, it all started with Raspberry Pi-s being either out of stock or absurdly overpriced.

I once noticed a listing for a LattePanda V1 on a local auction site. The price was reasonable at 45 EUR, and after I looked at the specifications, I had to get one to play around with.

Embrace the jank!

The auction site usually sells used items, but this LattePanda V1 was brand new in box, which surprised me. I got the 2GB RAM/32GB eMMC option.

I put the LattePanda through its paces, and here’s what I learned.

Please note that my use cases are more on the software side, I did not try out the hardware capabilities that SBC-s are also associated with (GPIO, camera/display ports etc).

The hardware

What makes the LattePanda V1 special is that it’s based off of an x86 CPU. Mine came with an Intel Atom x5-Z8350. 4 cores, not very fast ones, but similar or better to what you’d find in a Raspberry Pi 4. Most single-board computers usually come with an ARM-based CPU.

The board behaves like any normal PC. You power it on, you see the LattePanda logo, and the OS just starts up. Smashing the “Delete” key during the boot will direct you to the UEFI settings. Since this board is marketed towards the technical crowd, it has so many settings that you can play with. Definitely more than what I’d know to do with.

The board comes with Windows 10 preinstalled. I booted it up and gave it a go, and it was slow as molasses while it downloaded the latest updates. At one point I reinstalled Windows 10 and ran into issues. It “works”, but I had trouble getting any Intel GPU drivers installed on it, which rendered it almost useless for any media playback use case that I could have used it for.

The port selection is okay. It comes with 100 Mbit/s Ethernet, one USB 3.0 port and two USB 2.0 ports. I could get around 350 MB/s out of the USB 3 port with an external SSD, and the USB 2 ports pulled anywhere between 15-40 MB/s, depending on my luck. If you fancy a gigabit connection, then an USB-to-Ethernet adapter might do the trick.

Adding one external HDD or SSD over USB is okay and works well, but after adding a second one I noticed issues with either one of the disks not receiving enough power. If you intend to use the LattePanda V1 as the brains for a cheap NAS, then use drives that are powered externally.

The eMMC storage on the board is decent. Regarding performance it’s somewhere between a hard drive and a proper SSD, and the sequential read speeds max out around 150 MB/s. My main concern with it is the durability. The eMMC chip is probably hidden behind the metal shielding on the board and replacing it (if that is even possible) might turn out to be tricky. ls -lah /dev/disk/by-id/ lists the eMMC chip as mmc-DA4032, which a little bit of Googling refers to it being a SanDisk chip.

The microSD card port works, but either booting from the microSD card is not possible or the board was a bit temperamental when I gave it a go with Ubuntu Server 22.04.

The board runs hot in its barebones configuration, any decent load over a longer time period will result in the CPU throttling itself. I recommend rigging up a metal heat sink to the bottom of the board to avoid throttling and stability issues under very high load. A thermal pad, a piece of metal, and a bunch of zip ties will do wonders.

Do not look at the bend.

The Linux experience

When experimenting with Linux, I found a couple of annoyances. The LattePanda V1 seems to have a display connector on board. However, in Linux it turned out to be an annoyance as the system always presented a 1024x768 display being connected to the system. Booting into a Fedora Linux liveUSB environment will lead you to staring a blank view with only the wallpaper present. Playing with settings in UEFI did not lead to any improvements. What works is adding a kernel parameter: video=DSI-1:d. With Fedora Linux, if you apply this fix in GRUB when first booting into the liveUSB environment, it will persist after the installation is done.

Once I had Fedora Linux booted up, I did not have a good time. The display flickered when I dragged the mouse near the edges and the performance was awful. It didn’t take long for the desktop session to crash.

One of the main reasons I wanted to experiment with this setup was the low power usage of this board. The LattePanda V1 idles at around 3 watts, and under a high CPU load generated with stress -c 4 I could get it to 5-6 watts. This made it a great candidate for running this website and Wireguard VPN 24/7 at home. I ended up installing Ubuntu Server 22.04 on it and actually ran this website off of it for about a week or so.

The reason I stopped using the LattePanda V1 for that purpose was the performance. The 100 Mbit/s Ethernet wasn’t a disaster, but it did set limits to how much traffic I could push through the board. What made me reconsider was the CPU performance. Simply put: SSL encryption speeds were about 5-10x slower than on my ThinkPad T430 that ran as a server before.

My performance testing wasn’t scientific or anything, but I did run simple tests. One of the involved running curl against resources, such as an image on my blog, in an infinite loop and over multiple threads. The ThinkPad T430 peaked at around 650 Mbit/s of network bandwidth while the LattePanda V1 struggled at around 60 Mbit/s.

After looking for some other benchmarking options that I could set up really quickly, I also gave ApacheBenchmark a go. This more-or-less confirmed my findings and the requests per second results are the following:

ThinkPad T430: 301.72
LattePanda V1: 55.86

ApacheBenchmark results for the ThinkPad T430.

ApacheBenchmark results for the LattePanda V1.

The LattePanda V1 was over 5 times slower in that test. With a slow internet uplink this might not matter much, though.

I ended up putting the T430 back in service after doing more calculations. One idea I had with the LattePanda V1 was to have that run 24/7 due to its low idle power usage and have another beefy server run during certain hours. I liked that idea until I calculated the difference in power savings. Having the main server with around 12 W of idle power turn off for 6 hours and the LattePanda running at the same time would have actually increased the total power consumption of the setup compared to having the main server running 24/7.

The math will make sense if your main server uses much more power. Most desktop PC-s that I have seen idle around 35-50 W and rack-mounted servers use even more than that, so in those situations this might make more sense. Or you can whip up a solution that starts and stops machines in your homelab on-demand.

Oh, and don’t expect to run CPU or GPU based transcoding off of this thing with Jellyfin. It works okay for certain video formats and resolutions, but a 1080p H.265 video transcoded to H.264 is just not fast enough on this machine.

A suitable use case

I might change my mind at one point and give the LattePanda a go again as a low power web server. I hate unused hardware, which is why this LattePanda V1 is now serving as an offsite backup. The USB 3 port makes sense for attaching a bigger storage device and if you run ZFS, you can also send your whole filesystem to it with zfs send/receive or syncoid. I haven’t had much luck with ARM and ZFS, but with the LattePanda and its x86-based CPU I have not had issues.

LattePanda V1 in its final form: a backup endpoint.

Conclusion

All-in-all, it’s a neat little board that shows signs of instability from time to time. If you intend to just run Ubuntu Server on it and your workloads are not very performance critical, then it might work out well for you. Think of it like a juiced-up Raspberry Pi 4 to get an idea of what it might be suitable for regarding performance.

It did not fit the use case I intended to use it for, but I’m still happy that I could play around with this board. It was fun!

Shrinkflation, SanDisk style

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Wed, 15 Feb 2023 06:00:00 +0200

2023-02-16 update: this post is not about the “GB vs GiB” issue, or the “lost disk space” that you notice after formatting the device in an OS. Please read the article in full before making inaccurate comments online.

For many years, I have used two SanDisk Ultra 16 GB USB flash drives as OS installation media. Since I do not trust Ventoy yet, and alternative “one stick that does it all” solutions have had odd bugs from time to time, I prefer having separate physical USB sticks for each OS, one for Fedora Linux, and one for Windows 10.

I hate reflashing sticks all the time and would like to have at least one around that always has Fedora Linux on it, so I decided to get two more USB sticks of the same brand and size. Once they arrived, I got annoyed.

The old flash drive shows up like this in GNOME Disks.

Here’s the new one.

Although it does not matter one bit for my use case, I am annoyed that the new one is shipping with much fewer bits than the old one. I’m a whole 607125504 bytes short! That’s 579 MB, or about half a gigabyte.

The only difference that I can make out on the surface is that the new one was manufactured in Malaysia.

Other than years worth of wear and tear, they look almost identical.

But it seems that SanDisk (or their new owner Western Digital) has legally covered their butt with this knowledge base article.

Here’s the line that explains why some bytes are seemingly missing on formatted media. This is what consumers are usually concerned about once they use their storage device.

SanDisk defines 1 GB as 1,000,000,000 bytes. Operating Systems define 1 GB as 1,073,741,824 BYTES.

However, this section is the one actually relevant to my finding.

A portion of the total capacity is used to store certain functions including optimizations of the memory that support performance and endurance and therefore is not available for user storage. This is disclosed on our packaging and marketing materials when you see the statement “Actual user storage less.”

Based on this, I assume that SanDisk cheaped out and decided to not add enough flash to provide both the advertised storage of 16 000 000 000 bytes available to the user and some room for over-provisioning that flash storage benefits from. And somehow they are the only ones who do it this way, as all other storage media that I own shows up with the proper number of bytes.

And for those interested, here’s f3 output for both the old and new sticks.

Old:

# f3probe /dev/sdc --destructive --time-ops
F3 probe 8.0
Copyright (C) 2010 Digirati Internet LTDA.
This is free software; see the source for copying conditions.

WARNING: Probing normally takes from a few seconds to 15 minutes, but
         it can take longer. Please be patient.

Good news: The device `/dev/sdc' is the real thing

Device geometry:
	         *Usable* size: 14.91 GB (31266816 blocks)
	        Announced size: 14.91 GB (31266816 blocks)
	                Module: 16.00 GB (2^34 Bytes)
	Approximate cache size: 0.00 Byte (0 blocks), need-reset=no
	   Physical block size: 512.00 Byte (2^9 Bytes)

Probe time: 1'49"
 Operation: total time / count = avg time
      Read: 463.8ms / 4814 = 96us
     Write: 1'48" / 4192321 = 25us
     Reset: 1us / 1 = 1us

New:

# f3probe /dev/sdd --destructive --time-ops
F3 probe 8.0
Copyright (C) 2010 Digirati Internet LTDA.
This is free software; see the source for copying conditions.

WARNING: Probing normally takes from a few seconds to 15 minutes, but
         it can take longer. Please be patient.

Good news: The device `/dev/sdd' is the real thing

Device geometry:
	         *Usable* size: 14.34 GB (30081024 blocks)
	        Announced size: 14.34 GB (30081024 blocks)
	                Module: 16.00 GB (2^34 Bytes)
	Approximate cache size: 0.00 Byte (0 blocks), need-reset=no
	   Physical block size: 512.00 Byte (2^9 Bytes)

Probe time: 1'02"
 Operation: total time / count = avg time
      Read: 338.2ms / 4814 = 70us
     Write: 1'02" / 4192321 = 14us
     Reset: 1us / 1 = 1us

I’m not even mad, I’m just disappointed.

2023-02-16 update

This post received more attention than usual, so here are some additional details and clarifications.

Why do you not trust Ventoy?

Ventoy is a great tool from what I’ve seen online and the use case it fills does save time and resources. However, I have some reservations about it. If I had to compromise a bunch of critical systems over a long time period, then publishing a great tool and having it tamper with your OS installation media silently would be a really good pick. At this time, I don’t trust the developers of the tool enough and I don’t have the time or skills to perform repeated audits of the software every time they release a new version.

MBR vs GPT partitioning

That was an oversight on my part, I should have presented both disks with the same partitioning scheme to avoid unnecessary confusion. This has now been corrected.

Alternatives to Ventoy

One reader pointed out a physical device that can be used for a similar purpose as Ventoy: IODD. I have not tested on myself and reportedly it has some bugs and is not 100% perfect, but it seems like a neat hardware-based alternative.

I have personally tried out things like DriveDroid in the past, but that required a rooted Android phone and it was not 100% bug-free either.

Everyone knows that GB and GiB are not the same, why is this post even a thing, the author is clearly incompetent and dumb.

Read the article.

ThinkPad as a server: the follow-up

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Fri, 27 Jan 2023 06:00:00 +0200

It’s been a while since I last wrote about my ThinkPad T430 that was tasked with being a home server. After doing some experiments with my setup a couple of times, it is once again the main server for all my self-hosting needs. However, this time I have made some adjustments.

Background

The monstrosity that you’re looking at is what I’d call “the minimum viable ThinkPad as a server”. To be fair, you could go even more minimal by ripping out everything but the motherboard and the drives, but for my use case this is how far I was willing to go.

In my previous post about this setup, I mentioned that the T430 suffered unexpected shutdowns under certain situations, which usually involved running the fan at the lowest speed, moderate to high CPU load, and a lot of storage activity. It wasn’t the CPU as the temperatures were way below the critical point, so I suspected either the motherboard or the storage running hot and tripping some sensor that I wasn’t aware of.

For a while now, I have been thinking about running the T430 as bare as possible. One idea involved taking the bare motherboard and the storage and slapping it on an acrylic sheet with some stand-offs, which should give it the best chance at cooling. Unfortunately I didn’t have enough time to commit to this, and that setup would have had some downsides as well. At some point an idea popped to my mind: I could probably achieve a similar result by stripping everything but the bare necessities from the laptop and improve the cooling as a result. That’s how I ended up with what you just saw.

T430, stripped

I ended up removing the following from the T430:

palm rest
smart card reader
Bluetooth chip
WiFi chip
the whole display assembly

This was in part motivated by the fact that those parts are the ones that are not in the best shape on my main ThinkPad T430 that I use as an actual laptop. The other part was this nagging feeling that the most powerful PC that I own (the ASRock DeskMini X300) was woefully underutilized as my home server.

The keyboard had to stay because the power button is part of the keyboard assembly. There are ways to power on the laptop without the keyboard:

put the laptop in a dock and use the power button that is present on the dock
short a specific pin on the keyboard connector to ground

I do have some extra ThinkPad docks around that I could have used for this purpose, but it would have made the setup bulkier, which is why I did not go for that, at least not yet.

There is also an option in UEFI settings that can be used to turn the laptop on by connecting the charger (“Power on with AC attach”), but that only seems to work if you have a battery connected and after you have turned the laptop on manually once. The same caveat applies for Wake on LAN functionality. If the laptop overheats or it runs out of battery completely, then you need to manually power it on.

The cooling situation

Removing the palm rest reveals something that might be related to the unexpected shutdowns that I observed with this setup previously.

Under this area is a chip on the motherboard that seems to generate a modest amount of heat. While the machine is running, it gets warm to touch and gets hot if you throw a high load at the system, especially if you include storage in the mix. With the laptop lid closed and the palm rest being present, the cooling situation for this area can be quite bad indeed. Now that the palm rest is removed, this area of the laptop is now exposed, which should help with the cooling for this chip.

Having the keyboard connected in this state isn’t too bad. It does restrict the airflow a bit, yes, but at least it can be fastened using two screws, which means that it won’t move around and cause any other issues.

With this setup, the cooling has improved noticeably. Unfortunately it didn’t come to my mind to make a before/after comparison to better illustrate this, so I will just write about how well it operates in this configuration.

Currently, I have the laptop fan controlled by a modified version of the minimum viable fan control script. The fan only kicks in after the CPU reaches 60°C, ramps up to a moderate speed at over 85°C, and lets the default fan behaviour handle everything if the temperatures reaches over 95°C. This results in the fan being off completely for most of the day as the server is idling at around 5-10% CPU usage and temperatures being anywhere between 45-60°C. When there is a workload on the system that taxes the CPU, the fan will kick on with a small delay.

I’ve also disabled turbo boost on the Intel i7-3632QM that is currently in this T430, which limits the clock speed to 2.2 GHz max. Performance takes a small hit, but the CPU is now running much more efficiently. This is one of the tweaks that allows the laptop to be near silent. At a full 8 thread CPU load using stress -c 8, the CPU core temperatures peak at 70°C at 22°C ambient temperature, with the fan running at around 2600 RPM.

As a result of all of this, the T430 is very quiet while using about 10-12W of power while idling. Under the maximum CPU load generated by stress, the total system power usage is around 34W.

Putting the “paste” in thermal paste

Originally, the T430-as-a-server ran vertically using a basic laptop vertical stand. Back when I concluded the initial test in 2022, I disassembled the machine and noticed something very odd with the thermal paste.

Based on this image and the fact that I ran the machine quite warm most of the time (CPU temperatures around 60-80°C), I assume that the excess thermal paste started slowly dripping down. The part that concerns me the most is that the top of the chip was barely covered by the thermal paste. Either that part of the chip had great contact with the heatsink, or the thermal paste drooped down with time and left it mostly uncovered. It could explain some of the issues, but I’ll avoid making any definitive conclusions from this alone.

To avoid similar issues in the future, the laptop is now on a stand at about 60° angle. Not sure if it will work, but it’s worth a shot.

Closing thoughts

Overall, I’m happy with how this experiment turned out. Cooling is no longer an issue, and the T430 that I initially got for spare parts is actually being useful. The amount of performance this machine offers while using very little power seems to be a good balance for the workloads that I run on my home server.

Anything's a portable speaker if you're brave enough

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Mon, 02 Jan 2023 07:00:00 +0200

I hate buying things that are single-purpose, which is why I ended up with this setup.

Take a speaker, a battery, put them together, and what you now have is a portable speaker. Since I had access to both, I felt no need to buy a separate portable speaker for use in social events.

The fact that this JBL soundbar supports Bluetooth was what sealed the deal. Just make sure to keep the setup a fair distance away from any bigger bonfires, liquids and bugs, and you should be good to go.

About the time I used Google Drive as a CMS for a web app

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Mon, 12 Dec 2022 07:00:00 +0200

Near the beginning of my software development career, I worked on a pretty standard web application. The project was not a commercial success, but it did give me a good technical foundation that turned out to be very useful for my career so far.

For those interested, the tech stack was relatively boring (in a good way) with the backend service written in Java and frontend written in Angular 2+.

One of the most memorable aspects of that project was how we built the news/blog section. I might have forgotten some of the specifics, but the high-level idea should still be accurate.

The implementation

It all started with a need to be able to display articles written by the company. By that time the rest of the product was mostly in place and the section meant for articles had a simple placeholder.

I’m not completely sure on how we even landed on the choice that we eventually ended up making. I do know that the solution had to be simple and convenient enough to be used by non-technical people. Running a separate service, such as Wordpress, just to be able to post articles was probably not a reasonable choice.

The idea for the CMS implementation ended up relying on Google Docs. The person writing the articles would create a new document in a specific Google Drive folder. The title of the document would be used as the title of the article, and the content would be shown as-is on the web page. To support localization, there was one subfolder for every language as well.

The backend service would periodically query the contents of the shared Google Drive folder. If new posts were found, it would export them into HTML and store them on the backend. If I recall correctly, it might have also done some sanitization or tweaks to overcome some limitations related to this setup.

The frontend would then load the latest post and show it on the landing page. There was also a view that showed the full list of articles. By default, Angular 2 would strip out all of the styling in order to sanitize the data, and it does make sense for most use cases, since you probably don’t want an user named Little Bobby <script>alert('lol');</script> Tables causing trouble. You could work around that by using a functionality that turned all of that off. Unless you had malicious users writing the content, you should be fine. And if those people had access to the Google account hosting those documents, then you probably had bigger issues anyway.

If you look at this solution from another angle, it makes a lot of sense. What we ended up doing was using Google Docs as the editor and host for any content that we wanted to show. You don’t need to implement an editor into your back-office interface if you rely on a big company that has done all that work for you. No need to retrain users as well, just point them to the folder and let them work with the editor that is likely already familiar to them.

I cannot say much about the long-term viability of this solution, as the project ended up being shut down not long after. Based on experience gained ever since, I’d reckon that this solution could fail in all sorts of fun ways:

Google changing the Google Drive API
Google automatically banning your account because its “AI” deemed your content to be in violation of some policies
Someone accidentally deleting the shared folder or contents within it

Conclusion

I’m not sure if I would consider something similar in a future project. Implementing a good chunk of this as a junior developer was pretty fun though. It also serves as a good example of creativity and not reinventing the wheel when it’s not necessary.

I am happy to see that others have come up with similar solutions since then. If you look up “google drive cms”, you’ll find some relevant results:

While I can’t say that the solution we built was the first of its kind, I do feel validated that other smart people have also come up with a similar solution.

What changing the CPU on a laptop looks like

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Fri, 09 Dec 2022 07:00:00 +0200

This post illustrates something that is not common on modern laptops: changing the CPU.

The CPU we're replacing in the T430: a 45W Intel i7-3820QM.

Loosen the CPU by turning the screw with a flathead screwdriver.

A CPU socket in a laptop.

Make sure the golden triangle aligns with the marking on the socket, and pop that new CPU in.

A bit of 98% alcohol goes a long way.

Background

One of the modifications that you can do to a ThinkPad T430 laptop is changing the CPU to a more powerful quad core model. That’s exactly what I did back in 2017, but due to the poor availability of compatible CPU-s in my region I went for one with a 45W TDP, which is a bit much for a laptop designed for CPU-s with a 35W TDP.

You could work around this issue with software, and that’s what I also did for a long time. Since I now own two T430-s and the availability of 35W parts is better now, I decided to go for one to see how well these two options compare (result: it doesn’t overheat as often, but it managed to do it once anyway).

Unlike my T430, most modern laptops come with a soldered CPU. There do exist laptops that come with a socketed CPU, but those are not that common and are designed for very specific use cases.

Soldering the CPU right to the board has its benefits, such as the machine being thinner, and CPU-s don’t really die all that often. However, what we have lost out on is the option to upgrade older machines with more powerful components, extending the useful lifetime of the device.

Yes, you can replace soldered CPU-s, but that requires a skillset and equipment that few people have.

HoloISO: the unofficial Steam Deck experience on your PC

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Thu, 01 Dec 2022 07:00:00 +0200

If you have spent any time in gaming hardware circles, then you’ve probably heard about the Steam Deck, the Linux-based handheld gaming PC built by Valve. Yes, you heard that right: not Windows, but Linux.

I’ve had my fair share of attempts at gaming on Linux. When it works, it’s amazing. When it doesn’t, it’s incredibly frustrating. The last thing I want to do when playing a game is to become a developer and start troubleshooting issues with all the layers between the hardware and the game.

What makes the Steam Deck different is that the compatibility layer (Proton) and all the bits and pieces associated with it are integrated in a way that should make it unnoticeable to the user. Just start up your Steam Deck, pick your game, play, and you’re good to go.

Although I’ve thought about buying a Steam Deck myself for tinkering and testing purposes, I could not justify that purchase. But thanks to videos from the ETAPRIME YouTube channel I learned about the existence of HoloISO, an unofficial SteamOS 3 installer.

HoloISO

HoloISO aims to be as close to the native SteamOS 3 experience as possible. The only reason it exists right now is because Valve does not yet officially support SteamOS 3 as a distro that you can go and install. That might change in the future, but for those that are impatient like myself, HoloISO is a nice option.

To test out HoloISO and give it a good chance at succeeding, I chose my ASRock DeskMini X300. It sports an AMD Ryzen 7 5700G, which is a high-end APU and should result in performance that’s roughly comparable to the Steam Deck in most scenarios. The GPU cores are not on the newest GPU architecture released by AMD, but the higher power budget should make up for any architectural deficiencies.

The installation of HoloISO is very easy, just download the installer and write it to an USB stick. Once you’re booted up, follow the installation instructions and reboot. You should now be at the SteamOS 3 setup screen.

First impressions

I’ll be the first one to admit that my Steam library is a bit out of date, which is why my experience is limited to older titles. I can confirm that BeamNG drive and Dirt Rally 2 work fantastically though.

The only way to play GTA V on Linux without issues used to involve creating a Windows VM and passing a GPU to it. This is why I was very impressed to see that installing and starting up GTA V worked out of the box. I could even go to GTA Online and play around without being kicked out or banned!

To make choosing and playing games a better experience for casual gamers, Steam has also come up with the Deck verified program. You shouldn’t be discouraged if your favourite game is shown as unsupported or not tested, though. Try to start it up and see if it works. Burnout Paradise and Absolute Drift were not officially supported, but worked like a charm.

The unfortunate reality of SteamOS is that not every game will work out of the box. Some older games, such as GTA III and GTA Vice City, would not work at all. You might be able to check out places like ProtonDB and see if a game has a tweak that you can apply. Tweaks can involve passing additional launch options to a game or using another version of the compatibility layer (Proton).

What turned me off of this platform was Forza Horizon 4. On this setup, it did not launch properly, the sound was there but the screen was black. Based on results in YouTube, it seems like Forza Horizon 4 and Forza Horizon 5 do work on Steam Deck, so this issue might be down to a compatibility issue on my hardware or HoloISO distribution of SteamOS 3. I wanted to play that game though, so I had to eventually end the experiment and install Windows 10 again.

As someone who wants to get a good overview of resources usage during gameplay to detect and resolve any performance issues, I tend to use tools like MSI Afterburner. With SteamOS 3, you can easily achieve similar results by enabling the performance overlay. I was very happy to see that addition because that means installing and configuring one less tool on my gaming box.

Quality of life issues

Even when things generally work, I did notice some annoyances when playing games. Sometimes I ran into issues with controls. Steam allows you to pick gamepad templates and layouts, which can involve community-provided setups. Sometimes it would default to a layout that simply did not work. Turning off Steam input altogether would sometimes improve the situation. If that did not work, then picking an alternative template would do the trick most of the time.

One major issue that I ran into was related to my XBOX controllers. Because the XBOX wireless adapter does not work out of the box on Linux, I connected them via Bluetooth. However, the latency was horrible and borderline unplayable. There do exist drivers that you can install, mainly xone and xpadneo, and those do improve the situation.

Desktop mode

One aspect of SteamOS 3 that made me actually consider buying a Steam Deck is the desktop mode. Not only can you play games, but you can also boot into a normal KDE desktop environment. Install a web browser, emulators, tweak the system or do actual productive work on it, it just works!

Hell, you can even install an SSH server and manage your gaming box with Ansible!

Whenever I’ve had a gaming PC, I’ve usually hooked it up to a big screen, installed Windows and Steam on it, and called it a day. If you want to watch some media, open up the browser and you’re good to go. I’m happy to report that something similar can be achieved on SteamOS as well.

At the time of testing, there were some issues that I didn’t expect to have with desktop mode. Firefox is included as a Flatpak and it did not support any form of hardware acceleration for video playback, which made it a no-go for a home theater setup.

After browsing /r/SteamDeck for a while, you’ll notice that a lot of people are into modifying their consoles, and all of that is possible because Valve did not lock anything down. Pop the hood and work with the Linux internals as much as you want to! With other gaming consoles you’re going to have to wait until someone discovers an exploit that allows you to have proper control over hardware that you physically own.

Gaming outside of Steam

With the presence of desktop mode, you’re not only limited to games purchased from Steam.

To see how well I could emulate games that I have purchased on older consoles, such as PS1, PS2 and PSP, I tried setting up some emulators. I’m happy to say that setting them up is just as fiddly as it is on Windows, but they do seem to work quite well, at least for these consoles.

I gave Minecraft a go as well by installing it as a Flatpak from KDE Discovery software manager, but for some reason it didn’t run and crashed on startup. I didn’t look into it further, but it seemed like something that an update can fix.

GOG is a platform similar to Steam, with one notable exception: their games are DRM-free. I don’t use that often, but with a game like art of rally I had to get it from there just for that reason alone (and the developers publishing native Linux ports). Although there isn’t an official client for GOG on Linux, there do exist open source implementations. I gave HeroicGamesLauncher a go and although it is a bit rough around the edges, it got the job done and I could play art of rally just fine.

Caveats

The HoloISO experience tries to be as close to the one provided by SteamOS 3, but it is not 100% there yet. It seems that there are differences present under the hood, such as Steam Deck shipping with the root filesystem being read-only by default. I might not be aware of other differences between the HoloISO and official Steam Deck installation, so keep this in mind when making any decisions based on info from this post.

The choice of hardware will also affect your experience. The requirements regarding GPU-s is relatively strict and you’ll likely have the best experience with a modern AMD GPU since that’s what SteamOS 3 is built around. Intel and NVIDIA GPU-s may work, but are not guaranteed to.

Conclusion

SteamOS 3, even in its HoloISO implementation, is very impressive. So impressive, in fact, that I almost bought a Steam Deck. The uncertainty about support for my favourite games and less than ideal performance on the big screen were what held me back. For now.

In its current form, SteamOS 3 (and by extension the Steam Deck itself) are a tinkerers’ dream. There are some rough corners and caveats that you should be aware of as well. Not everyone is into tinkering and experimentation, and that’s OK.

I sincerely hope that Valve can use the momentum that Steam Deck has achieved to push gaming on Linux even further. After having to reinstall Windows more than I’d like to admit, having my gaming PC run plain Linux and be fully manageable with Ansible just feels so right.

I have a 'Dall UD19PB ThundeRbglt Dock': my experience with the HP Elitebook 845 G9

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Mon, 21 Nov 2022 07:00:00 +0200

I’ve decided to give a new laptop a test drive, mainly because I had the opportunity at work, and also out of morbid curiosity.

It’s the HP Elitebook 845 G9, and it has pretty good specifications:

CPU: AMD Ryzen 7 6800U
RAM: 2x16GB DDR5
Storage: 512GB KIOXIA NVMe SSD (KBG50ZNV512G)
OS: Fedora 37, kernel version 6.0.8

This laptop was also shown in a recent LinusTechTips video and praised for its ease of repair and maintenance.

However, I can’t seem to enjoy nice things. This attempt at using a newer laptop has had some frustrating challenges. Perhaps the common denominator is Linux. Or me. Can’t rule anything out.

This is simply my experience with this laptop, including my attempts to overcome certain issues with workarounds. If you find something that’s factually wrong about any part of this post, then please reach out!

Wi-Fi

It all began with the Wi-Fi card not working. The card can be replaced, but the results can be mixed.

An Intel Wi-Fi card that I had lying around would not be recognized by the system at all. The situation was fixed by the reseller who replaced the Realtek RTL8852BE chip with another Realtek one, this time with the Realtek RTL8822CE.

Yes, there exist drivers that you can install, but it’s far from an ideal solution. You will have to reinstall the Wi-Fi driver after every kernel update, and the kernel updates frequently on Fedora Linux.

The HP website allows you to pick between a Realtek and Qualcomm Wi-Fi chip. My recommendation is to go with the Qualcomm option because the Realtek RTL8852BE does not work out of the box.

Airplane mode

This is an issue I didn’t think I would ever have to worry about.

The laptop would switch itself to airplane mode at random on boot. Not on every boot, but about 50% of the time. When you try to disable Airplane mode from GNOME settings, it would flip back on again, no matter how often you tried.

To turn Airplane mode off properly, you have to use the Airplane mode keyboard shortcut, and even then it might take an attempt or two.

Some HP users suspect that the issue might be related to a botched firmware update. There’s a kernel patch in the pipeline, and some users say that blacklisting the hp_wmi kernel module could also fix the issue.

Others say that this behaviour might be related to certain events being triggered on the laptop, such as lid closing or the screen rotating. The proposed fix did not work to resolve this issue on my laptop.

Blacklisting the hp_wmi kernel module was what improved the situation for me.

USB4 and Thunderbolt

This laptop is noteworthy due to its support for USB4, which brings with it support for Thunderbolt, making it one of the first AMD-based laptops to properly support that feature.

At my current employer the general trend has been towards standardization: every meeting room and desk has a single USB-C port that you plug into, and you should be good to go.

I can confirm that USB-C docks seem to work quite well under Linux, especially the one integrated into my Dell monitor at work.

I can also confirm that the Dell WD19TB Thunderbolt 3 dock does work.

On Windows 10.

Not on Linux though.

At some point the errors I saw in kernel logs got quite ridiculous.

The device name reads like a counterfeit merchandise listing at Amazon.

When reconnecting the port, the results were slightly different almost every time.

I think the dock has suffered a stroke.

The UEFI settings allow some control over the behaviour of the USB-C ports. You can even disable Thunderbolt support and use them as plain USB-C ports with DisplayPort support.

The behaviour of the dock would also be different depending on how you connect it. If the dock is connected during boot, your USB devices and Ethernet port will all be functional, but there’s no display output. If you boot while the dock is disconnected and connect it once you have reached the GUI, your display will work, but you have no USB or Ethernet support.

Even if one of the two critical functions of the dock work, reconnecting the cable will eventually result in nothing working. You also have a 50% chance of the machine locking up completely.

The reasons for these issues can be any one of these:

the proper kernel support is not there yet
HP messed something up with the latest UEFI update for Linux users
I was using it wrong

Dell WD19TB Thunderbolt dock workaround

After multiple sessions of troubleshooting, I seem to have come up with a solution that seems to work for my use case, which involves connecting a couple of USB devices, an Ethernet cable, and one 1440p monitor. You’re giving up Thunderbolt capabilities and forcing the dock to fall back to USB+DP altmode setting, but in return you’ll get a setup that actually works.

To get the Dell WD19TB Thunderbolt dock working, do the following:

go to UEFI settings and turn off Thunderbolt mode
save settings and reboot
connect your monitor over HDMI

Yup, that was it. I could not get DisplayPort working. Only HDMI.

The UEFI settings do contain a setting that allows you to sacrifice USB and Ethernet bandwidth to allow for higher resolution DP signals to be passed through, but that’s simply not acceptable to me. Based on the description, I think that the setting forces all USB devices to run at USB 2.0 speeds. This setting may fix the issue if you want to use DisplayPort, but I haven’t tried it yet.

Suspend

Doesn’t work. The screen will go black and the power button indicator will keep blinking. Only way to break that cycle is to force the laptop to reset by holding the power button down for an extended amount of time.

On Lenovo laptops I’ve seen an option that allows you to change the S3 state behaviour for Linux compatibility. I haven’t found a way to do something similar on the HP yet.

Stability

This is the straw that broke the camel’s back. When playing back video, I noticed issues with glitching and stuttering. The video would play fine, until a frame froze for a few seconds, and then everything continued as normal.

What ruined everything for me were the random crashes that only occurred when playing back video.

When I checked system logs from a previous boot (journalctl -b -1), I found that something was up with the GPU driver.

This was not a one-off event and once it had happened for the fourth time, I gave up on this machine and switched back to my trusty ThinkPad T430.

Conclusion

Given enough time, I’m sure that at least some of the issues will get fixed, either with a kernel update or HP rolling out a UEFI update that fixes things for Linux users.

It’s also possible that a specific set of UEFI settings will contribute to fixing any of the issues listed here. The number of combinations you can have is too high for me to bother testing all of those, so I won’t be doing that.

In its current state, I cannot recommend this laptop to Linux users. You can get most things working, but you will still be incredibly frustrated with the issues that remain.

Something has to change, and it’s going to be how I choose hardware in the future. Linux will stay, which means that the newest and “greatest” hardware is off the table for me. Honestly, lagging behind the newest hardware 3-5 years (or 10 in my case) doesn’t sound too bad, you’ll still get the performance improvements at a fraction of the original price.

If that’s what’s necessary for a good Linux experience, then so be it.

I finally found an use case for my Raspberry Pi Model B+

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Tue, 01 Nov 2022 06:00:00 +0200

You have probably heard about the Raspberry Pi. It’s a nice little affordable single-board computer with a huge community using it for all sorts of projects.

I got my first Raspberry Pi, the Model B+, during my first year at university, which was around the winter of 2014/2015. The idea of a super tiny PC that could actually do useful things was just very fascinating to me, and I loved the way that green PCB looked.

Over the years, I’ve used that Pi and newer revisions of them in all sorts of use cases: a simple Kodi media box, a Minecraft server, a web server, retro game emulation box, and much more.

Because the goal of the Raspberry Pi was to be an affordable platform for experimentation and learning for everyone, they had to keep the costs down. This meant that the computing power that the Raspberry Pi 1 packed wasn’t much. With the Model B+, you get one ARM CPU core that you can overclock to 1 GHz, and 512MB of RAM. It wasn’t much even at the time of release, and it definitely isn’t great in 2022.

Due to the very limited performance, the use cases for a Raspberry Pi 1 are quite limited in 2022. Use cases that don’t require much computational power are often better solved by other platforms. Use cases that are actually useful to me and solve a problem I have are too much for the Pi 1. At the same time, I absolutely hate it when I have computing equipment sitting around on a shelf doing nothing. I can’t even sell this Pi, because it is not worth much any more and it has some sentimental value for me.

Taking all that into account, and looking at my parts box, I decided to give one idea a go: let’s build the slowest damn Syncthing backup endpoint imaginable.

The build

Here’s a quick rundown of the parts:

SBC: Raspberry Pi 1 Model B+
Storage (microSD): SanDisk 8GB that I had lying around
Storage (USB): Crucial BX500 1TB SATA 2.5" SSD, in an IcyBox USB-SATA enclosure
Power: official Raspberry Pi microUSB power adapter
Networking: TP-Link TL-WN722N USB WiFI adapter
OS: Raspberry Pi OS 32-bit (lite)
Acoustic foam: yes.

Alternate angle of the setup.

Use case: networked offsite backup of files that I cannot afford to lose, powered by Syncthing, btrfs as the filesystem, and btrbk as the snapshotting solution.

I set the OS up using the Raspberry Pi imager due to the useful options it offered, such as automatically setting my SSH public key authentication up. Once booted, I logged in and ran sudo apt update -y. I knew what to expect regarding performance, and was still surprised at how slow things have become. Updating the system and deploying the configuration using Ansible took hours.

The 1TB SSD is formatted as a btrfs file system and mounted to /storage. For the Syncthing service, I created a separate user syncthing with the home folder on /storage/home/syncthing. The synced data will live on /storage/syncthing. Keeping the home folder on the SSD is intentional: Syncthing keeps track of application state in a database and storing it on the microSD card will wear it out much faster. The database can also be quite big if you have a lot of files to sync.

I’m not going to go into the btrbk setup in much detail, but if you’re interested in that, then I do have a write-up about btrbk. Long story short: snapshots are cool, and they help prevent data loss in case of accidental deletions.

I set up Syncthing to allow access to the GUI from over the network by changing the ~/.config/syncthing/config.xml file while the service was stopped. Just change the GUI listen address to 0.0.0.0:8384, that will make the GUI accept connections not only from localhost, but from all machines on the same network. The next step was to set a password to the GUI and check the “Use HTTPS for GUI” box, because you probably don’t want any rando messing your machine up.

The choice to go with the WiFi for networking is related to the environment this system will eventually be deployed to. Ethernet isn’t an option there.

Once the system was all set up and files were syncing extremely slowly, I kept an eye on the Pi, especially the red power LED. I had set the CPU to run at 1 GHz using raspi-config, and with both the WiFi adapter and SSD connected, I had concerns that the Pi could not supply enough power. The power limitations of the Pi are a common source for issues and that has come up in the past as well, especially with external storage connected to the Pi. However, with this particular setup this has not been an issue so far. The fact that Pi is so underpowered that it cannot even make full use of the SSD is probably a contributing factor to the overall stability.

The file sync was very slow, but at least it was working. The typical transfer speeds hovered around 1.5 MB/s. Not great, but given the fact that this setup will operate in an environment where the download speeds are typically capped around 15-20 Mbit/s, this will be more than enough.

`htop` showing how much the Pi is struggling under operation.

Typical transfer speed that I observed with the setup.

How well will this setup work over a longer time period? Only time will tell. Just like with other disastrous tests that I have done, I’ll try to update this post whenever anything noteworthy happens.

Does this setup make any sense if you need a performant Syncthing endpoint? Hell no. However, I’m very happy that this little Pi finally has a purpose.

Alternatives

This project was about finding a purpose for an SBC I already had lying around.

If the price and availability of Raspberry Pi-s wasn’t a complete shitshow in 2022, I would go for a Raspberry Pi 4 in a heartbeat. I can’t, and that makes me and most of the Raspberry Pi community very sad. The more performant Pi-s open up so many possibilities.

An Orange Pi Zero running Armbian was something I also considered, since I had one lying around, but it was a bit too competent for the job, and I might find a better use for it eventually.

If you wanted performance, then any x86-based PC will do just fine. My recommendation is to go for an used PC, be it a laptop or a very small and efficient PC. Just keep in mind that the power usage is typically higher on these setups compared to a small ARM-based single-board computer. If a machine runs 24/7, then the difference between 2-3 W and 10-20 W will start to matter, especially when energy prices are quite high in your region.

Recovering a password-protected ThinkPad T60

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Thu, 13 Oct 2022 08:00:00 +0300

A couple of years ago, I worked on a ThinkPad T60. That ThinkPad had been in use by a family member before that, and I started its cleanup by disassembling the whole machine and making sure that it was pristine. However, once I put it all back together, I saw that it was password protected. I wasn’t even able to access the BIOS. After repeatedly asking for hints for the password and trying some simple cases myself, I never cracked it.

Not too long ago I got curious and looked for a way to overcome this issue. That’s when I stumbled upon this SuperUser post about resetting the supervisor password on a ThinkPad.

Once my to-do list was down to a reasonable size, I decided to finally give this a go, since it didn’t seem that difficult. Here’s an illustrated guide on how I approached this task.

Sources that are relevant to this topic:

Setup

What you’ll need to pull this off:

your ThinkPad T60 laptop that has a supervisor password enabled
a screwdriver
anything that conducts electricity and can be used to short two pins
a steady hand
a magnifier, if your eyesight isn’t that great

For shorting pins, I came up with this contraption.

To begin the process, you need to remove the palm rest of the ThinkPad. Follow the service manual if needed. Removing the keyboard screws is optional, but it might help you access the correct pins better.

T60, slightly disassembled.

The process

The steps for performing the supervisor password clearing:

turn on the laptop
when you see the ThinkPad logo, short two specific pins
keep shorting those pins!
press F1 to go to the BIOS
navigate to the section in BIOS where you can set the Supervisor Password
enter the supervisor password setting prompt and enter a blank password in the first field
stop shorting the pins, and then enter an empty password to the second field
save settings and restart
after rebooting, you should be able to boot without having to enter a password

What follows is an illustrated guide of the process.

The pins you need to short are located on the left side of the laptop, under the palmrest.

Once you see this, short those pins!

Just apply pressure to those pads and make sure the contact is solid.

This is the step where I messed up initially. The contact between the pins was not good and I got prompted with the supervisor password prompt. Power the machine off and try again if you face the same issue.

Once the pins are shorted, you should be able to access the BIOS with a little bit of F1 key spamming.

While the pins are still shorted, you should be able to get access to the BIOS.

Leave the first field blank, stop shorting the pins, and leave the other field blank as well.

After saving the settings and rebooting, you should be greeted with a successful boot.

It's booting! Not into Linux, since the SSD is removed, but still, it works!

Once you’ve confirmed that everything’s OK, assemble the machine again and enjoy the laptop that you just saved from becoming e-waste.

Caveats

This guide might not be terribly relevant in 2022. I still remember starting my software development career in 2016 with a ThinkPad T60 and it was painful even back then. These laptops are classics, but they are really starting to show their age. I’ve even encountered issues like the WiFi chip causing lots of trouble, with the connection being very spotty and borderline unusable.

But hey, at least it’s not completely unusable any more!

Testing a cheap ExpressCard to NVMe SSD adapter on my ThinkPad T430

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Tue, 04 Oct 2022 05:00:00 +0300

I stumbled upon this post in /r/thinkpad, which got my attention. It showcases an ExpressCard to M.2 NGFF adapter that they purchased from AliExpress. The adapter has a similar purpose to the one designed by thinkmods.store, with the added bonus that it could fit longer NVMe SSD-s if you didn’t mind them sticking out of the case. You could also order one instantly, so I did.

For those interested, the title of the listing at the time of writing of this article is “ExpressCard interface to m.2 NGFF nvme solid state disc x201 t430 hp8570 w520”. The listing I chose was this one. I’m not affiliated with the manufacturer or the seller in any way.

Bits and pieces.

The idea

I try to avoid buying things that I don’t need, which is why I came up with an excuse for getting this adapter: adding a cache SSD to my ThinkPad-as-a-server.

The OS is running off of an mSATA SSD that I could theoretically use for the same purpose, but eventually killing the SSD that also hosts the OS with constant writes sounded like a really bad idea. I also didn’t want to have excessive writes on my pair of Samsung 870 QVO 4TB SATA SSD-s, because replacing those will be quite expensive down the line.

Here’s what we’re working with:

Sabrent Rocket Nano NVMe 2242 SSD (512GB)
the aforementioned ExpressCard to M.2 NGFF adapter
ThinkPad T430 with an ExpressCard34 slot

Quick word on the packaging: it was packaged in a small cardboard box, which held up okay, but a more rough treatment of the box would probably have broken the tiny PCB.

The SSD, adapter PCB, and the case.

Assembly

According to comments I saw before purchasing the adapter, I expected the assembly to be a bit tricky, and after assembling it I have to agree with that sentiment.

First, you’ll have to squeeze the SSD in there. Unlike M.2 slots that are common in laptops and desktops alike, this one requires you to horizontally push the SSD in without breaking anything. Don’t be afraid to use a bit of force here, but be careful in how you apply it.

na-no

Bottom of the adapter.

After the SSD is in, screw it in properly and place the assembly on one of the metal parts.

Now comes the fun part: you’ll have to clip on the top part as well. My recommendation is to first clip in the part that sits near the contact pins, then the one at the other end of the adapter, and then push in the metal shielding at the edges.

The adapter, 100% assembled.

For some reason, mine ended up a bit crooked near the connector.

This opening allows for the installation of longer M.2 SSD-s.

Tests

To test this adapter out, I just plugged it into my main ThinkPad T430 running Fedora 36 and Linux kernel version 5.19.12. It worked right out of the box.

Yes, it has a blue LED that blinks.

When not blinking, it looks something like this.

Fedora picked it up with no issues.

My ThinkPad T430 can now support hotpluggable NVMe SSD-s, and 4 internal SSD-s in total. Not quite tricked out as this setup, but not bad for a 10 year old laptop.

4 SSD-s on a ThinkPad T430. Bonkers.

Please note that these tests are not done with any meaningful testing methodology and are just quick one-off tests that aim to showcase what to expect when using this adapter with the configuration I’m using it with.

To check how well the adapter performs, I opened gnome-disks and did a quick benchmark offered by it.

Results of the quick one-off benchmark.

The average read speed is at 411.2 MB/s, and writes at 366.7 MB/s.

The results are in line with the limitations of a PCIe 2.0 x1 link, which has a theoretical cap of 500 MB/s.

What’s more noteworthy is the latency. Average access time is at 0.03 ms, which compares well to a Samsung 870 EVO 1TB SATA SSD that’s connected to the same machine, which reports an average access time of 0.07 ms.

When copying files to the SSD, the temperature sensor reported a maximum of 60°C. Additional read operations in the form of an ZFS scrub hit 63°C, which was the maximum I observed during my quick testing.

When I threw both read and write operations to the SSD, the total I/O bandwidth for both operations was capped at around 390 MB/s according to zpool iostat 1 -v. If your workloads do a lot of read-write operations in parallel, then take this into account.

I/O stats during heavy read-write operations.

As for my use case, it seems to be holding up okay. I’ll have to report back after this adapter and SSD combo has been in use for a longer time.

The minimum viable fan control script

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Mon, 26 Sep 2022 08:00:00 +0300

I’ve always been a fan of tinkering with cooling setups on my computers. I’ve even went as far as writing crappy solutions to make up for deficiencies on the hardware level. After years of dumb experiments I’ve seen how little you can get away with in cooling and how to run your machines as quietly as possible without giving up too much performance.

I also appreciate simplicity, which is why I’m currently running the simplest damn solution you can imagine to control how my ThinkPad T430 and ASRock DeskMini X300 run: a shell script.

It also comes with some added benefits:

self-documenting
highly customizable
low resource usage

These scripts are not “battle-hardened” and using them incorrectly may or may not result in hardware failure. Use at your own risk. The only guarantee I can give is that it works on my machine™.

If you’re someone who doesn’t like tinkering with their computers and wants their machines to “just work”, then this article likely isn’t for you.

ThinkPad T430

With the ThinkPad T430, the script does two things:

control the fan speed
preemptively throttle the CPU and GPU

This type of behaviour is optimized for mostly quiet operation. The fan is running at its lowest speed when under a small load, and the CPU/GPU throttling has an added benefit of reducing the overall power usage, too. The difference between non-turbo and turbo speeds on the Intel i7-3820QM can be as much as 13-15 watts.

To enable fan control on ThinkPads, you need to have a file at /usr/lib/modprobe.d/thinkpad_acpi.conf with the following contents:

options thinkpad_acpi fan_control=1

Reboot, and you can now have full control over the fan speed in your ThinkPad!

The script itself looks like this:

#!/bin/bash

set -e

while true; do

  temp=$(cat /sys/class/thermal/thermal_zone1/temp)

  if ((temp > 90000)); then
    echo level 7 >/proc/acpi/ibm/fan
  elif ((temp > 75000)); then
    echo level 5 >/proc/acpi/ibm/fan
  elif ((temp > 60000)); then
    echo level 3 >/proc/acpi/ibm/fan
  elif ((temp > 30000)); then
    echo level 1 >/proc/acpi/ibm/fan
  else
    echo level 0 >/proc/acpi/ibm/fan
  fi

  if ((temp > 97000)); then
    echo 350 >/sys/class/drm/card*/gt_max_freq_mhz
    echo 350 >/sys/class/drm/card*/gt_boost_freq_mhz
  elif ((temp > 90000)); then
    echo 650 >/sys/class/drm/card*/gt_max_freq_mhz
    echo 650 >/sys/class/drm/card*/gt_boost_freq_mhz
  else
    echo 1250 >/sys/class/drm/card*/gt_max_freq_mhz
    echo 1250 >/sys/class/drm/card*/gt_boost_freq_mhz
  fi

  if ((temp > 70000)); then
    echo 1 >/sys/devices/system/cpu/intel_pstate/no_turbo
  else
    echo 0 >/sys/devices/system/cpu/intel_pstate/no_turbo
  fi

  sleep 0.5
done

What we’re doing is reading the temperature of the CPU package and then setting the fan speed and throttling based on that. Note that the temperatures are represented without any decimal places. A reading of 78000 means that the chip is running at 78 °C.

On my ThinkPad T430, the fan speed levels can be roughly described as such:

level 1: lowest speed, barely audible
level 3: audible, but tolerable
level 5 and above: it’s probably loud enough to bother you

The second section controls the integrated GPU. What I’ve found in my testing is that a weak integrated GPU can still chug up to 20 watts, which is a considerable amount, especially on a laptop. The card* wildcard is there because the integrated GPU may change between card0 and card1, and I’m too lazy to make the script any smarter.

The values written there correspond to the maximum GPU clock speed. 1250 happens to be the max clock speed (in MHz) and at that speed the GPU is using a lot of power. 650 is the speed at which the integrated GPU is running most efficiently. 350 is the lowest speed, and you will notice it due to the performance being extra crappy.

With the CPU, I chose to just turn the turbo boost on or off. With turbo boost off, the CPU will top out at 2.7 GHz and around 22 watts of power usage. With turbo boost, the CPU can run anywhere between 3.4-3.7 GHz, depending on the number of cores under load. You’ll likely notice much higher CPU temperatures and power usage climbing to around 35 watts. Fast at short and bursty workloads, and yet efficient and relatively cool at sustained loads.

And at the very end, you have a simple sleep statement. I’ve set it to half a second, but feel free to set it as you see fit. Higher intervals might not be that good of an idea because the script may not be able to respond quickly enough to changing thermals, just keep that in mind.

ASRock DeskMini X300

Although I’ve opted for a more experimental setup with my home server and the ASRock DeskMini X300 is no longer running as one, I think it’s still worthwhile to share how I configured the fan control logic on this machine.

With the ASRock DeskMini X300, my goal was simple: let it run as quietly as reasonably possible without losing too much performance.

This machine was running 24/7 as a server and has a powerful 8-core AMD Ryzen 7 5700G CPU in it. One thing this script is heavily relying on is the fact that Ryzen CPU-s have well-engineered boosting logic in them, meaning that they’re already designed to run at the highest speed possible and only limit their speed if they hit power or thermal limits.

One thing you may need to do first is installing lm-sensors and running sudo sensors-detect. By default the CPU fan speed controls were not properly exposed, but after running sensors-detect and saying yes at every prompt, they were picked up.

The script looks like this:

#!/bin/bash

set -e

# /sys/devices/platform/nct6775.656/hwmon/hwmon3

echo 1 >/sys/devices/platform/nct6775.656/hwmon/hwmon3/pwm2_enable

while true; do

  temp=$(cat /sys/class/hwmon/hwmon4/temp1_input)

  if ((temp > 94000)); then
    echo 170 >/sys/devices/platform/nct6775.656/hwmon/hwmon3/pwm2
  elif ((temp > 90000)); then
    echo 140 >/sys/devices/platform/nct6775.656/hwmon/hwmon3/pwm2
  elif ((temp > 70000)); then
    echo 120 >/sys/devices/platform/nct6775.656/hwmon/hwmon3/pwm2
  elif ((temp > 40000)); then
    echo 30 >/sys/devices/platform/nct6775.656/hwmon/hwmon3/pwm2
  else
    echo 0 >/sys/devices/platform/nct6775.656/hwmon/hwmon3/pwm2
  fi

  sleep 5
done

The hwmon values and which fan you’re controlling is likely different on your own PC. Take a look at the paths found in the script and poke around while also physically looking at your PC. Try enabling PWM control by writing to pwm*_enable handles and then setting the fan speed. As a result of your poking, you should see the CPU fan change its speed. If it doesn’t, try another value.

The values you can write are in the range 0-255, where the higher value corresponds to a higher speed.

I’ve limited the TDP of my CPU to 35 watts using a setting found in UEFI settings. This results in the CPU running cool and quiet even under a full load.

systemd service

To run your fan control script as a systemd service, drop the script to your preferred location (I placed it in /root/.local/bin/fancontrol), then create a file /etc/systemd/system/fancontrol.service with the following contents:

[Unit]
Description=Quick fan control software

[Service]
ExecStart=/root/.local/bin/fancontrol

[Install]
WantedBy=multi-user.target

And once that is done, enable and start your fan control service with sudo systemctl enable --now fancontrol.service.

If you encounter issues or something isn’t working right, check the status of the service with systemctl status fancontrol.service or full logs with journalctl -u fancontrol.service.

Conclusion

If you want something that works out of the box and does the smart stuff for you, then give this Arch Linux wiki page a look. If you’re not into that and want to easily tune the behaviour of your machine, then feel free to use these scripts as a baseline for your own experimentation.

Accidentally turning the ASRock DeskMini X300 into a semi-passively cooled PC

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Sat, 17 Sep 2022 08:00:00 +0300

After playing around with fan curves on my ASRock DeskMini X300 based home server, I noticed one interesting thing. At least it was interesting to me.

Background

I had one goal: to try to see if I could turn it into a semi-passive PC by only turning the fan on when the PC was under load while keeping it off for as long as possible. Running the fan only when necessary means less noise and less dust accumulating on the heat sink. No dust, no maintenance required.

The cooling solution used is the Noctua NH-L9a, a low-profile CPU cooler. The two main selling points for me were that it was small enough to fit into the DeskMini, and it is much quieter than the stock AMD Ryzen Stealth cooler.

I started with something reasonable: only kick the fan on when the CPU is over 55°C. This did end up with the fan occasionally turning off, but after around 10-15 seconds it would kick on again. With time, I bumped this limit up by 5-10 degrees each time and ended up with the current configuration. The fan now turns on after 80°C and starts spinning at a modest level at over 90°C.

By turning this limit up, I noticed that the PC could be run in a fanless mode for much longer. This meant that the PC was also much hotter itself and was warm to touch, but nothing extreme or concerning.

Results

I added a couple of lines to my fan control script to track the CPU temperatures and the fan speeds, here are the results:

CPU temperatures over a 19 hour period.

Based on the data I gathered, the CPU fan was off for 90.7% of the time during this test period. The temperature in the apartment was around 26-30°C, which is pretty much the worst case scenario in my region.

My running theory is that a hotter heat sink is able to dissipate more heat passively. The motherboard and the metal case may also be small contributors to cooling the PC this way. The PC is also acting as a 24/7 home server, which means a lot of idle time that’s interrupted by occasional short-term load. Heat from those bursty loads can be stored in the heat sink relatively easily without it heating up enough to trigger the fan.

This approach will likely not work that well if your workloads are more demanding and result in a higher continuous loads on the system.

Yes, I’m aware that running a PC like this may have negative consequences. I ran the server like this for about a week and while I liked it being a semi-passively cooled server, but decided to go back to the old behaviour.

While there are plenty of fanless designs out there that run hot by design, I decided that I don’t want to take this risk with the DeskMini. It would be a cool experiment, but my main server is probably not the best candidate for it. The CPU could handle it without breaking a sweat, but I’m more concerned about the rest of the components, especially the motherboard. The stickers on the DRAM modules were already warped due to the heat, I prefer not to damage anything else.

For those who want to try something similar (and at your own risk, of course), here’s my fan control script. Just make sure to install lm-sensors first and run sudo sensors-detect so that the controls are exposed by your system. The exact paths and values may be something you need to change to match your system.

#!/bin/bash

set -e

# /sys/devices/platform/nct6775.656/hwmon/hwmon3

echo 1 >/sys/devices/platform/nct6775.656/hwmon/hwmon3/pwm2_enable

while true; do

  temp=$(cat /sys/class/hwmon/hwmon4/temp1_input)

  if ((temp > 94000)); then
    echo 120 >/sys/devices/platform/nct6775.656/hwmon/hwmon3/pwm2
  elif ((temp > 80000)); then
    echo 70 >/sys/devices/platform/nct6775.656/hwmon/hwmon3/pwm2
  else
    echo 0 >/sys/devices/platform/nct6775.656/hwmon/hwmon3/pwm2
  fi

  sleep 5
done

Honorable mentions

I’m not the only one who likes to experiment with the ASRock DeskMini X300. This all started with an idea of fitting a passive cooler into the DeskMini and me discovering that it was already done.

More fun ideas:

Overcoming hardware limitations: the time-to-sleep script

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Fri, 26 Aug 2022 08:00:00 +0300

As some of you might own, I’m still rocking a ThinkPad T430, a laptop model originally released in 2012. It’s not the fastest laptop out there, but it is plenty fast for a number of tasks, even most software development work.

I also try to keep my machines up to date and properly backed up. On my laptop, this means having around a couple of scheduled tasks, such as restic backups. They don’t really take up a lot of time, but if an intensive scheduled task starts while you’re having a video call over Google Meet while also sharing the screen, things can get a bit slow, ruining the experience.

After a couple of these instances, I started thinking about finding a solution.

Scheduled tasks

Here’s a quick rundown of regular tasks that I like to run on my laptop.

Backups using restic: hourly backups to my home server. The file scanning process of the backup is quite intense, eating up a lot of my CPU time.

System updates using dnf and flatpak: usually run daily, because I’m too lazy to install the updates using a GUI, and at the same time I cannot afford to run out of date software.

Backups of critical files from my server to my laptop: there are some files I just cannot afford to lose. For this reason, I make a daily backup of the files to my laptop.

Filesystem maintenance: I don’t trust my storage devices (and neither should you), which is why I run a weekly scrub on my SSD that’s running the btrfs filesystem. If there are errors detected, I will know as soon as possible.

These tasks can be quite intensive, especially on hardware that’s a bit older.

With some tasks, such as system updates, you may also run into random issues. Updating Mozilla Firefox while it’s actively running results in the browser requiring a restart immediately after opening a new tab. This is very annoying and inconvenient when you’re in the middle of your work.

The “time-to-sleep” script

To avoid intensive scheduled tasks running while I’m doing actual work, I tried rescheduling these tasks to a more quiet time. All I needed to do was to edit the systemd timers so that the jobs start after midnight and leave the laptop running.

This solution has quite an obvious downside: my laptop will be powered on pretty much 24/7, and since electricity is super expensive and laptop maintenance a bit of a pain, I’d be wasting resources, time and eventually money.

I could make the last scheduled task make the machine shut down, but what if I have a long-running file copy operation running? It would be interrupted, unless I specifically disabled the scheduled task.

Then it clicked for me.

I know when I’m planning on going to sleep. Sleeping also has a regular cycle as it happens daily. Why not just put all the scheduled daily tasks into one script that I can run whenever I intend to shut down my laptop?

And thus, the time-to-sleep script was born. It’s a simple Bash script that lives in /root/.local/bin/time-to-sleep. To make sure I can also reasonably observe the behaviour of the script, I’ve implemented it as a simple systemd service that lives in /etc/systemd/system/time-to-sleep.service. It’s disabled by default, but can be started manually.

[Unit]
Description=Run maintenance tasks

[Service]
ExecStart=/root/.local/bin/time-to-sleep

I have an alias in ~/.bashrc that I use to start the service.

alias time-to-sleep="sudo systemctl start time-to-sleep.service"

Whenever the time comes to go to sleep, or when I’m just going to do something else and can leave the laptop running at my desk, I just open a terminal window, type time-to-sleep and I’m good to go. If I’m somewhere with a very poor internet connection or need to conserve battery power, I can opt to not run that script.

Going with this approach also saves me a lot of trouble on the technical side as well. I don’t need to think of a magical way to automatically detect if I’m working at the moment, or if I’m running on battery, or if I’m behind a limited internet connection. Sometimes your job becomes so much easier if you make some assumptions about when and where the script runs. You’re in control of choosing when to run it, after all.

In case something goes wrong, I can rely on the set -e line in my script to fail hard. When I come back to my machine, it will be quite noticeable that something has gone wrong, as the laptop will still be running. Who needs a fancy monitoring setup when you have visual evidence of failure?

As for the hourly backups, I solved that problem by only making backups of more important files. It takes much fewer resources compared to the full /home folder backup that runs as part of the time-to-sleep script.

Conclusion

After all this work, I don’t have to worry about any scheduled tasks hogging up all the resources while working. At the same time, I still get all the benefits that the scheduled tasks bring.

Example

In case you need it: here’s an example of the script that I’m running, with all the personal details scrubbed out. Feel free to use it as inspiration for your own time-to-sleep script!

#!/bin/bash

set -e

echo "Restic full home folder backup."
export RESTIC_REPOSITORY='sftp:backupuser@backupserver.lan:/path/to/repo/'
export RESTIC_PASSWORD='someverysecurepasswordgoeshere'

restic unlock

restic backup --verbose --exclude-caches --cleanup-cache \
  --iexclude=/home/*/downloads \
  --exclude=/home/*/.cache \
  --exclude=/home/*/.gradle \
  --exclude=/home/*/.local/share/Trash \
  /home

restic forget --prune \
  --keep-hourly 0 \
  --keep-daily 7 \
  --keep-monthly 0

echo "Restic full home folder backup done."

echo "Backing up important data from the server."

rsync -az --delete-before \
  user@myserver.lan:/path/to/nextcloud/ \
  /storage/backups/nextcloud/

chown -R localuser:localuser /storage/backups/nextcloud/

echo "Backing up important data from the server done."

echo "Updating system."

dnf update -y

flatpak update -y && flatpak uninstall --unused -y

echo "Updating system done."

echo "Scrubbing disk."
btrfs scrub start -B /
echo "Scrubbing disk done."

echo "All done! Going to sleep."

shutdown now

This site will be offline on 2022-08-17 between 18:00-19:00 EEST

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Wed, 17 Aug 2022 15:45:00 +0300

There really isn’t much to say here, nor is there a rational reason behind this. I just see it as an opportunity to send a signal and test the capabilities of my UPS.

I’ve prepared for this moment with my self-hosting setup, let’s see how well it holds up. Hopefully better than the electrical grid.

Winter is coming.

:harold:

I tried out the CyberPower UT650EG UPS

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Thu, 04 Aug 2022 07:00:00 +0300

After some spicy feedback to my post where a ThinkPad T430 acted as a server, I decided to try out one UPS that was recommended on the basis of it being much more power efficient compared to the APC UPS that I previously ran.

That UPS? CyberPower UT650EG.

My ThinkPad-as-a-server, and the UPS that's powering it.

Why switch out a perfectly good UPS?

The APC Smart-UPS 750 is good, but uses a lot of power compared to the whole setup, making up roughly half of the total power consumption at idle: ~17-18W. It’s also very big compared to what it is actually powering.

One of the main selling points of this particular CyberPower UPS is that it uses very little power itself. Depending on the model, the UPS can use around 1-2W of power, at least according to the tests that they publish on their website.

Once I set up everything with the new UPS, I gave it a go. After the battery was fully charged, I did observe a dramatic drop in power usage. The whole setup now ran around 19-21W when idle, which confirms the power consumption numbers claimed by CyberPower.

Overview

The UPS itself is great for someone who just wants to connect a couple of devices into a standalone UPS. The model I got came with three EU-style sockets. Compared to the APC UPS, this is a much cleaner setup as I do not have to connect an extension cord using an adapter.

The APC Smart-UPS 750 can provide anywhere between 100 and 120 minutes of runtime with my setup. I did a quick runtime test with the CyberPower UPS as well and found that with around 25W of load it ran for 37 minutes. Not the best runtime, but given its size it’s a respectable result.

Uninterruptible power supplies like to be noisy when running off of batteries. This CyberPower unit is no exception. However, by letting it run on batteries and double-clicking the power button I managed to silence the alarm and in subsequent power cut scenarios it has not started beeping. I was unable to find a simple manual explaining how to operate the UPS and what other button combos might be out there. The piece of paper that came with the UPS was also lackluster. I’m not sure if this was intended functionality or not, but I do like that turning the alarm off is an option in the first place.

The status LED-s on the unit are okay. There aren’t many of them and the legend explaining the behaviour of those lights is just enough to understand what’s going on.

The not-so-good parts

This particular unit has an issue. I did a poor job reading the spec sheet at my local retailer and missed the fact that this specific model does not come with USB connectivity. If the UPS runs out of power, it cannot warn my server to shut down in advance, resulting in a dirty shutdown scenario. Given how stable the power at my apartment generally is, this won’t likely be a major issue, but it’s definitely something I miss. The APC UPS supports this, plus you can log everything about how the batteries are doing and even detect input voltage anomalies that you probably weren’t even aware of.

What I’m concerned about is the battery. The specifications say that the battery is not user replaceable. I hope that this is simply something the legal team thought of to avoid any liability when dealing with users who messed this process up. If not, then it will be a major problem down the line. Batteries are consumable items, and according to various threads on the web about UPS batteries, the consensus is that you should replace them every 3-4 years to avoid problems with the batteries swelling up and potentially causing fires.

If the battery dies and I cannot replace it, then what am I supposed to do, throw the otherwise functional UPS straight to the landfill? Not acceptable.

Speaking of concerns, I happened to see some troubling reports about certain CyberPower UPS-es catching on fire. The culprit seems to be a certain type of glue used in the manufacturing of the UPS that ends up conducting electricity after a longer time period. I unplugged my UPS, powered it off and gave it a look, and it seems like I should be fine. At least I hope so.

Inside of the UPS.

I sure hope that this piece of glue won't burn my house down.

Based on this view inside the UPS, I feel like changing the battery should at least be possible, assuming that replacement batteries are readily available. Just make sure to not touch the rest of the components as they can probably kill you, and that’s just not a fun thing to do.

Conclusion

If you want a dumb and efficient UPS that protects you from basic power cutouts and input voltage changes, then do consider this UPS. However, if you care about USB connectivity and ease of maintenance, then I recommend looking for other options.

btrbk is awesome

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Sat, 09 Jul 2022 16:00:00 +0300

I like having a safety net whenever I’m doing something potentially destructive, which is why I use the btrfs file system for my operating system and my data. Snapshots are one half of my “whoops, there goes all my work” strategy (backups are the other half).

I’ve written about how I use snapshots on btrfs using snapper, but lately I’ve become annoyed with it.

Snapshots: probably invented by a developer who owns a cat.

Shortcomings of `snapper`

snapper is great while you’re on the happy path, but when you wander off of it, it gets a bit frustrating. This is 100% my own personal experience and I cannot rule out any PEBCAK scenarios, but it’s how I felt using the tool.

The snapshots are on the same subvolume, such as /home/.snapshots, which means that I have to specifically exclude the .snapshots folder in every tool and script that I use for making backups. Without that change, the tools ended up scanning the folder and eating up a lot of resources, mainly the CPU and storage.

Backing up snapshots to something like an external backup SSD wasn’t something that snapper supported out of the box as well, meaning that for offline backups I still had to resort to a good old rsync copy. It’s inconvenient and I won’t have a backup that also contained recent snapshots, but it was manageable.

I also reinstalled Fedora Linux on my ThinkPad recently to see how well my Ansible playbooks hold up (quite decently!) and found that setting up snapper is a bit of a pain. I tried to set up snapper again, but it wasn’t as straightforward as I was hoping for, with it not properly picking up the configuration file changes. At the end, it just ran into weird errors when I tried it to prune some snapshots according to the policy I set up. Since I was running out of disk space, I just gave up, uninstalled snapper and destroyed the snapshots manually.

After all that (and a recommendation by a friend) I decided to bite the bullet and try out btrbk.

btrbk

Setting up btrbk on Fedora Linux 36 was a breeze. Install it via dnf install btrbk, copy your configuration to /etc/btrbk/, make sure that the directory used for snapshots exists with mkdir /btrbk_snapshots and you’re good to go! btrbk is just a huge Perl script, so no fancy dependencies are needed.

My btrfs filesystem has three subvolumes: /, /home and /storage. Data stored on these subvolumes has different levels of importance, which is also reflected in the snapshot retention policies.

The configuration on my setup is the following:

timestamp_format        long

volume /
  snapshot_dir /btrbk_snapshots
  subvolume /
    snapshot_preserve_min   12h
    snapshot_preserve       24h
  subvolume /home
    snapshot_preserve_min   48h
    snapshot_preserve       7d
  subvolume /storage
    snapshot_preserve_min   6h
    snapshot_preserve       7d

To take the snapshots, I’ve set up a systemd timer that runs every 5 minutes and runs the following command:

btrbk -c /etc/btrbk/btrbk.conf run --progress

A successful run will have output like this:

--------------------------------------------------------------------------------
Backup Summary (btrbk command line client, version 0.32.1)
    Date:   Sat Jul  9 14:00:00 2022
    Config: /etc/btrbk/btrbk.conf
Legend:
    ===  up-to-date subvolume (source snapshot)
    +++  created subvolume (source snapshot)
    ---  deleted subvolume
    ***  received subvolume (non-incremental)
    >>>  received subvolume (incremental)
--------------------------------------------------------------------------------
/
+++ /btrbk_snapshots/ROOT.20220709T1400
--- /btrbk_snapshots/ROOT.20220708T1300
/home
+++ /btrbk_snapshots/home.20220709T1400
/storage
+++ /btrbk_snapshots/storage.20220709T1400
--- /btrbk_snapshots/storage.20220709T0745
--- /btrbk_snapshots/storage.20220709T0750
--- /btrbk_snapshots/storage.20220709T0755

I love how the output is very concise about what it just did.

To make a backup of my drive to an external SSD that’s also a btrfs filesystem, I created a separate config at /etc/btrbk/backup-to-ext-drive.conf:

timestamp_format        long

target_preserve_min    no
target_preserve        7d

volume /
  snapshot_dir /btrbk_snapshots
  target /mnt/backup/T430
  subvolume /
  subvolume /home
  subvolume /storage

In this example, /mnt/backup/T430 is a btrfs subvolume itself.

Instead of running rsync, my backup script contains the line:

btrbk -c /etc/btrbk/backup-to-ext-drive.conf run --progress

I love how the default behaviour in btrbk is to keep snapshots separate from the subvolumes themselves. Making a full copy of /home or any other filesystem doesn’t require me to always exclude the .snapshots directory, which is nice.

The snapshot names themselves are also human readable. If I accidentally messed up and my changes from an hour ago were lost, I’d just have to go to /btrbk_snapshots, look up a snapshot from an hour ago, and copy whatever I need from the snapshot. This has already saved my butt at least once.

Conclusion

It’s not often that a piece of software gets me that excited, but btrbk has somehow managed to do it. It’s a single script that has all the example use cases documented in its README and the functionality does exactly what I want.

That’s all I really need from software.

Huge thanks to Digital Integrity GmbH for building, maintaining and sharing this tool with the world!

The absolute state of Bluetooth audio in 2022

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Wed, 29 Jun 2022 11:00:00 +0300

I have the Sony WH-1000XM3 headphones. They’re good for consuming content. Audio calls with your colleagues? Forget about it.

Thanks, Bluetooth. #notsponsored by Värska, it's just very hot right now.

This is the setup I have to go with, and it has all to do with how Bluetooth works. Your options with Bluetooth headsets are the following:

high quality sound output, no audio input
low quality sound output, low quality audio input (HSP/HFP)

That’s it.

Choice of codecs exposed by Fedora 36.

This isn’t a problem with Linux, either. I’ve done a test call with the help of my friend on my iPhone SE 2020 using the same headset on Discord, and the audio quality was still crap. Even the microphone on the phone itself had better quality.

The only reasonable choice for doing voice calls is attaching a separate wired microphone to your Bluetooth headphones. The mic quality is much better now and you still have the option of disconnecting the mic whenever it’s not needed. The mic I’m using in the first image is the ModMic.

It’s a damn shame that I need to resort to this type of setup though. Who pushed for going all-in on Bluetooth and throwing out the headphone jack from phones before making sure that Bluetooth actually works properly?

Rant over.

Disabling the crappy Broadcom Bluetooth adapter in your ThinkPad T430 running Linux

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Wed, 01 Jun 2022 12:00:00 +0300

You’ve disabled the boneheaded Lenovo WiFi adapter whitelist using a tool like 1vyrain, installed a better adapter like the Intel Dual Band Wireless-AC 7260, and found that Bluetooth is not working as intended on your ThinkPad T430 running Linux?

Here’s a guide on what you can try to resolve some issues you may encounter.

The integrated Bluetooth adapter

The ThinkPad T430 comes with Bluetooth out of the box. Mine is this one:

Bus 001 Device 004: ID 0a5c:21e6 Broadcom Corp. BCM20702 Bluetooth 4.0 [ThinkPad]

And it looks something like this.

I run Fedora Linux and frankly the experience with the stock adapter is horrible. Getting stuck while pairing a device or the GNOME Bluetooth settings GUI hanging isn’t a rare sight. Sometimes I’ve had to run a script to reset all USB devices to get things sorted out, or restart the machine altogether. This is not fun when a work call started 3 minutes ago and your Bluetooth headset is just not working.

After installing the new Intel WiFi adapter, I thought that I’d just disable the integrated Bluetooth controller via the UEFI settings. That should do the trick, right?

Nope. Apparently this disabled ALL bluetooth devices, including the one provided by the Intel adapter.

Well, no problem, I’ll just physically remove the Broadcom device and that should force the ThinkPad to use the Intel adapter, right?

Nope. Now the machine just shows no Bluetooth devices.

udev to the rescue

If you install the Intel adapter and don’t touch any of the settings, you’ll have two Bluetooth adapters. On my laptop, the OS used the Broadcom adapter by default, which was evident by all GUI interactions interfacing with the Broadcom one and throwing out errors in dmesg -w as a result.

Since disabling the Broadcom bluetooth adapter via other means did not work, I had to look for some solutions. One that I stumbled upon relied on udev rules.

Essentially, what you want to do is drop a file in /etc/udev/rules.d/, name it something like 81-bluetooth-hci.rules and make the contents something like this:

SUBSYSTEM=="usb", ATTRS{idVendor}=="0a5c", ATTRS{idProduct}=="21e6", ATTR{authorized}="0"

Make sure to replace the idVendor and idProduct with the correct values for your Bluetooth adapter. Output of lsusb will likely contain those values.

In my case, the output of lsusb is

Bus 001 Device 004: ID 0a5c:21e6 Broadcom Corp. BCM20702 Bluetooth 4.0 [ThinkPad]

The idVendor and idProduct info is in this string: ID 0a5c:21e6.

After making this change and restarting, you should see thatbluetoothctl list outputs only one Bluetooth device.

[host@T430 ~]$ bluetoothctl list
Controller AC:FD:CE:30:EB:BA T430 [default]

Because Bluetooth is still a hot mess in 2022, I still have to occasionally beat it with a hammer. However, the experience with the Intel adapter is much better and a simple “turn the adapter off and on again in the GUI” solved 99% of the issues I’ve encountered since then.

“Have you tried turning it off and on again?”

If for whatever reason this solution does not apply to you, and you feel like you want to hit your machine with a big hammer without having to restart it, then here’s a script that resets all the USB devices on your machine. I’ve observed that most Bluetooth adapters run over USB, which is why this script works with those as well.

It has helped me countless times when desperate for a quick Bluetooth fix on various desktops and laptops that I’ve used.

Run it at your own risk.

#!/bin/bash

for i in /sys/bus/pci/drivers/[uoex]hci_hcd/*:*; do
  [ -e "$i" ] || continue
  echo "${i##*/}" > "${i%/*}/unbind"
  echo "${i##*/}" > "${i%/*}/bind"
done

Can a laptop from 2012 be a viable home server?

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Tue, 10 May 2022 09:00:00 +0300

I’m a man of many ideas. It’s a shame that most of them are quite stupid.

This is one of them, but at least it makes for a fun experiment.

Intro

After adding an UPS to accompany my server, my setup was as complete as it could reasonably be. The server is low power, but packs quite a punch with fast storage and plenty of compute power around. The UPS provides a battery backup for both my server and the router.

There was still one annoyance lingering at the back of my head: this setup took 40 watts of power in idle. That’s a lot for something that runs 24/7. Yes, /r/homelab regularly showcases more power hungry setups, but my standards are quite high when it comes to power use. I want to go as low as reasonably possible.

I recently bought a spare ThinkPad T430 so that I can have a backup machine in case something happens to my laptop that doesn’t suck. I’ve also had ideas about a server setup involving this laptop.

That’s when I decided to try running my services off of the ThinkPad T430.

The setup.

Another angle of the setup.

The setup

Here’s a quick rundown of the relevant specifications:

CPU: Intel i5-3320M, 2 cores 4 threads @ 3.3GHz max
RAM: 16GB DDR3
OS disk: 256GB Micron mSATA SSD
Main storage: 2x 4TB Samsung 870 QVO SSD

One of the main reasons that this setup is viable in the first place is the fact that the ThinkPad T430 supports up to 3 SSD-s if you replace the optical drive with a HDD caddy. Your OS can run off of an mSATA SSD and your mirrored storage can utilize the two 2.5" SATA slots.

You probably don’t want to use a setup like this if you want to store a lot of big files, such as your legally obtained media collection. If your data can fit on a pair of SATA SSD-s, then this setup will be just fine.

Why use a laptop as a home server?

If you don’t have a home server yet and want to start off as cheap as possible while also being mindful of the power usage, then an used laptop with a competent CPU is a fantastic starting point. In an age where 6 CPU cores is considered mid-range, two CPU cores might not sound like much, but for most home server workloads it’s absolutely fine.

Laptop CPU-s are designed to use as little power as possible. After all, they need to run off of a small battery most of the time and have to run cool as well to make do with the limited cooling setup that they have. This makes them ideal for a machine that runs 24/7.

I measured the power usage of my ThinkPad T430 in this configuration and found that it used around 15-20 watts when running with a low load. This is comparable to the ASRock DeskMini X300.

What sets laptops apart from the usual desktop-based home server builds is that they often have a working battery, which can act as a poor man’s UPS. If you momentarily lose power, then the server will happily keep on working until the battery runs out, which might take 1-2 hours (or more if your battery is in a good state). This is comparable to the runtime I saw with the DeskMini server build and the APC Smart-UPS 750.

Most laptops also come with an Ethernet port, and if the laptop is made some time after 2005, then it’s likely running at gigabit speeds. Unless you have a valid use case for 10 gigabits or more, then this will be plenty of network bandwidth for most home server use cases.

One aspect that’s often overlooked with a laptop-based home server setup is the inclusion of an integrated keyboard and mouse. If you accidentally mess up your configuration and your server does not start up properly, then you can just open the lid and start fixing it on the spot. On any other setup, you’d have to find a monitor to hook the server up to and find a spare keyboard to use as well, which can be a hassle.

Performance

A home server that serves the needs of a couple of users at most and runs some scheduled tasks from time to time can run perfectly fine off of a dual core Intel Core series CPU from 2012. You probably don’t want to run intensive computational workloads off of it, such as transcoding video files on a service like Jellyfin. However, you can utilize the integrated GPU to off-load specific workloads to it, and media encode-decode for select video codecs is one of those. Intel QuickSync is an absolute godsend for situations where you need to transcode media on a CPU that isn’t that powerful.

My home server runs a lot of services: Docker containers, this blog, data archival jobs, backup jobs etc. When running them on the T430, the average CPU usage was around 20%. Not ideal, but still leaves plenty of headroom for services.

The overall usability and performance that this machine offers was completely acceptable. It’s not as lightning fast as something that runs a modern CPU, like the AMD Ryzen 5 PRO 4650G in the DeskMini, but it gets the job done.

Caveats: thermals

A home server built around a used laptop brings a lot of benefits. However, it also has some potential downsides that you should be aware of.

Most laptops have fans, and they can be damn loud. The ThinkPad T430 is notoriously loud if you have the cooling fan manufactured by Toshiba. In a home setting this can potentially a deal-breaker, unless you can wedge the laptop into a corner of the room where it’s not audible.

There exist solutions to the fan control problem. You can install a tool like thinkfan or just write your preferred fan level to /proc/acpi/ibm/fan to avoid the cooling fan running too loud (instructions here).

Running the fan at the lowest speed works fine most of the time since your CPU can throttle itself when it’s running too hot. However, with my setup I had an odd issue where the machine would power off suddenly when it was doing a lot of I/O workload, such as scrubbing the ZFS pool on the two 4TB SSD-s. This happened even when the CPU was not even close to throttling, running around 75 C. I do not know what the actual reason for the sudden shutdown is, but I suspect that it may be related to the motherboard overheating or the system treating the hot SSD-s as hard drives that are over the critical temperature limit of around 60 C.

I was able to work around most of the cooling-related issues by mounting the laptop vertically and limiting the maximum CPU clock speed to around 2.4 GHz using the knobs made available with intel_pstate.

The CPU temperatures during normal operation were mostly around 60C, with a high load reaching max 75C with the tweaks applied and the CPU fan running at the slowest possible speed.

Caveats: maintenance

If you have a machine that runs 24/7, maintenance will become an issue. No matter how often you clean your home, dust will accumulate. On desktop PC-s, cleaning out your CPU heatsink is just a matter of taking off the PC case side panel, removing the heatsink fan, cleaning off the dust and putting it all back together. You probably want to replace the thermal paste from time to time as well.

On most laptops, cleaning out the CPU heatsink is much trickier. You will likely have to disassemble most of the machine to even access it, which takes up much more time compared to a desktop PC, which also means more downtime.

Depending on the model, a simple change, such as upgrading your storage or replacing a faulty drive, can also be more complicated. I’ve seen laptops where you actually have to disassemble the whole machine to access the hard drive. The ThinkPad T430 has an easily accessible setup when it comes to storage, but the main storage bay does have an annoying pull-tab setup that probably won’t survive multiple subsequent changes.

Conclusion

Aside from the setbacks that may or may not be due to the specifics of my setup, I find that a laptop can make for a good home server. If you have more specific requirements, then a custom-built server is likely the way to go, but if you’re just starting out, then definitely do consider this option.

After running this setup for a week or two, I decided to go back to my previous setup based on the ASRock DeskMini. The reliability woes weren’t something I was too happy with, and I didn’t really have another use case for the DeskMini either, so it made sense for me.

Trying out VR on an AMD Ryzen 4000 series APU

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Wed, 04 May 2022 08:00:00 +0300

You may know that I really like small, efficient APU-based builds. At this point in time, they’re so close to being a viable mainstream option for gaming, especially the Ryzen 6000-series mobile APU-s. Forza Horizon 5, at 1080p high settings? Yes, they’re that good.

I don’t have the new Ryzen 6000 series APU-s to play around with, but what I do have is an AMD Ryzen 5 PRO 4650G and a HTC Vive Cosmos VR headset. I’ve noticed that my NVIDIA GTX 1060 manages to hit frametimes of 2-3ms in Beat Saber while the limit is at 11.1ms (90 Hz), which means that Beat Saber is probably quite easy to run. But can it run on integrated graphics and still be playable?

There’s been some work done in this area before as well:

Test setup

Here’s a quick rundown of the specs of the test PC:

CPU: AMD Ryzen 5 PRO 4650G
GPU: Radeon Vega 7 integrated graphics
RAM: 16GB DDR4-3200 kit
Motherboard: ASRock X570M Pro4
Cooling: adequate
Storage: plenty

To get the authentic, out of the box experience, I did not overclock any component, unless you count using the XMP profile on my memory sticks to run them at 3200 MT/s as overclocking.

The test PC usually has an NVIDIA GTX 1060 6G inside it, because it’s a gaming PC, after all. In my first test attempt I kept the NVIDIA GPU inside the PC since I was lazy. I connected my TV to the HDMI port located on the motherboard, and connected the VR headset to the DisplayPort present on the motherboard. What I found out was that by default games would still utilize the NVIDIA GPU, passing the final image to the integrated Vega 7 graphics, which then shows it on the big screen. This type of setup is commonly found on laptops that have a discrete GPU in addition to the integrated GPU, I just didn’t expect this to work out of the box on a desktop setup.

To do proper testing, I removed the dedicated GPU, updated the AMD GPU drivers to the latest available version at the time of writing, and got down to testing.

First attempt with Beat Saber

Beat Saber is a very popular VR game, and for good reason. It’s the perfect candidate for testing low-end VR gaming as it’s not that complex graphically and even runs on standalone headsets, such as the Quest 2.

I set everything up, started the game, and was surprised that it even loaded! The success stopped there, as I quickly noticed that the resolution used for the headset was automatically set to the lowest available resolution, resulting in text that was absolutely unreadable.

Unreadable text didn’t stop me, so I navigated to the song list and picked something familiar. The result? It works, barely.

First test with Beat Saber on integrated AMD Vega 7 graphics.

The game ran at a very low resolution, and even then the best it could do was between 45-80 frames per second. As you can see from the screenshot, the experience was not a fluid one. I could still hit the boxes, but it just didn’t feel good. At least it’s somewhat playable, and the headset still rendered my head movements in 90 Hz, reducing the risk of motion sickness.

Judging by the SteamVR performance statistics, I noticed that it was perfectly happy with running the game at half the framerate (45 Hz), and unhappy about inconsistent frametimes.

Closer look at the frametimes graph provided by SteamVR.

Test attempt #2

To understand the performance fluctuations better, I started up MSI Afterburner and enabled the on-screen display.

With the second attempt, I also made one important change to the configuration. I set the TV connected to the machine from 4K resolution down to 720p to avoid taxing the APU with any extra load.

I launched Beat Saber and while the resolution was still garbage, there was a notable improvement in the performance.

Still not quite ideal, but much more consistent than before.

The frametimes were much more consistent now and very close to the 90 Hz mark. It does still stutter, as you can see from the SteamVR performance statistics, but at least now it’s a much more playable experience.

Framerate and frametime graphs, courtesy of MSI Afterburner.

From the statistics provided by MSI Afterburner, you can clearly see that the performance is so close to being playable, with the integrated GPU maxing out at a somewhat stable 85 FPS, but then dropping down to a more stable 45 FPS and staying there for a while. The game fluctuates between those two levels of performance.

You can still call the game somewhat playable, though. It’s not the best experience and you’re not going to set any world records while running at half the refresh rate, but it’s something.

APU-s, VR and the future

The performance increases in APU-s aren’t fast enough to catch up with discrete GPU-s just yet. Relying on system memory that’s optimized for latency rather than memory bandwidth and then sharing that same memory with the CPU is also a compromise that you cannot work around that easily.

This test shows what the integrated Vega graphics are capable of. With the release of the Ryzen 6000 series mobile APU-s, the performance of the integrated graphics has been increased yet again, with the GPU architecture now based on RDNA 2.

Based on the results of my testing and seeing how the new APU-s perform in games, I’m betting that the new APU-s should be able to run some VR titles at a more stable 90 Hz on a similar screen resolution.

Is this enough to consider them as a viable option for VR gaming in general? Probably not. Games like Half Life Alyx can bring the NVIDIA GeForce GTX 1060 to its knees, and that’s at the lowest graphics preset! But if you’re like me who spends 90% of their time in VR playing a graphically simple game like Beat Saber, then it might be an option worth considering, especially in a market where discrete GPU-s cost an arm and a leg. The great GPU shortage of 2021-2022 is showing signs of easing, but we’re not quite there yet.

Future steps

It’s probably possible to improve the APU VR gaming experience by applying some tweaks and fixes, such as the VR performance toolkit. However, that won’t reflect the out of the box experience that you’d get. Casual VR players are unlikely to go and seek such tweaks out.

Trying to lock the refresh rate to something like 45/60/75 Hz may also improve the stability of the experience.

For the time being, I will just continue playing VR with a dedicated GPU, but I am cautiously optimistic about the future of VR on APU-s.

Strangling your service with a Kubernetes misconfiguration

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Mon, 18 Apr 2022 06:00:00 +0300

This is a quick story about a fun discovery that I made a while ago.

For legal reasons, all of this is made up and no such service ever existed.

Accurate representation of the state of affairs in the project, circa 2019.

Once upon a time, we had this Java service that handled all the backend work that you’d expect to occur for a product with a web interface. The service wasn’t the newest thing on the block and had seen dozens of developers work on it over many years.

At one point, it was moved to a Kubernetes cluster. That meant configuring all the bits and pieces. In YAML, of course.

A couple of years ago I joined the project and started work on it. Things were constantly on fire, so the fact that the service took anywhere from 5 to 15 minutes to deploy to the staging environment wasn’t something I focused on. After pushing a commit, the team either took a coffee break, went to play some table tennis or started work on something else while the pipeline did its job.

It didn’t help that running the service locally was hand-waved away as something that was too impractical to do by people who had been in the project longer than me, so the team simply relied on local tests and checking the staging environment after they pushed a change.

A year or so later, I had enough. The project wasn’t also constantly on fire, only occasionally, so I decided to take the time to dig into our CI pipeline configuration to see what is causing the pipeline to be so slow. My investigation lead to Kubernetes configuration, specifically the part where resource limits were configured. For whatever reason, I found that the service was allowed an absurdly low CPU allocation, measured in millicores.

I increased the limit to something sensible, such as 1 full CPU core. The result? The startup time of each pod took 30-40 seconds now, resulting in deployments that took 2 to 3 minutes max. This is an insane improvement over the old deployments that took up to 15 minutes regularly.

Artists rendition of the CPU usage patterns that the service exhibited, before and after the fix.

To go even faster. I tweaked the maxSurge property of the rolling update deployment strategy to start up more new pods in parallel, further shortening the time that it took to deploy the service. The only thing I had to keep in mind was the number of database connections afforded to each pod and the maximum number of connections offered by the database. Start up too many pods in parallel, and you’ll find that your deployment fails due to the service exhausting the available database connections.

Some time later I learned the reason behind such a CPU resource limit configuration. Apparently it’s a good practice to set your resource limits based on the average load that your service exhibits after it has properly started up. It does make sense, especially if you don’t want to have your Kubernetes nodes sit idle due to inefficient resource usage.

This example case shows that it’s a trade-off that you’ll have to take into consideration, especially if your service starts up quite slowly and not in mere seconds.

At the time of writing this post, the tweaked configuration is still up and running in production.

Oversimplified guide into snapshots on the btrfs filesystem

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Tue, 05 Apr 2022 06:00:00 +0300

Friday afternoon. You’re trying out a script that you wrote to mass-rename and move some files around. You finish the script and test it out.

Oops.

All the files now have all the wrong names, and some have been randomly moved 10 folders deep. It’s a mess.

And you didn’t make a backup of your files right before this step because you thought that you wouldn’t need it. It’s just a simple script, after all…

If you’ve found yourself in a similar situation and want to know how to avoid problems in the future, then this article is for you

btrfs: overview

btrfs (butter-fs, bee-tee-arr-eff-ess) is a modern filesystem present in the Linux kernel that has a lot of great features:

different RAID levels: RAID0, RAID1, RAID10…
data integrity guarantees
snapshots
and many more!

btrfs does have some downsides and has had its fair share of issues and controversy in the past, but at least during the last couple of years it’s been fine in my simple RAID1 + snapshots use cases.

In this post, we’ll focus on the snapshots feature that btrfs offers.

If you’re not sure what snapshots are, then you can think of them as a still image of the state of your filesystem at a certain point in time. You can create a snapshot of the filesystem, delete all your files, and still be able to get your files back just like you left them.

If you’re a developer who has used git and doesn’t mind oversimplified (and probably incorrect) explanations: you can think of snapshots like git commits. You can take snapshots (commit), roll back to a previous snapshot (reset/revert) or view the contents of a previous snapshot (checkout commit).

One thing to keep in mind with snapshots is that they will take up space, especially if you make a lot of them and move files around a lot.

There are many tools available to manage btrfs snapshots. The one I’ve opted for is snapper.

Snapper quick-start

Here’s a quick guide on how to get started with snapshotting your btrfs filesystem. The example is based on a system with a btrfs filesystem mounted on /home

Install snapper
- Ubuntu/Debian: sudo apt install snapper
- Fedora: sudo dnf install snapper
Setup the configuration: sudo snapper -c home create-config /home
/etc/snapper/configs now has a text file named home that you can configure to your liking. My personal recommendation is to use the TIMELINE snapshots configuration to configure the maximum number of hourly and daily snapshots.
snapper ships with two systemd timers that automatically take new snapshots and remove old ones. Make sure that they are enabled:
- sudo systemctl enable --now snapper-timeline.timer snapper-cleanup.timer
After a while, you should be able to see snapshots appear in the output of sudo snapper list.
If you want to browse the files for the old snapshots, go to /home/.snapshots.

Example output of sudo snapper list on my machine:

> sudo snapper list

    # | Type   | Pre # | Date                     | User | Cleanup  | Description | Userdata
------+--------+-------+--------------------------+------+----------+-------------+---------
   0  | single |       |                          | root |          | current     |         
   1  | single |       | K 16 märts 2022 08:12:01 | root | timeline |             |         
 242  | single |       | K 16 märts 2022 13:00:16 | root | timeline |             |         
 302  | single |       | K 16 märts 2022 14:00:05 | root | timeline |             |         
 362  | single |       | K 16 märts 2022 15:00:10 | root | timeline |             |         
 422  | single |       | K 16 märts 2022 16:00:19 | root | timeline |             |         
 461  | single |       | K 16 märts 2022 17:20:29 | root | timeline |             |         
 501  | single |       | K 16 märts 2022 18:00:02 | root | timeline |             |
...

That’s it!

Snapper tech tips

If you’re new to snapshots, then I recommend keeping an eye on your disk usage and tuning the snapper timeline related configurations to your liking. For my /home folder, my configuration looks like this:

# create hourly snapshots
TIMELINE_CREATE="yes"

# cleanup hourly snapshots after some time
TIMELINE_CLEANUP="yes"

# limits for timeline cleanup
TIMELINE_MIN_AGE="300"
TIMELINE_LIMIT_HOURLY="48"
TIMELINE_LIMIT_DAILY="7"
TIMELINE_LIMIT_WEEKLY="0"
TIMELINE_LIMIT_MONTHLY="0"
TIMELINE_LIMIT_YEARLY="0"

With this configuration, I can undo my screw-ups for up to 7 days, with more granular control for changes done within the last 48 hours.

If you have data that changes more often than other data on the same filesystem, then I recommend utilizing btrfs subvolumes and creating separate snapper configurations for each.

For example: a folder containing gigabytes of cat pictures is unlikely to change all that often, so a longer snapshot retention policy of 24 hourly/30 daily snapshots is a sensible one.

However, the contents of my downloads folder changes more frequently and may contain bigger files that I need temporarily, such as the ISO file for the latest Fedora/Debian release. In a situation like that, having a 6 hourly/7 daily snapshots policy will make more sense, especially if I don’t want to run out of space due to excessive number of snapshots taken for that filesystem.

My self-hosting setup has an UPS now, here's my experience with it

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Mon, 21 Mar 2022 06:00:00 +0200

I changed my setup again.

I’m now the proud owner of an APC Smart-UPS 750.

The UPS is many times larger than the actual server.

Why? Because I got it as a gift, and I have never actually had an UPS before, so it makes for a good introduction to this world.

Overview

My setup is quite basic: everything that I have on my table (server, monitor, laptop) is connected through the UPS. Since my setup uses very little power, I’m running way below the output limit of 500 watts.

When I’m not using my laptop, the UPS reports itself at 10% utilization. My server and the UPS itself seems to consume around 35-40 watts, as measured by an energy meter. When the laptop and monitor are also powered on, the load is around 15-25%, depending on the workload.

To make sure that I get the full benefit of the UPS, I connected it to my server with an USB cable and installed apcupsd. By default, the daemon could not detect the UPS, which was then fixed with the help of this StackExchange answer:

I had the same problem with COMMLOST with a USB connection, after a lot of research I found by pure hazard that with a USB connection you should change DEVICE /dev/ttys0 to DEVICE in /etc/apcupsd/apcupsd.conf with nothing after it, this way apcupsd search everywhere on the system to find the UPS and connect correctly, no more COMMLOST.

After that, restart apcupsd with:

$ sudo /etc/init.d/apcupsd restart

With the UPS connected and the service up and running, I could now see more detailed specs. The output for apcaccess looks something like this:

APC      : 001,043,1007
DATE     : 2022-02-04 21:49:08 +0200  
HOSTNAME : deskmini
VERSION  : 3.14.14 (31 May 2016) debian
UPSNAME  : deskmini
CABLE    : USB Cable
DRIVER   : USB UPS Driver
UPSMODE  : Stand Alone
STARTTIME: 2022-02-01 05:45:50 +0200  
MODEL    : Smart-UPS 750 
STATUS   : ONLINE 
LINEV    : 241.9 Volts
LOADPCT  : 9.1 Percent
BCHARGE  : 100.0 Percent
TIMELEFT : 98.0 Minutes
MBATTCHG : 5 Percent
MINTIMEL : 3 Minutes
MAXTIME  : 0 Seconds
OUTPUTV  : 241.9 Volts
SENSE    : High
DWAKE    : -1 Seconds
DSHUTD   : 90 Seconds
LOTRANS  : 208.0 Volts
HITRANS  : 253.0 Volts
RETPCT   : 0.0 Percent
ITEMP    : 33.3 C
ALARMDEL : 30 Seconds
BATTV    : 27.4 Volts
LINEFREQ : 50.0 Hz
LASTXFER : No transfers since turnon
NUMXFERS : 0
TONBATT  : 0 Seconds
CUMONBATT: 0 Seconds
XOFFBATT : N/A
SELFTEST : NO
STESTI   : 14 days
STATFLAG : 0x05000008
MANDATE  : 2007-03-16
SERIALNO : < serial number goes here >
BATTDATE : 2007-03-16
NOMOUTV  : 230 Volts
NOMBATTV : 24.0 Volts
FIRMWARE : 651.13.I USB FW:7.3
END APC  : 2022-02-04 21:49:10 +0200

I’ve seen some unintentionally funny abbreviations, but CUMONBATT takes the cake.

Battery runtime test

To test out the state of the batteries and understand how the UPS handles power failure scenarios, I decided to pull the plug and see what happens next.

The first thing you’ll notice is the beeping. It’s not quite as loud as a smoke detector, but I couldn’t blame my neighbours for thinking that this was the case. To silence the UPS, just press the power on/test button (the one with the | on it).

While on batteries, the UPS emits a hum that’s not too different from the one present near some power stations and transformers. Other than that, it doesn’t get awfully loud or anything, at least on a small load.

I continued using my laptop and monitor while keeping an eye on the apcupsd daemon and apcaccess output. The estimated runtime was around the 60 minute mark and it ended up being quite close to the actual figure.

Around the 10% capacity mark apcupsd triggered a shutdown of the server. The UPS still provided power at that point. I decided to supply the UPS with power again and turn on my server. To my surprise, the UPS now cut off all output power and started blinking some lights, one of them indicating some sort of an issue with the battery. I turned everything off, then on again and the UPS was just fine.

I’m not sure if the last incident was intended behaviour or my inexperience causing an issue.

Power usage

The goal of moving my setup to an efficient machine was to experiment with how low can I go with the power usage of my setup. Unfortunately with the introduction of the UPS I have wiped out almost all of the gains that I got from the move. The UPS itself seems to constantly consume around 17-18 watts, which is just a bit more than my server itself is using.

However, I hope that the reliability improvements and various protections that come with the use of an UPS make up for that downside.

LAN resilience

With the initial setup that I had, the server and the UPS were both on my desk. In the event of a power outage, the server, my laptop and all the connected peripherals would continue working. The only issue is that the networking won’t work in such a scenario, as both the ISP provided modem-router box and my own router would be knocked offline. What use is a server disconnected from the network?

Since I use my laptop for work, I’m not too worried about it losing power. Laptops have batteries, so in the event of a power outage, it will just continue running off of the internal battery.

I’ve configured my router to resolve my own domains internally, meaning that even when I cannot connect to the Internet, my locally hosted services will keep working in some capacity.

With all of this in mind, I decided to move my setup closer to my main router.

Not the prettiest setup, but it does do the trick, and it makes for a good conversation starter.

Power goes out, my server and local network will still keep on working for up to 2 hours due to the low combined power usage.

Desktop integration

When testing the UPS on my main workstation (ThinkPad T430 running Fedora 35), I was positively surprised by the out of the box experience that I had with the UPS. With the USB cable connected to the laptop and the apcupsd service running in the background, I could see the remaining battery life of the UPS the same way that the laptop battery shows up.

The UPS showing up as the main battery in GNOME 41.

And once I ran out the batteries, I got a notification saying that the UPS was about to run out of power soon.

Warning notification that pops up once the UPS is running low on power.

I love seeing these relatively niche use cases supported out of the box. Props to the developers behind these quality of life improvements!

Conclusion

Would I have gone out and bought this UPS if I didn’t receive it as a gift? Probably not, because I didn’t see much of a need for one.

Now that I have it, will I keep using it? Definitely, assuming that the inevitable battery replacement cost is reasonable and the process of changing them is easy enough for an enthusiast to handle.

From the manual:

For Professional Business Applications – Not For Consumer Use

Look, ma, I’m a professional now!

How I treat my urge to hoard data

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Tue, 08 Mar 2022 07:00:00 +0200

The tagline for /r/datahoarder reads: “It’s A Digital Disease!”. I agree.

Why can't I hold all these hard drives?

At some point I realized that the pursuit to hoard all the things will just keep on consuming more and more of my time and money. Storage is cheap up until to a point, once you find yourself tracking hard drive prices via camelcamelcamel and being excited that a 12 TB external hard drive is at it’s lowest price, you may start suspecting that you have a hoarding issue.

I don’t have anything against what people at /r/datahoarder and other archival efforts do. Data archival done by volunteers is what allows us to preserve history and cultural artifacts of our time (YouTube videos, memes etc.). I just had to find a solution to my situation before it got too bad.

Something is better than nothing

Once you find yourself with terabytes of data and decide to scale down, you will need to make some changes. Most of the time this results in deleting a lot of data that you don’t care about that much. This works, but there are some files that will be difficult to part with.

In my experience, most of the storage is taken up by media files. I would still like to have a copy of some videos, but if they are downloaded in 1080p or 4K resolutions, they will take up a lot of space.

That’s when I decided to change my approach with data like that. I would still like to have a copy, but it doesn’t necessarily have to be full quality all the time. When you rewatch YouTube videos that you have archived from 2006-2010s, you’ll notice that the video quality is bad. That doesn’t mean that you enjoy the videos any less. Watching these old videos is just an act of going through your memories and feeling nostalgic, the medium itself is just the spark needed to revive the memories.

With that, I decided to start following this simple rule: something is better than nothing.

I don’t need 4K versions of all the channels I follow. bestvideo[height<=1080] in yt-dlp configuration will limit the resolution of all the videos you download to 1080p, which is still good enough.

Data, value, and you

If you’re someone like me who watches YouTube frequently, then you probably have some channels that you care about a lot.

With channels that I truly care about, I still keep the highest quality copies of all their videos on my server. This has a real cost in terms of storage, but the value I have got out of the videos is much greater than that.

However, there are some exceptions. LinusTechTips is a great channel and has had a huge influence in my life. However, they upload almost daily and do long streams from time to time. Archiving the whole channel is just not feasible at my scale. Even when limiting the resolution to 480p, the full channel archive takes up over 300 GB of space in 2022. Instead, I’ve opted to recognizing the videos that are the most memorable, such as the petabyte project, and have archived those individually.

There’s also a certain type of data that isn’t that common: data created by you and your friends/family. Extra care and focus should be put on this type of data, because it’s very unlikely that someone else has a copy of gigabytes worth of cat pictures you’ve taken over the years. Make plenty of backups and don’t worry too much about hoarding it now, as long as it’s not a big burden for you.

Constraints

Constraints force you to get creative. It also helps with tuning down your data hoarding habit.

Set physical limitations to the data you can store. For example: limit yourself to running only two hard drives at a time. The only way to get more storage is to upgrade vertically to bigger hard drives, which is naturally throttled by the relatively slow increases in hard drive sizes.

For reference, my current setup has some pretty harsh limits when it comes to storage. The only way to get more is to spend a lot of money on bigger SSD-s.

The Internet Archive

If you don’t want to go through all the hassle, but still want to contribute to archival efforts, then you can donate to the Internet Archive instead. Buying and running all those hard drives isn’t free, you know.

And if you’re planning on uploading data to the Internet Archive, then please do not abuse it. Only upload data that you feel is worth preserving.

Surviving the front page of HackerNews on a 50 Mbps uplink

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Wed, 09 Feb 2022 06:00:00 +0200

Around a month ago I shared my blog post on HackerNews. I guess I lucked out with the choice of the topic, because it brought out a lot of enthusiasts who shared their own experiences with older machinery that still works in 2022. I really appreciate the feedback and the experiences shared!

Anyway, what is noteworthy in my opinion is that my blog runs off of a residential connection that has an upload speed limit of 50 Mbps. Once I noticed the post getting traction, I was worried for a moment. It’s not a rare sight to see a post on the front page of HackerNews and see it being down due to all the attention it got.

Somehow, my post managed to weather the storm. Here’s what happened.

Analyzing the logs

My page doesn’t have any sort of first-party or third-party analytics software running. Tracking users across the web is a big no-go for me and I will live by that, especially on my website. I do have nginx logs, though.

After a quick look around, I found goaccess, a tool that can parse nginx logs and put together some basic statistics. I found it to be good enough for my purposes. Here are some notable statistics.

To understand the HackerNews effect, note that the post was published on 2022-01-10 07:42. These logs also include requests towards services that I host myself.

The summary of the week during which one of my posts got popular on HN, as reported by goaccess, looks like this.

During quiet days, nginx logs around 50K requests a day. On 10th of January, there was a 12x increase. The effect was also noticeable the next day, during which I assume people caught up with their 100+ tabs that they usually have open. This may also be down to the post being shared around as well.

Based on this statistic, it seems that the post got around 75K views, either by real people, crawlers or preview generators.

When we look at the number of requests made towards specific assets, such as the images in the post, we can see that real impressions are likely to be around 55K. The first image has been downloaded more than the other two, likely indicating that it was used as the preview image when the article was linked on other sites.

This statistic also highlights a surprisingly large cost of web fonts.

The OS results are likely to be biased, since my self-hosted services go through the same reverse proxy. The “Unknown” section is likely related to crawlers and other bots pinging my server.

No surprise there: Chrome is the most popular browser used to reach my site. Firefox makes up a good chunk of hits as well, but a lot of those are likely requests made from my own machine.

The time distribution of the requests is interesting. You can notice the background noise associated with bots and crawlers that occurs during the night. My post was posted in the morning in my local timezone (UTC +2), which was around the time EU people wake up and get to work. The other spike around 14-15:00 can likely be attributed to our friends over the Atlantic ocean.

16:00 in my local time is a bit special. A lot of things coincide with this time:

the review embargo for the latest highly sought after CPU/GPU is over, followed by a barrage of videos on this product in YouTube
the stock market opens and news start rolling in about some big moves
if Slack is having issues, then it’s around this time, because I can only assume that people get to work and do a production release at the start of the workday.

To my surprise, my server survived this and there aren’t too many issues. The 400 and 500 errors are likely attributed to bots trying to exploit my server and crawlers visiting links that are not valid any more.

There’s likely a bias here as well regarding Europe.

How I built the blog

When I started writing blog posts regularly, I had some principles that I wanted to stick to:

the goal should be on the writing, not the part where I build the website itself
the site has to be static: build it once and deploy it, which should result in fewer opportunities for attacks and reduced load on the CPU

With this in mind, I decided to stick with Hugo. Hugo is just a single binary, has plenty of themes to pick from, and it seems like a reasonable-enough choice for a site. It’s not the easiest to use as you’ll be mainly operating in Markdown, and with the theme that I use, I have to copy-paste a <figure> section, change the image name that it points to, and make sure that I didn’t mess anything up. However, because this approach rules out me getting hit by the hottest Wordpress plugin vulnerability of the week, I think it’s a fair trade-off to make.

I picked this theme by panr and customized it so that it has a landing page of sorts as well. I haven’t updated it yet and it has some flaws, but it gets the job done, and that’s what matters to me the most.

Hosting media, such as images, seems like a no-brainer, just put them on the page and be done with it. Pictures taken with an iPhone SE 2020 are quite big, though (3-4 MB per image), which will result in the page loading very slowly. To avoid images bloating the size of the page too much, I have set up a system where I keep the original images in one folder, copy them to another one, run mogrify -resize 1024x768 -quality 85 *.jpg to keep the images small, but still detailed enough, and then deploy those converted images along with the rest of the blog.

With converted images, a page with three images can fit in less than 1 MB of transferred files without any issues. With original images, the same page would require 10 MB of files to be transferred. The math is simple: with a limited uplink, optimizing the images results in your server being able to serve 10x more requests.

Residential connections and DNS

If you’re like me, then you probably have a crappy router/modem box from your local ISP, and a dynamic IP address that usually changes whenever you reboot said box. This presents a challenge when you try to host anything from your residential connection since the IP address could change at any time.

To resolve this, there are two options I’m aware of: dynamic DNS providers, or setting up a script that talks to your DNS provider over a standard API.

I haven’t personally used any dynamic DNS providers, such as DuckDNS, mainly because my domain registrar has a handy API that I can use to automatically update my IP address with any time it changes. And yes, I did have an issue with the script where I triggered a change every minute, resulting in an angry e-mail being sent to me. Free tech tip: only propagate changes when the IP address actually changes.

There’s one downside with this issue: you can set your domain TTL (time-to-live) low, but no matter how low you set it, there will be a period of time after an IP address change where some DNS servers will point to your old IP address. This is an availability risk you have to consider when setting up a service on a dynamic IP address.

Future steps

At one point I took a look a the assets of the Hugo theme that I use and noticed that it includes a dependency named prism.js, which might be a good thing to include if you want to display code snippets with syntax highlighting. My issue with it was that it was included on every page load and took up a significant chunk of transferred data. I decided to remove it, and just like that, the page loads even faster.

There’s still room for improvement. The page currently also includes some custom fonts. If I decide that a built-in font is good enough, then there’s potential for an additional 0.3 MB of savings.

The web is bloated enough already, but at least I can control what I send to the client machines on my website.

Conclusion

If you don’t want to go through all the hassle and just want a website up and running, just get yourself a cheap virtual machine at a cloud provider, or use something like Github Pages. Those solutions are less likely to hit limits with bandwidth.

If you like tinkering and the decentralized, self-hosted web to exist (not Web 3.0!), then feel free to use this post as inspiration for your very own website. There’s a lot that you can do with limited resources, and it’s fun to push the limit.

And as a quick tech tip: you can use The Wayback Machine as an insurance policy for your website. If you are concerned about it going down, have them take a snapshot of it and link it somewhere in a comment.

ASRock X570M Pro4 motherboard overview

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Wed, 02 Feb 2022 06:00:00 +0200

Not too long ago, I had the goal of consolidating all my computing needs onto a single desktop PC. That meant moving from an mITX-based build to something that had a bit more expandability. Since I already had a Fractal Meshify C Mini, I decided to go for an mATX motherboard. And that’s how I ended up with an ASRock X570M Pro4.

ASRock X570M Pro4. Looks great, doesn't it?

mATX: a dying breed?

One of the requirements for the build was that it had to support IOMMU well so that a VFIO setup would be possible. While AMD B-series motherboards may work just fine in such scenarios, the X-series boards allegedly have better IOMMU groups and thus better compatibility with VFIO.

Once I started the search for the new motherboard, I was surprised to see that locally there was only one mainstream option available. There were plenty of B450 and B550 boards available, but only a single X570 board in this form factor.

I checked the board out, saw that it had plenty of NVMe and SATA storage options, and went for it.

PCIe connectivity

When you have a bunch of PCIe devices to connect, the way those lanes are connected start to matter.

Based on my own testing, I’ve found the following:

the first M.2 slot (under the heatsink) is connected to the CPU
the first PCIe x16 slot is also connected to the CPU
the rest of PCIe connectivity goes through the chipset
most (if not all) SATA ports seem to go through the chipset as well

Hardware topology, as shown by `lstopo`.

This matters, because anything connected to the CPU via the chipset shares the bandwidth of a limited x4 link. This was apparent when running two NVMe drives and noticing that one of them was much slower when I also put load on SATA SSD-s.

GPU-s can still work fine when connected via the chipset, but there’s definitely a performance penalty involved. While I don’t have the exact numbers available at this time, I do remember a CSGO benchmarking map showing at least a double-digit framerate difference compared to the full x16 slot.

The GPU works just fine in the chipset PCIe slot, albeit at lower performance.

I also gave PCIe bifurcation a go. I borrowed an ASUS PCIe adapter card that can support up to 4 NVMe SSD-s, hoping that I could set up a crazy storage setup. The bifurcation works, but for the CPU I used (AMD Ryzen 7 5700G) I was only provided the option of 2x4 split, which meant that at most I could run two SSD-s in the riser card at a time. A non-APU Ryzen CPU is likely needed to take full advantage of the bifurcation support.

The setting that controls bifurcation in ASRock UEFI settings.

ASUS Hyper M.2 PCIe card installed in the system.

For VFIO, this board is fine. Anything in PCIe slots, including the chipset ones, I was able to pass through to the VM. USB and SATA ports were a different story, I wasn’t able to pass through a single controller, the VM would not start up properly.

If you have an use case where you need reliable and fast PCIe connectivity, then I’d recommend another platform that has lots of real PCIe connectivity, such as the AMD Threadripper/EPYC platform. This board exhibited some odd behaviour when trying to utilize all the PCIe slots, or when trying to use the ASUS Hyper M.2 card in bifurcation mode while also having the GPU connected at the bottom PCIe x16 slot.

Chipset fan

One aspect of X570 boards that stuck out to most people on release was the inclusion of a chipset fan. Not too long after, people started to report that those fans are very loud and low quality, resulting in them failing easily.

For this particular board, ASRock provides an option in UEFI settings that allows you to run this chipset fan semi-passively. It will spin up once on boot and sit idle the rest of the time, unless the temperatures are high enough. I was not able to get that fan to spin even with a full GPU load and all SSD-s running heavy read operations at the same time.

If you have PWM fans, it’s also possible to manually control these in Linux with the use of fancontrol. For me, it just took a bit of trial and error to figure out the mapping between the software fan controls and the actual fan in the PC, after that I could fine-tune them to run as quietly as possible. Yes, even the chipset fan was controllable using the same method.

AM4 socket long term support

AMD introduced the AM4 socket back in 2016 and promised to support it until 2020. I applaud that effort and am happy that I was able to run the latest AMD Ryzen 5000 series APU on a B450 board. If you happen to have a newer CPU that requires a UEFI upgrade for it to be supported, then you will be in a world of hurt.

I found myself in a situation where I had a motherboard that didn’t have the newest UEFI version from the factory, and a CPU that was just a bit too new for it to work. The board supported AMD Ryzen 5000 series CPU-s, not 5000 G-series CPU-s. I also didn’t have an older CPU available that I could use to perform this upgrade on my own. Luckily I was able to find a loaner 5000 series CPU, perform the upgrade and finally assemble the machine.

At least ASRock puts a sticker with the UEFI version on the chip itself so that you can check the CPU support once you physically have the motherboard, but it’s pretty much useless when buying this board online. There’s no indication as to which version it will ship with, the best you can do is to look for some Reddit threads where others have shared their experiences with the board and the UEFI version it shipped with from the factory.

Pro tip for CPU and motherboard manufacturers: if you’re going for long-term support on your platform, please include a CPU-less UEFI upgrading solution. Some high-end motherboards already ship with that type of functionality, but I believe it should be made mandatory for all boards. Avoiding headaches like that will be worth the extra cost.

Verdict

It’s fine, looks good and gets the job done. Can’t recommend it for someone who needs a lot of performance and PCIe lanes, but for hobbyist workloads it will probably be just fine. UEFI flashing situation really soured the experience though.

ASRock DeskMini X300: the future form factor of desktop PC-s?

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Mon, 17 Jan 2022 07:00:00 +0200

I wasn’t happy with my “one machine that does it all” setup, which is why I finally bit the bullet and got myself an ASRock DeskMini X300 bare-bones kit. This is a fantastic little desktop PC kit that is very small, quiet and can still be configured to have ridiculous amounts of power. In the past. I once built a low-power server based off of the predecessor, the DeskMini A300, and instantly fell in love with this form factor. Since then, I’ve tried to come up with a reason to get one, and finally I’ve got one.

The ASRock DeskMini X300, up and running.

Configuration

Because this is a bare-bones kit, you have to supply your own parts.

My configuration:

CPU: AMD Ryzen 7 5700G, locked to 35 W TDP in UEFI settings.
Cooling: Noctua NH-L9a
RAM: 2x 16 GB DDR4-3200 SODIMM
NVMe storage: 2x Samsung SSD 980 500GB
SATA storage: 2x Samsung SSD 870 QVO 4TB
USB storage: 1x Samsung SSD 860 EVO 250GB in an IcyBox USB enclosure

Initial test configuration with Ryzen 5 PRO 4650G and 16GB of RAM.

The use case for my kit is simple: it’s my server that hosts all my services and data, including this blog you’re reading.

Why this setup in particular?

Most of my storage needs can be served by two 4 TB SSD-s.
8 cores with SMT are perfect for multithreaded workloads, such as Jellyfin transcoding. This also works well for just running a bunch of services on the same machine.
Can support up to 64 GB of RAM, which should be plenty in the foreseeable future.
Supports fast NVMe SSD-s, which are perfect for more IO-sensitive workloads, such as databases.
Low power usage. Even with my almost maxed out kit, it still uses around 14 W in idle. At most this PC can use around 50-60 watts due to the CPU TDP limit and SSD-s hitting full load.

Never assemble the whole PC without checking if it actually works.

Why is this form factor so good?

Desktop PC-s are big. Really big. While the small form factor PC movement has become more popular lately, the most commonly available cases and parts still end up taking a lot of space and resources. There are legitimate use cases for these types of setups, especially if you’re running video renders, scientific simulations or a lot of VM-s as part of a lab. Not every workload needs such a big setup, however.

The DeskMini X300 looks like a miniature version of a desktop PC in a tower case. Similar shape and features, but at a fraction of the size and raw resources usage. One might say that it even looks adorable, similar to how Raspberry Pi-based builds look compared to desktops.

This PC takes up just 1.92 liters in volume (excluding the power adapter). For comparison, a micro-ATX case like the Fractal Meshify Mini C, takes up a whopping 33.4 liters. I don’t have the raw numbers regarding material usage or the weight difference, but this comparison should give you an idea of the material usage difference between the two.

Included driver CD for scale. Not sure where you'd insert it, though.

Caveats

I love this piece of kit, but I do have to acknowledge that there are some trade-offs that you should be aware of.

Starting with the obvious one first: you’re trading off some expandability. No discrete GPU-s will fit in this thing, not even an external one due to lack of Thunderbolt. On the other hand, if you don’t foresee a need for such additions to your PC, then the DeskMini will be a great choice.

Something that’s specific to the X300 is the lack of USB connectivity. You get two USB 3.0 Type-A ports, one USB 2.0 port and a USB 3.2 Gen 1 Type-C port as well. If you have a USB hub, then this won’t be much of an issue, but this configuration is quite limited compared to most modern desktop motherboards.

The motherboard inside this thing can also be a point of concern. It’s a non-standard form factor, meaning that in case of failure you’re unlikely to easily find a replacement. The most obvious step here would be to either repair it or get another DeskMini kit to replace your existing one.

One aspect of the DeskMini X300 that you may care about is the fact that the power supply is an external 19V 120W power brick. This does mean that replacing the power supply won’t be much of an issue, but you do have to take this into account when planning your build. The power brick is quite big and has to be placed somewhere, after all. In my setup, I’ve opted for sticking it to the underside of my desk with double-sided tape.

How to hide bulky power supplies: the tutorial.

GPU-s, APU-s, and the future

For anyone who’s followed the GPU market in 2021/2022, the ridiculously high pricing of GPU-s shouldn’t come as a surprise. As a firm believer in thinking that low power computing is the future, I think that this presents a great opportunity to start thinking differently about our computing architecture.

AMD introduced the concept of APU-s back in 2011. In its current iteration in the form of AMD Ryzen 5000 series APU-s, they provide a compelling package, combining powerful CPU cores with a decent GPU that’s perfectly capable of many tasks, including low-spec gaming.

I’ve previously covered what I think about the state of computer hardware and the resource usage associated with it. It’s possible that we’re simply in a time in history where resource abundance allows us to push the limit of what computer hardware is capable of without thinking much about the cost. At some point we do need to address that issue, and I think APU-s will play a role in it.

We already have examples of what APU-like chips can be capable of in the form of the last two console generations. To get an idea about the power usage numbers, check out the power usage statistics shared by Sony for the PS4 and PS5. The power usage of the PS5 is still almost 200 W under load, which isn’t that great, but that’s likely due to the high performance goals set for the console itself. You’d have to give up some image quality or performance for a more efficient setup, which is something the marketing department isn’t probably very happy about, given that they’re in direct competition with Microsoft with their newest XBOX lineup that is also pushing the limits of the custom APU inside it.

The integrated GPU solutions that AMD and Intel have put out are becoming more and more capable. Even an AMD Ryzen 3 2200G from 2018 with its measly 4 cores and a Vega 8 GPU is still capable enough to run GTA V at 1080p with low-medium settings smoothly (60 FPS or more). You do have to make a conscious trade-off in image quality and performance when relying on APU-s, but if you’re willing to make that trade-off, you can get a decent gaming experience.

Even the Steam Deck is shipping with a custom AMD APU. What makes this one different from AMD-s previous APU-s is the inclusion of a modern GPU core based on the RDNA 2 architecture, which should be a big jump from the relatively old Vega-based GPU-s that desktop APU-s have shipped with.

If it weren’t for the progress in gaming tech, such as high refresh rate monitors and ever-increasing system requirements for newer games, APU-s would be a good default choice for casual gamers, negating the need to get a huge box that you have to somehow also fit on your desk.

I have run on an APU-based setup and was happy with the experience, unfortunately an APU-based machine just doesn’t cut it for virtual reality workloads, yet.

Comparison to TinyMiniMicro PC-s

ServeTheHome’s TinyMiniMicro series gives a good overview of the options available for someone who wants a very small 1 liter PC to do all their work on. Since these machines are also very small, power-efficient and yet still quite powerful, I think a comparison between the two is a reasonable one.

One area where the TinyMiniMicro PC-s win is an obvious one: the size. They are about half the size of the DeskMini X300: 1 liter vs 1.92 liters.

TinyMiniMicro PC-s are also abundant on the used market with various configurations. In most cases, the storage and RAM are upgradeable, and with some older models you’ll find that the CPU is also socketed. Depending on your market, this means that you may be able to get a whole PC for cheaper than the DeskMini + the rest of the components (CPU, RAM, storage, Wi-Fi etc.).

The size and cost benefits do come with some compromises that are not acceptable for my use case. At 1 liter, the only option you have is to include a blower type cooler in your PC. This type of cooler is similar to the ones found in laptops and has in my experience resulted in a lot of annoying noise under heavy load.

The UEFI settings on these machines may also be quite limited. Sure, if you just need the basics, these will be fine, but if you want to configure the fan behaviour, you might find yourself unable to do so. On the DeskMini, you can setup the CPU fan to run just the way you like it. In my case, the fan only turns on when the system is under load, the fan is completely stopped while the machine is idle.

The DeskMini also allows you to change out the CPU cooler with anything that’s AM4 socket compatible. The case of course sets some limits to that, unless you’re a madlad who doesn’t let boundaries bother them. With the Noctua NH-L9a AM4, the setup is very quiet, especially once you limit the CPU TDP and adjust the fan curve. In my experience, the “coil whine” noise that the system produces under load is more audible than the CPU fan itself.

Low profile Noctua cooler: looks good, is quiet and keeps the system cool.

When it comes to storage options, the DeskMini is an obvious winner. In modern TinyMiniMicro machines that I’ve seen, you can at most install two NVMe SSD-s. This is a step up from older machines and you can still do a lot with that, but the DeskMini trumps that with the addition of two SATA ports, which allows you to go crazy with the storage configuration.

8TB of solid state storage. Bonkers.

Both the DeskMini and TinyMiniMicro-series PC-s are fantastic, but for my use case the DeskMini X300 was the obvious choice.

Tech tip: VESA mounts, for what?

The DeskMini X300 is so light and small that you can easily attach it to a monitor arm using a piece of foam and long velcro strips.

DeskMini X300 mounted on a monitor arm.

Behind the scenes.

Future plans

I’m hoping that the current DeskMini-based server configuration is the one that will last me a while. The obvious upgrades in the future would be to max out the RAM at 64 GB and upgrade the NVMe and SATA storage to whatever options are available in the future. I’m not expecting the DeskMini X300 to get a CPU upgrade, which isn’t too tragic, given that the Ryzen 7 5700G has 8 fast cores.

My adventures in low power computing may take me to another setup in the future. Until then, I’ll see how the DeskMini does. So far, it has met my expectations in performance, power usage, reliability and noise.

Future of desktop computing?

Predictions are difficult to make, especially in turbulent times that we’ve found ourselves in. I do hope that this form factor takes off and that we reevaluate our current computing workloads to be more suitable for machines that use a lot less power.

Constraints often result in clever tricks and fantastic end results. Hopefully we’ll see something similar in the future.

To get an idea of what you can achieve with very limited hardware, check out the Coding Secrets YouTube channel that showcases all the tricks and thinking behind some of the most impressive effects on game consoles from early 90’s.

Sure, you can argue that not having to think about such limitations can help create solutions faster and that definitely has some value, but at some point we need to start optimizing. Computing resources are cheap right up to the point where every app on your PC thinks the same way.

Why I went back to using a ThinkPad from 2012

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Sun, 09 Jan 2022 11:00:00 +0200

This post is inspired by this article from the Low-tech Magazine. I highly recommend giving that a read as well!

Over my lifetime, I’ve used a bunch of different computers, mainly due to new ideas and requirements popping into my mind every time I’m content with my current setup. One of my last changes might be a bit of a headscratcher for some.

ThinkPad T430 in all its glory.

Background

At my current place of employment, I’ve had the opportunity to try out recently released laptops from Lenovo and Dell, with the goal of evaluating them for software development workloads and pick a default option for new hires. My work laptop at the time was the Lenovo ThinkPad T480 with some decent specs and an NVIDIA GeForce MX 150 GPU that was weak and throttled itself whenever it hit 70 degrees Celsius in Windows 10, making it absolutely useless.

I tried out a fair number of makes and models: Lenovo ThinkPad P14/T14 (gen 1 and 2) in both Intel 11th gen and AMD Ryzen 4000/5000 series configurations, plus some Dell Latitudes with Intel 11th gen CPU-s as well. And the result? A lot of disappointment.

The non-exhaustive list of issues I ran into with these machines on Fedora Linux 34:

The touchpad would sometimes randomly not work on one of the ThinkPads.
On AMD models, performance was less than stellar for my workloads and not a significant jump over a laptop from 2018.
The Intel CPU-s had throttling issues that made them unusable for basic things like calls over Google Meet as they would throttle to 400 MHz.
On one of the Dell machines, it was very easy to overheat the SSD, which lead to the CPU throttling to 400 MHz. Yes, the SSD caused the CPU to throttle.
Wi-Fi/Bluetooth would not work on one of the laptops, even when I was using an up-to-date Linux distro. Likely related to the type of adapter used (not Intel).

Once that experiment was done, we settled on the least crappy version of the ThinkPads that had an AMD Ryzen CPU, at least those didn’t sound like jet engines under load and didn’t have insane throttling issues.

It’s not all that bad, though. I was now committed to using my current ThinkPad T480 for as long as possible. It, too, had a rough start, but at least all the issues it came with have been ironed out over time.

Hopping between machines

At this point in time, I had three machines:

desktop PC (for work and personal stuff)
work laptop (for work stuff only)
personal laptop (for personal stuff, of course)

I could not rely on the work laptop for personal use as it has limited storage options. Call me a freak, but I feel very uneasy running on a single SSD, even if my data is backed up to my NAS and on external backup drives. With my desktop PC use case affecting other workloads as well, I had to come up with a solution.

Syncing data between machines was not the issue here. Syncthing is an absolute open-source gem and had no issues with things like node_modules folders. The main issue was the fact that I didn’t want to carry two laptops around or buy an extra USB-C dock for use at home.

I took the risk and jumped back to the ThinkPad T430 for both my personal and work use cases.

ThinkPad T430: the history

My ThinkPad T430 has a rich history. I got it in 2016 to replace my aging ThinkPad T60. Getting this laptop felt similar to that time a desktop PC that ran Windows 98 got replaced with a dual-core “beast” in 2006.

In 2006, my webpage load times went from 30 seconds to a second. Getting Android app build times from 60 seconds to around 10 seconds on the T430 felt the same.

This ThinkPad T430 has survived all the following:

A bicycle crash while it was in a backpack. The latch mechanism broke and there are a couple of cracks in the palm rest, but the rest is working fine.
A coffee spill, which luckily only discoloured the casing of two USB 3.0 ports.
Liquid metal experiments, had some pretty close calls there with my dumb ass almost shorting the system out.
Use as a budget low-power server machine for a bit.

5 years later, it’s back in my possession again after a short stint at a family members’ hands.

There’s more to this than simply a rich history, though.

The modifications

Over the years, I’ve been inspired by whatever modifications people over at /r/thinkpad have come up with. If you want a short summary, you can check out the definitive T430 modding guide to get some inspiration.

To make this machine viable in 2021, you need the CPU, RAM and SSD upgrades at the very minimum.

Here’s the list of upgrades I’ve done:

CPU upgrade to a quad-core Intel i7-3820QM (45W TDP).
RAM upgrade to 16 GB DDR3 memory.
Storage: 250GB mSATA for OS, 2x 1TB SATA SSD-s for data, made possible with a HDD caddy that replaces the optical drive.
New third party 9-cell battery. I’ve had mixed experiences with these, but the one I have now seems to be good enough.
Replaced the heatsink and fan assembly with one that has a Delta fan (FRU 04W3270). It’s quieter and doesn’t have the high-pitched whine that the Toshiba fan exhibits.
At one point, I ran an external GPU off of the ExpressCard34 slot.
Installed 1vyrain to get rid of annoying limitations, such as the Wi-Fi whitelist.
Upgraded the Wi-Fi card to Intel Wireless-AC 7260.

Delta fan (top) vs the annoyingly loud Toshiba fan (bottom).

There are some things I’d like to eventually tinker with:

Replace the display with a compatible 1600x900 panel that has better image quality. The current screen is awful.
Do something fun with the ExpressCard34 slot. There’s the thinkmods.store ExpressCard34 SSD adapter, but I haven’t seen that released yet.

eGPU setup that I used to run back in 2017 with surprisingly good results.

With these modifications, it feels like an usable laptop again.

But why?

The reasons why I went back to the T430 are quite simple.

Build quality: it has survived a nasty fall, and it will probably survive the next ones as well.
CPU performance is roughly on par with the T480 under sustained load.
No dedicated GPU that takes up valuable space and power.
Battery life with the 9-cell battery is on par or even better in real world use scenarios, compared to the T480.
Ridiculous storage configurations: dual-boot, triple-boot, RAID1 setups, 16+TB of solid-state storage, it’s all possible on this old laptop!
Decent selection of ports, including VGA and gigabit ethernet.
Docking stations can be dirt cheap and be bought for less than 10 EUR (used).
I have plenty of Lenovo barrel plug chargers, but only one USB-C Lenovo charger. I really didn’t want to buy more chargers when I have plenty of perfectly working ones.

There are some downsides, though:

No HDMI, I’d need a mini-Displayport to HDMI dongle for that to work.
No USB-C, which might be an issue if your environment at work is optimized for that.
It runs hot, but doesn’t have the same level of on-chip thermal protections that the T480 has, resulting in this awful piece of software I wrote back in 2018.

That’s nothing compared to the unreliability I’ve experienced with the T480. I’ve witnessed the keyboard dying once and the motherboard being replaced twice, once due to charging related issues, and the other time due to random system crashes and screen glitching.

Verdict

I’ve been using this setup for over a month now, and it has been surprisingly adequate. Yes, opening Java projects in IntelliJ will make things slow, and to record my desktop with OBS and acceptable performance, I had to drop my screen resolution to 720p. I can’t expect everything to work super well on this machine, but for a computer that’s released almost 10 years ago, it’s still holding up well.

I’d like to thank Intel here for making this possible. The CPU innovation stagnation between 2012-2017 has resulted in 4 cores still being an acceptable low-end CPU in early 2022. Without this, my laptop would likely be obsolete by now.

2022-01-12 update

This post got a lot of feedback and comments on HackerNews with people sharing their experiences, feel free to check it out and join the discussion!

2022-09-01 update

This post was translated into German and featured on golem.de, check it out!

VR, VFIO and how latency ruined everything

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Wed, 29 Dec 2021 06:00:00 +0200

I’ve been running my all-in-one PC for a while now. It was my desktop, my NAS and my gaming PC. However, during the next couple of months I kept hitting small bumps along the road. Most of these were quite straightforward to fix, but there is one that finally convinced me to go back to a simpler setup.

iSCSI

As mentioned in one of my previous posts, I had set up the storage on my gaming VM over iSCSI. If you don’t know what iSCSI is: think of it like a hard drive that you can hook up to your machine over the network. This setup had a couple of benefits: the amount allocated to the VM was flexible and I would always have a snapshot of my game data in case the Windows VM decided to nuke itself.

What I didn’t take into account was the configuration. tgt allows you to define the IP address of the client machine, which can be seen as an additional layer of security. My gaming VM has two virtual NIC-s, one pointing to the actual physical network, and another one to a virtual one. This resulted in the iSCSI connection being flaky on startup: the disk was not present in the VM on first boot, but was there after a reboot.

This issue was gone once I removed this restriction from the iSCSI target configuration on the Linux VM. My best guess is that the iSCSI stack in Windows might have tried to initiate a connection over the wrong NIC, depending on whichever one was brought up first.

After “solving” this issue, it had been smooth sailing.

GPU audio issues

When the Windows VM had been on for a while, the HDMI audio output would start to glitch out after a while, with the audio sounding slow, robotic and garbled. After doing some research, I found some resources that mentioned enabling MSI interrupts in Windows. The guide for this can be found over at Guru3D forums.

After tweaking the Windows registry settings and reminding myself that I do have snapshots to fall back to in case I mess up, I got the issue fixed.

GPU driver issues

Whenever there’s a new big and popular game released, NVIDIA usually releases an updated version of their GPU drivers. I try to keep my software up to date, so I usually install these updates when they come out. Just one problem: when I upgrade the GPU drivers in the VM, the screen will stay black. After some time passes, I have to force a reboot using virt-manager.

I have not looked into this issue yet as it only happens during driver updates, but it’s definitely an annoying thing to have.

UEFI and CSM

UEFI updates have a bad habit of resetting all the custom settings that you have made, and at least with the motherboards I have used, the settings cannot be restored if the backed up settings are from an older UEFI version. This means that I have to navigate the UEFI quite often to configure all the settings that I need to have in place for VFIO to work.

There is one setting that I didn’t expect to cause that much trouble though: CSM support. CSM (compatibity support module) is generally enabled by default and allows you to boot off of legacy operating systems if needed. I don’t have that requirement, so I went ahead and set my boot options to boot in UEFI mode only.

Reboot, start up Fedora, and boom, anything VFIO related is spewing errors like crazy.

Turns out that enabling UEFI only mode in UEFI settings somehow grabs the GPU and causes problems when you try to then pass the GPU to a VM. While in this mode, I noticed that the GPU was also shown in a separate menu in UEFI settings. Enabling CSM support fixed this issue for me.

VR and USB issues

With the situation in the world being less than optimal, I decided to spice up my gaming setup with the addition of a HTC Vive Cosmos in case in-door activities become much more popular suddenly.

First issue: I can’t get the damn thing working. My gaming VM setup has the inateck KT4006 passed through so that I can plug peripherals to the PC and make them available to the gaming VM with ease. After plugging everything in according to instructions and turning the link box on, I’m met with a barrage of USB connected-disconnected sound effects. 6 or 7 cycles later the sound stops, and with that do the rest of my USB devices.

I took the USB 3.0 card and tried it in an older machine that doesn’t have a fancy VFIO setup on it. Same issue. However, when connected to an USB 2.0 port, the setup shows a warning in VIVE Console, but at least it works! Since my PC literally has no USB 2.0 ports, I had to get a bit creative and plug an USB 2.0 extension cable between the PC and the USB 3.0 cable coming from the link box.

With the help of my friend I got confirmation that the issue is indeed with my USB ports. I also ordered a different USB PCIe card from inateck, this time the inateck KTU3FR-4P. I made the assumption that since the VIVE support page also recommends an inateck card, then they must be fit for purpose.

Nope. The second card apparently has the same chipset and the same issue.

What did end up working was a random VIA USB 3.0 PCIe card, which shows up as VIA Technologies, Inc. VL805/806 xHCI USB 3.0 Controller on my machine. Just one issue: it has PCIe device reset issues. If I shut down the Windows VM and start it up again, I would sometimes get a bunch of IO_PAGE_FAULT errors in my kernel logs. No worries, guess I’ll just avoid rebooting a notoriously reboot-happy OS.

Latency: the straw that broke the camel’s back

If there’s one thing you don’t want to see in a VR gaming setup, it’s latency.

I’ve covered the tweaks I’ve made to my setup in a previous post. Turns out that those might not be enough. The CPU isolation using systemd works well in general, but not with kernel threads.

Whenever I did something IO heavy on an NVMe-based btrfs file system, it would introduce stutter to the gaming VM. Running a scrub operation on the filesystem was enough to put a huge load on the CPU, likely due to the raw speed that the NVMe SSD-s can provide. This results in the gaming VM experiencing stutter that ruins the whole experience.

Here’s an illustration.

SteamVR performance statistics with a light load on the host system.

The purple spikes mean that you are having a real bad time right now.

For comparison: VR performance running natively on Windows 10.

Similar issues could be observed whenever I tried to run another VM that shares the same cores.

I looked into alternative solutions as well, and it does seem to be possible to also isolate those CPU cores from the kernel. They will probably work to some extent, but implementing them feels a bit hacky at the moment. I’m hoping that in the future these kinds of tweaks will be exposed as a simple checkbox in virt-manager itself.

At that point I had enough. If I’m not going to be able to use the machine for other purposes at the same time without experiencing all these issues and having to work around them, then I’ll just go back to a simpler setup. I’ll lose some of the benefits, but at least I don’t have to spend time debugging all these issues.

Conclusion

VFIO and virtualization are still interesting topics to learn about and try out, but for my use case they are just too limiting. I’d imagine that an use case that didn’t have such strict latency requirements would still be able to run just fine, unfortunately gaming just isn’t one of them.

This experience has been quite fun in general, but I’m calling quits, at least for now. The one machine that does it all turned out to be a jack of all trades, master of none.

Self-hosting Wikipedia using Kiwix

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Thu, 09 Dec 2021 06:00:00 +0200

Do you have a need to host Wikipedia on your computer?

Or a StackExchange site, like Super User?

It’s easier than you think!

Step 1: download stuff!

The Kiwix project provides ZIM files for many popular websites, including Wikipedia. These files can be downloaded over at Kiwix wiki.

Take a look at the content packages available and download those that you are interested in.

Step 2: install Kiwix

Kiwix provides all sorts of clients for different operating systems. If you don’t want to go crazy with setting up a server and you just want to browser your offline copy of Wikipedia or any other resource, then go ahead and use that.

However, if you want to make the resources available for other people as well, then you need to install kiwix-serve. On Debian and Ubuntu systems, this is just a sudo apt install kiwix-tools in your terminal.

Once you have it installed, all you have to do is to run kiwix-serve -p 8080 your-file-here.zim and your content package is served to anyone who can connect to your machine with a browser over port 8080.

If you have multiple ZIM files in the same folder, then you can serve all of them at once with something like kiwix-serve -p 8080 *.zim.

Step 3: enjoy!

You should now be up and running with your very own hosted Wikipedia instance! Just keep in mind that your copy might not be 100% up to date. The Kiwix project does repackage these sites quite regularly so keep an eye out for updates.

Tips and tricks

I have set up my own Kiwix instance and the way it works is quite simple.

I use Deluge to download and seed the content packs that I have downloaded.

Then I have also set up a systemd service that automatically starts Kiwix on boot. It is located at /etc/systemd/system/kiwix-serve.service and looks like this:

[Unit]
Description=Serve all the ZIM files loaded on this server

[Service]
Restart=always
RestartSec=15
User=kiwix
ExecStart=/usr/bin/bash -c "/usr/bin/kiwix-serve -p 8080 /path/to/your/files/*.zim"

[Install]
WantedBy=network-online.target

Note that I have set up a separate non-interactive user for this purpose. The path to the ZIM files simply refers to the download folder of the Deluge torrent client.

Making this available to the world is simply a matter of either exposing the selected port to the world, or using nginx as a reverse proxy. There are plenty of guides available on how to achieve the latter so I won’t go into detail about that here.

That’s it!

2024-09-05 update

It’s been a fun experiment, but LLM-bots are overwhelming my Kiwix instance and I can’t be bothered with setting up any protections right now, so the kiwix.ounapuu.ee instance is shut down for now.

Tech rants: PC-s use way too much power in 2021

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Wed, 03 Nov 2021 07:00:00 +0200

Welcome to 2021. We have:

supply chain issues
no reasonably priced GPU-s
consumer-grade CPU-s with peak power consumption at 296W
GPU-s that consume 350-400W of power under normal use

Accurate representation of CPU-s in 2021.

At the same time, we have made great leaps in CPU/GPU architectures and chip manufacturing technologies, which should result in faster and more efficient devices, right?

Well, yes. However, with some fierce competition between AMD vs Intel and AMD vs NVIDIA all reason is thrown out the window and the power limits are raised in order to preserve the performance crown. In the end, all that matters is “but my CPU is 5% faster in this benchmark!” or “Oh yeah Intel has the performance crown even if it took 290W to get there”.

This all sounds absurd during a time when it’s clear that energy usage is becoming a big problem in the near future. Using up more power daily also makes it more difficult to rely on renewable (not necessarily green!) energy sources due to the simple fact that building more capacity is more expensive.

All that power, but at what cost?

What is often missing from this conversation is that it’s not only the CPU or the GPU that uses up a lot of energy while running.

When you have a hungry chip in your system, you will likely need to get a bigger cooler in your system to dissipate all that heat. Whoops, more raw materials (aluminium, copper) required.

The GPU is so hot that the manufacturer had to put on a giant heatsink, multiple fans and made the whole darn thing barely fit in your case. Same story.

Something has to deliver all that power as well. And just like that, the required amount of electrical components on the GPU board or the motherboard has just multiplied.

The hot air gets stuck in your PC case? Sounds like someone is going to need to make an investment into some PC fans.

And to finish it all off, you discover that your power supply cuts out during high loads, because 800W power peaks are acceptable now I guess. Time to go to your local PC parts store to get a new power supply.

You might look at your monthly electricity bill and not even notice the power required to run such a machine, but the thing is that the real cost comes from everywhere else. That cost is not low.

Okay, you convinced me, now what?

If you already have bought hardware that fits the above description, then there isn’t much to do other than limiting the power usage. CPU-s and GPU-s generally follow this kind of rule:

Rough representation of the relationship between power usage and performance of a chip.

If you use Windows, then one example of making a positive change is to change the power limit of your GPU. For AMD GPU-s, this is present in Radeon settings and is a simple percentage slider. For NVIDIA cards I usually opt for using MSI Afterburner. While doing any changes, definitely have something like FurMark running in the background, that helps measure performance and get readouts for your current power usage and the performance that you get out of it.

A modest reduction of the power limit will likely yield a small drop in performance, but a much bigger drop percentage-wise in the power consumption of the GPU.

Power limit slider present in MSI Afterburner.

For CPU-s, there are two types of options:

configure power limits in UEFI settings (depends on your CPU and motherboard manufacturer)
limit the CPU power usage using tools available in the operating system

Here are some examples of the knobs and tools that you can use to achieve this:

AMD CPU-s and UEFI: cTDP option in UEFI settings. Just set the TDP that you want to run the CPU at and you’re done! Allows you to run your 105W TDP CPU at 35W, for example.
AMD Precision Boost Overdrive (PBO) and UEFI. Sometimes the cTDP setting is not present, which means that you’ll have to go to the overclock settings and input the correct parameters to set the same limit.
AMD and Linux: cpufreq. Honestly, not the best way to do it since the steps don’t seem to be that granular, but it’s better than nothing.
Intel and UEFI: supported CPU-s likely support similar techniques to AMD, but since I don’t have experience with Intel, I cannot say what exactly you can change.
Intel and Windows/Linux: disable turbo boost. Peak performance will definitely suffer, but you end up running your CPU at the base clock, which is likely very close to the efficiency point of the cpu.
- in Linux, use the intel_pstate driver.
- in Windows, you can set the “Maximum processor state” under the legacy power options UI to 99%, which will disable turbo boost.

This list is not complete and there might be better options out there. Just go with the one that is easiest for you and prefer UEFI-level settings to any OS settings as those will persist even if you change your operating system.

I changed my mind. Why should I care?

Using less power actually comes with a lot of benefits:

smaller electricity bill (just don’t expect any dramatic changes)
your PC will run much cooler now, negating the need for a big and wasteful cooling setup
you can run your PC much quieter now as well, less heat to dissipate -> fans don’t need to run as fast now
you can pick cheaper (in price, not quality) options when it comes to PC components since you don’t need all that overbuilt capacity.

Why do you care?

Once you limit yourself with the amount of resources available to you, you’ll soon discover that you can do a lot even with a small power budget. I’m running all my self-hosted services, gaming and work off of one decently configured PC that is quiet and yet powerful enough to perform well at any task I throw at it. If you can do all of that with a CPU that was originally designed for laptops and has a rated TDP of 65 watts, then why get anything more powerful?

Apple has joined the game

We were in a Intel-lead stagnation, with new chips adding minor features and introducing small improvements in performance. AMD came out with the first Zen-based CPU-s in 2017, followed soon by APU-s that were used in laptop designs as well. At that point AMD had almost closed the performance gap in the laptop space, forcing Intel to up their core counts and the turbo boost power limits as well. With the introduction of Ryzen 4000 and 5000 series, AMD went above and beyond and managed to bring great CPU and GPU performance to the laptop space, with the one notable omission being Thunderbolt 3 support.

Then Apple came out of nowhere and provided heavy competition for both companies. The new Apple M1 chips are fast as hell for what they really are and consume so little power that the fans in the laptop rarely have to turn on. You also get the benefit of having an amazing battery life. That same design has also found use in the desktop lineup in the form of the new iMac, and who knows, maybe it will also end up in a proper workstation machine at some point?

While I don’t like the software stack that MacBooks ship with, and the poor repairability is a major downside for me, I still have to admit that Apple is moving in the right direction with their chip designs. Performing well without your laptop turning into a poor man’s version of a jet engine is exactly what we should be striving for.

Closing thoughts

I understand that the reason we often end up using the latest and greatest hardware is a result from the need to perform our work as fast as possible, because you’ll end up using less time, and less time spent on a task should result in more productivity. However, this has to be more sustainable, and continuing the trend that the “big boys” are going in goes against that.

I hope that we will see more progress in this area, especially from companies other than Apple.

DIY cloud gaming: NVIDIA and Moonlight

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Wed, 13 Oct 2021 05:00:00 +0300

As you might have read from my previous post on this topic, I have a pretty neat “cloud” gaming setup running. I have one powerful desktop PC, one virtual machine with a GPU attached to it, and client machines that can be used to stream games from. To keep things simple (and because my current ISP is unbelievably bad), I have so far used streaming only over my local network.

However, there were some pretty annoying issues with this setup:

The AMD GPU encoder is bad. 720p streaming was the best I could do if I wanted reasonable performance.
The Windows 10 VM would sometimes not be able to start the host encoder, and the GPU drivers reported issues. This was fixed with a couple of reboots to the VM and I suspect that this may be related to the notorious GPU reset issues that AMD cards are infamous for in the VFIO world, but I wasn’t interested in digging that deep into that topic.
The stream would sometimes randomly crash or lag. This might have been down to the AMD GPU or Parsec itself.

I read about this topic for a while and found that the overwhelming number of threads related to this topic recommend an NVIDIA GPU and Moonlight for the best streaming experience. And so I went out to get one.

The GPU

It’s 2021. The world is on fire and GPU prices are insane. With that in mind, I tempered my expectations and went for something simple: a NVIDIA GTX 1060 6G by MSI. Simple, relatively modern for my games and the price was around the retail price 5 years ago (260 EUR), which is the best you can do in these lousy times.

But hey, it works! From the testing I did in a previous post, I knew that NVIDIA had finally stopped intentionally throwing error code 43 in their drivers, which made this option viable. Replacing the GPU was simple, I just had to change the PCI device ID-s according to the VFIO setup guide and changed the devices attached to the Windows 10 VM using virt-manager, and that was it!

Parsec: AMD vs NVIDIA

Since I was already using Parsec for my streaming purposes, I decided to do a quick test to see what the difference is between the two cards.

The previous GPU was an AMD Radeon RX 570 4GB by Sapphire. These two GPU-s are in the same performance class, which makes for a good comparison.

Here are some results taken in two quick testing sessions in Dirt Rally. Take these results with a spoonful of salt as these have not been conducted with any kind of standards. I just picked the same graphics preset, resolution and let it run with Parsec statistics view open.

H.264

AMD RX 570: 10ms

NVIDIA GTX 1060: 3ms

H.265

AMD RX 570: 8ms

NVIDIA GTX 1060: 3ms

The difference in the encode performance is huge, especially if you consider that the time between two frames is 16.6ms for a 60 FPS stream. Some of that budget is also taken up by network and decode latency in the client, so the shorter the encode time is, the better.

Parsec is fine, but I have had some trouble getting GPU-accelerated decode working on my Linux clients, and it also lacks an iOS client. While the latter isn’t a deal-breaker, the first one definitely is.

Moonlight: an open-source gem

Moonlight is an alternative to Parsec. It implements the NVIDIA GameStream protocol and has clients for most common platforms. My testing has been performed on Linux clients running Fedora 34, a Windows 10 laptop (ThinkPad X230), iPhone SE 2020 and an old Google Nexus 5. I also made an attempt to run the Moonlight client on a Raspberry Pi 1 B+, but it seems that it is simply too old for the Moonlight client, resulting in an error on startup.

GTA V on a Google Nexus 5? More likely than you think.

To set up Moonlight, I opened up the client and tried the automatic detection over the network. Unfortunately that did not yield any results, so I input the IP address of the Windows VM manually. That worked well. This initiated a pairing process that I had to continue using Parsec, because you need to input a code on the host machine. After that, I had to quit Parsec and start streaming using Moonlight.

OK, not quite yet. Moonlight is able to detect the games installed on your system, but for a remote desktop experience, you have to do a small tweak for it to be equivalent to the experience you get from Parsec.

One thing that I noticed at the start was that the image quality on the desktop isn’t as good as on Parsec. However, it more than makes up for it in-game. Moonlight just works. There are plenty of knobs you can turn as well, including configuring the image quality by setting the resolution, framerate and network limits.

What makes Moonlight more suitable for me is that the Linux client packaged as a Flatpak is able to use the GPU for hardware-accelerated decode. The experience is great on a Linux machine and it just works, which is not something you can always expect to happen. Even an older XBOX ONE controller connected over Bluetooth works just fine over Moonlight!

The only issues that I had with Moonlight were later determined to be caused by networking issues. This shows up well on the integrated statistics that Moonlight provides using the Ctrl + Alt + Shift + S key combination under the Network jitter field. After stopping some services on my network, the issues went away.

The setup: one month later

After a month of use, I have made some adjustments to the setup. I used to keep the VM running so that I could just jump in and start playing whenever I liked it, but due to a tweak I made in the name of improving gaming performance, half the CPU cores on my system were not available for other purposes, such as suffering through IntelliJ indexing the project again. I decided to only power on the VM if I actually was about to take the time to play it. I haven’t yet figured out the best way to make this more convenient, though. I did make the “Calculator” key on my keyboard start/stop the VM on demand, but then GNOME started ignoring the shortcut I had setup and opened the calculator anyway, so that was a short-lived tweak.

A bigger change that I made to the setup was to store all my games on my ZFS pool that lives in another VM. For that, I setup an iSCSI target using this guide as a starting point . I used the tgt package instead. That has worked out surprisingly well, but the iSCSI target does not want to properly connect in the Windows VM on the first boot, which is mildly annoying. I soon discovered that spinning rust just does not cut it and got more SSD-based storage, otherwise other processes on the NAS could wreak havoc on the gaming experience in games like GTA V. Those issues were manifesting as long loading times and objects popping in too late, resulting in half the roads and building being missing while driving around the map. The iSCSI target solution also comes with the added bonus of having a proper backup of all my games. If Windows decides to blow itself up again and I don’t have a backup of the VM ready, then I could just reinstall it and not worry about downloading and installing all the games again.

Verdict

Overall, I’m very happy about the setup. I don’t use it all too often, but occasionally launching art of rally or BeamNG.drive on any random device I own is great. I do have some concerns about the future of this setup, especially with Windows 11 having more strict system requirements and newer games demanding more performance. For the time being, this works and I intend to keep it that way.

The one box that does it all: so far so good!

DIY cloud gaming setup with VFIO, Parsec and AMD

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Tue, 07 Sep 2021 06:00:00 +0300

This is a follow-up to my previous post where I covered the VFIO setup in general. For many people that would have been good enough, but my goal with this setup was to have a powerful gaming setup that I could access from my living room PC with 20 meters of Ethernet cables between the two.

Cloud gaming?

The concept of cloud gaming has become more popular lately, with Microsoft, Google, NVIDIA and others offering services that allow you to stream games to your device. These setups usually include a powerful server in a datacenter rendering the game and sending the compressed video stream to your device. This works surprisingly well, but due to high cost, poor internet connectivity in many places in the world and licensing issues , this setup might not work well for everyone.

However, if you already own a moderately powerful gaming PC and would like to use it to play games you already own over the network, then there are plenty of solutions that can get you there.

Here are some options:

Steam remote play: works for games launched via Steam.
Parsec: able to stream the whole desktop, including games.
Moonlight: same, but only usable on hosts with NVIDIA GPU-s due to the implementation relying on NVIDIA’s GameStream protocol.

In the past, I’ve had OK results with Steam remote play, but the limiting factor has been the reliance on Steam. I have bought a couple of games from GOG as well, which I could probably also play remotely by launching them via Steam, I just don’t want to go through the hassle.

For this setup I’ve opted to go with Parsec. It’s not perfect, but it’s still good enough for our purposes. The GPU we’re using (AMD Radeon RX 570) also limits our options because it won’t work with Moonlight.

The installation of Parsec is pretty straightforward: just install it on your host and client machines. In my case the client machine is a Lenovo ThinkPad X230 that was brought back from the dead . It’s not powerful, but at least it can do H.264 hardware decoding and it uses only 12W of power when idle, making it a perfect candidate for testing this out. It’s also worth mentioning that both machines are connected on the local network using an Ethernet connection to avoid Wi-Fi becoming a bottleneck.

VFIO, gaming and you

What turned out to be a bigger hurdle was the performance of games inside the VM. After getting everything running with the VFIO setup, I didn’t really spend any time trying to optimize the setup for the smoothest experience. When trying to actually run this setup, I ran into quite a few problems with performance which mainly manifested as stutters and unusually low framerates. Turns out that gaming has stricter latency requirements than other server workloads.

Thanks to the great guide in Arch Wiki, I could get a lot of ideas on what to try out to improve the performance of this gaming virtual machine.

Before doing anything, I ran some benchmarks to get a sense of what performance levels I’m dealing with. To test out each individual change, I used GTA V as the gaming benchmark since it was pretty good at pointing out any performance issues.

Here’s a list of things that I ended up doing:

setting the CPU model to host-passthrough in virt-manager
isolating CPU-s dynamically whenever the VM starts and pinning those CPU-s to the VM to avoid the host OS and other VM-s from using those cores
setting the CPU governor to performance on isolated cores to rule out issues with the CPU not clocking high enough and switching between idle/load power modes: echo performance > /sys/devices/system/cpu/cpu[4-7]/cpufreq/scaling_governor
enabling static hugepages to rule out issues stemming from poor memory access speeds
disabling SMT in UEFI settings to get rid of one additional variable
getting a 64 GB DDR4-3600 memory kit to allocate more RAM to the VM (16 GB) and leave enough for the host machine and other services

The golden rule of troubleshooting is to change one variable at a time and comparing results. Not every change that I made ended up being a positive one. For example, when configuring the VM to use 4 cores and 8 threads instead of plain 4 cores, I saw the framerates in GTA V drop 50%. I assume that the VM treated the “SMT cores” as real ones, causing the Windows scheduler to make incorrect decisions.

After playing around with this setup and accumulating these tweaks, I managed to get rid of most of the issues that bothered me during gaming, resulting in a much smoother experience. It finally felt like the gaming VM behaved like a machine with a 4 core CPU, 16 GB of RAM and an AMD RX 570 inside it.

Passmark PerformanceTest CPU results before any tweaks.

Passmark PerformanceTest CPU results after tweaks.

Passmark PerformanceTest memory results before any tweaks.

Passmark PerformanceTest memory results after tweaks.

At this point I’d consider the setup to be fantastic for someone who wants to play games with a VFIO setup. Since I was using this setup over the network with Parsec, I soon ran into more issues, but this time with the GPU.

GPU issues

Parsec and other similar solutions rely on the host GPU to encode the image into a video stream using the encoder present on the GPU die itself. You might see this being referred to as hardware encoding in settings. What I didn’t know before trying this setup is that AMD GPU encoders have a reputation for just being plain bad:

If the host has an AMD card, AMD is usually known for having worse encode than NVIDIA and even Intel. You should be fine at low resolutions. If all guests support H.265 and have it turned on, you may see better performance than with it off

And this shows. When trying to stream at 1080p, the result was a stuttery and inconsistent mess. Imagine trying to play at around 30fps with the frame timings graph resembling a heart rate monitor, that’s what it felt like. At 720p, the experience was so much smoother. Yes, the image quality suffers because of that, but at least it was mostly playable.

I decided to also try out if changing the codec from H.264 to H.265 has a significant impact. I did a quick test in Dirt Rally due to it having a benchmark loop mode and checked the statistics that Parsec shows. With H.264, I saw encode latency of about 10ms. With H.265, this latency was now at 8ms. Still not great, but it is technically a ~20% improvement. The downside of this for my setup is that the client laptop simply does not support hardware accelerated H.265 decode, which is something that Intel integrated graphics received support for in 7th gen CPU-s.

The GPU seems to also cause some trouble for Parsec, as it would fail to connect occasionally and reports a host encoder issue. This is usually overcome by rebooting the VM, which can get quite annoying after a couple of times. At other times, Parsec sometimes just froze during gameplay and caused the client to freeze at 100% CPU usage. Not what I’d call smooth sailing.

Another issue that I noticed by using MSI Afterburner is that the framerate was still not that stable in some games, such as GTA V. After all the fixes and tweaks, there were still small stutters, even with vsync enabled. I decided to look over AMD Radeon settings to see if a driver feature had an unintended side effect. I had picked the Gaming preset because that was what I was using this GPU for, but decided to switch to the Standard mode instead. And just like that, the stutters were gone! I suspect that the Radeon Anti-Lag feature might have been the cause to this, as that was one of the main settings that was disabled after switching to the Standard settings preset.

Is it worth it?

I like a technical challenge and going through all the attempts to get more and more performance out of this setup was interesting for me. However, it might not be the same way for everyone else. If you still feel like you want to go through this and learn something along the way, then feel free to use this as a guide on what you can try. Make sure to also read up on experiences that others have had with these tweaks and always measure your results to see if they had the intended effect.

For those who just want to play games and not worry too much about getting the expected performance out of your machine, I’d still recommend going with building a separate gaming PC.

If I had to build a machine specifically for this type of workload, then I’d make these changes:

Replace the CPU with a non-APU model, such as the Ryzen 9 5950X. Due to the physical core layout you could assign one CPU core complex to the gaming VM and leave everything else to the host. These CPU-s also have a lot of L3 cache, which should help in workloads that require low latency, such as games.
If you’re going for a streaming setup, then I’d try it first with an NVIDIA GPU. The error code 43 issues are gone now, making it a viable option with solutions like Moonlight.
More SATA or M.2 slots on the system. This makes passing through storage devices so much easier.

Future plans

I’m not planning on stopping this adventure just yet. I’ve recently looked around for an NVIDIA GPU to do a comparison against the AMD RX 570 and picked out an NVIDIA GTX 1060, which is from the same era and performance bracket. Should make for an interesting comparison in both Parsec vs Parsec and Parsec vs Moonlight scenarios.

I’m also hoping to eventually move to a more modern client PC that can support H.265 decoding and output at higher resolutions. Yes, 4K at 60Hz is demanding for a laptop from 2012. Given that the laptop is an older ThinkPad, I expect that to happen after 2025.

Regarding the storage setup, I’ve got input from a friend saying that I might want to try setting up a Samba share on my “NAS” VM and host my Steam library and other game files on that. After all, the virtual LAN has managed hit 2-3 Gbit/s in my testing and with L2ARC being persistent in ZFS 2.0, I might be able to take advantage of that as well ( assuming that I even need L2ARC, ARC efficiency is pretty good!). Currently I just have a Syncthing sync set up between the “NAS” VM and the gaming VM so that there exists at least a basic backup of all my games.

Testing GPU passthrough on AMD Ryzen 7 5700G APU

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Sat, 28 Aug 2021 10:00:00 +0300

Introduction

Before we jump into all the nitty-gritty details, I’d like to go over what we are dealing with here since these topics may be unfamiliar to you. VFIO is quite a niche topic and not everyone knows about it.

GPU passthrough: the process of allowing a VM (virtual machine) to use a dedicated GPU. This allows you to run GPU-heavy workloads inside the VM, such as gaming or anything that benefits from GPU compute power.

VFIO: the framework that allows us to perform this operation.

IOMMU: hardware feature that supports all of this.

virt-manager: GUI application that I use to manage VM-s.

Reasons why you may want to go through all the hassle and use this approach:

you’re on Linux but want to play games occasionally without rebooting into Windows
you want to do all your work, gaming and server workloads on one machine
you simply don’t want to build a dedicated PC for playing games and would like to utilize the existing resources on your main machine
you want more control over the Windows installation because you simply don’t trust Microsoft
you want to have more control over the Windows installation and the ability to revert the installation to an earlier point in time using a filesystem that supports snapshots (BTRFS, ZFS)
you have multiple separate VM-s for different purposes, but you would like to use one single GPU for all of them

One thing to keep in mind is that the hardware you use matters a lot here as that provides the foundation for getting this solution working. Here are some of the requirements you should know about:

The GPU you want to pass through has to support UEFI. GPU-s from the past 5 years should be good on that, but if you have a really old GPU that you want to use for testing, then that might not work out that well.
You need to have a good-enough CPU and plenty of RAM. After all, you’re essentially running a full PC within another PC.
Your motherboard has to support IOMMU and have IOMMU groups that allow you to isolate the GPU and only pass the GPU to the VM. Level1Techs does quite a few motherboard reviews and Wendell goes over the IOMMU groups and suitability for VFIO in them. It also helps if you look around the internet for enthusiasts that have already bought the same motherboard, they may have posted information about the groups. Here’s a handy script that allows you to check your IOMMU groups.
Some games may ban you if they detect that you are running the game inside a VM. It’s great that they’re trying to do something against cheaters, but unfortunately VFIO users get unfairly treated here. Please do some research beforehand if you play competitive multiplayer games a lot!

I hope that this introduction helped you understand the basics. Now, let’s just jump right into it.

Setup

We’re going to test this setup on hardware that I have covered previously. What makes this setup a bit special is that we’re using an AMD Ryzen 7 5700G APU on an mITX motherboard and a single dedicated GPU, allowing us to do big things in a small package. Yes, there are setups where you can do VFIO with a single GPU and pass it between the host OS and the VM, but that setup might be a bit tricky to use.

The OS is Fedora 34. To get this working, I’ve used various resources:

beginners guide on Level1Techs forums
Fedora-specific guide by Wendell from Level1Techs
this brilliant Arch Wiki guide that covers most of what you need to know about setting this up. It’s not only useful for Arch, a lot of what is written here applies for any distro.

Testing

I decided to do the initial testing with the Nvidia GT 710. It’s slow, buggy on Linux on Nouveau open source drivers, and I had it available. I also recently heard that Nvidia stopped being hostile towards their customers in one aspect by allowing GeForce GPU-s to be used in a Windows 10 VM without having to resort to workarounds.

The testing itself was relatively straightforward. The only issues I had were PEBCAK issues, possibly related to me performing this testing after work. The main issues I ran were either small typos in dracut configuration or the wrong device ID-s being added to kernel boot parameters. Once I discovered and fixed those, it was all smooth sailing.

I’m not going to rewrite a full guide here, please refer to the previously linked resources if you need more details than that. Those resources are also much more likely to receive updates about new finds and features.

The steps I took were roughly these:

Make sure that IOMMU was enabled in UEFI settings. It’s probably set to auto by default, make sure to set it to enabled.
Install the GPU that I want to pass through and connect it to a monitor.
Install virtualization packages on Fedora: sudo dnf install @virtualization
Enable IOMMU and preload the VFIO kernel module by adding amd_iommu=on rd.driver.pre=vfio-pci to Linux kernel parameters.
- I use GRUB2, these parameters live in /etc/sysconfig/grub.
- To apply these changes, you have to regenerate the GRUB configuration. Under Fedora, this is done using grub2-mkconfig -o /etc/grub2-efi.cfg
Get the device ID-s for the GPU and the related audio device that you’re planning on passing through
- To see the devices and ID-s, run lspci -nnk. The ID-s look something like 1002:aaf0.
Bind the GPU to vfio-pci to avoid the GPU driver from taking control of the GPU, otherwise you cannot pass it to the VM.
- I opted to go for the simple approach and added the device ID-s to kernel parameters: vfio-pci.ids=1002:67df,1002:aaf0
- Since I went with this approach, then I have to regenerate GRUB configuration again.
Make sure that the initramfs loads the necessary vfio drivers early in the boot.
- With Fedora 34, this means creating a file /etc/dracut.conf.d/10-vfio.conf with contents add_drivers+=" vfio_pci vfio vfio_iommu_type1 vfio_virqfd ".
- Make sure that you don’t have any typos in that.
- Regenerate the initramfs: dracut -f
Reboot!
Using virt-manager, select the VM you want to pass the GPU through and add two PCIe devices: the GPU and its associated audio device.
- If you don’t have the VM set up yet, then proceed with the normal installation without the GPU passed through yet. Make sure to create an UEFI VM, otherwise you might run into issues. This can be configured in virt-manager _ Overview_ section by selecting the Q35 chipset and setting the firmware to OVMF_CODE.df.
- If you have passed through the GPU, then make sure to remove the Display Spice device from the VM.
- If you want to also control the VM, you need to also pass through your USB devices, such as a Logitech wireless receiver.
Start the VM and cheer once you notice TianoCore appearing on the monitor that’s connected to the passed through GPU!

If you see this, then you've just successfully passed the GPU to the VM. Congratulations!

Results: Nvidia GT 710

Initial testing with Nvidia GT 710 was successful. By successful, I mean that the GPU displayed an image and did not install GPU drivers automatically.

The infamous Nvidia error code 43.

To overcome that last issue, I downloaded the latest official Nvidia GPU drivers and was good to go. The news were true, error code 43 was no longer an issue!

If your GPU shows up like this in Device Manager, then it should work properly.

Things got a bit less exciting when I got reminded that this GPU is weak. Very weak. Regardless, I decided to demonstrate its computing prowess by downloading the art of rally demo. Side note: it’s a great game, go try it out!

The demo ran! Not well, but it still ran!

Not pictured: the horrible framerate.

Satisfied with the results, I decided to go for gold and replace this GPU with the AMD RX 570 that I “borrowed” from this PC I built recently.

Results: AMD RX 570

After switching the GPU, I replaced the device ID-s with the new values and continued with the VFIO adventure. This time, the GPU drivers were automatically installed by Windows and everything just worked. That’s not supposed to happen, at least in my experience something always goes wrong. Always.

I installed the latest drivers from AMD-s website and continued with testing.

Furmark? Runs as expected.

VM on the left, VM performance stats on the right, mess of cables everywhere.

GTA IV? A stuttery mess, so yeah, runs as expected.

On the left: GTA IV, straight from the GPU. On the right: GTA IV, streamed via Parsec.

At this point the VM was assigned 4 CPU cores and I had performed no CPU pinning or optimizations of any kind, so the results were pretty good.

Storage

Because I built this setup on my current workstation/server machine, the storage situation is a bit tricky. The other VM that runs all the services has full access to the two 12TB hard drives and I wasn’t interested in setting up a networked storage setup. The only free spots I had were as follows:

120GB partition from the NVMe SSD. Good enough for storing the Windows 10 system files.
2x 250GB partitions on the Samsung SATA 1TB SSD-s. Just enough to hold my recently played games, but not much more. Running in striped configuration under Windows.

I later opted to expand the game storage partitions to 375GB, which meant that I had to get rid of the extra overprovisioning space that I had left aside. This setup is fine, but I’m losing out on some of the benefits that come with a virtualized Windows setup.

With regards to passing this storage to the VM, I had two options:

keep using the SATA virtual disk: works out of the box, but might not perform as well.
use virtio: you need to manually load and install the drivers for Windows to recognize these disks, but allegedly this has better performance than the SATA implementation.

I started off with SATA and did a comparison against virtio using CrystalDiskMark.

Left: SATA. Right: virtio.

virtio did come ahead in these comparisons, but I would probably have been fine with the SATA performance as well.

Conclusion

One of the main reasons I do these kinds of setups is the technical challenge. It sounds weird, but sometimes I am happier after having completed something technical and novel compared to actually using the setup.

Previously I’ve had mixed results with GPU passthrough, with ThinkPad T430 and eGPU somehow managing to run GTA V in a VM , while in another testing session I ran into issues which turned out to be related to the CPU being faulty (lots of PCIe errors).

I do have more practical future plans with this. The plan is to use this VM as a gaming VM that I can stream games from using Parsec to any device that I wish. One idea is to use the Nvidia Shield TV as a low-power box that is capable of performing streaming. Alternatively, I could also get a super tiny form factor Dell/Lenovo/HP PC that has plenty of power to drive a 4K display and uses a reasonable amount of power. More on that in a future post.

Regarding storage I foresee an upgrade coming soon, either to the drives themselves or the whole setup since ATX motherboards support more SATA ports and expansion cards. If you also pair that up with a case like Masterbox Q500L with a good power supply placement, then you’d still have a relatively small setup.

Turning leftover PC parts into a decent gaming PC

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Sat, 21 Aug 2021 18:00:00 +0300

Introduction

About a year ago, I bought an used PC based off of a Lenovo Erazer X510. It had a dual-core Intel Pentium G3220 CPU, 4GB of DDR3 RAM, a crappy Codegen 400W power supply and a 60GB SSD. I added an Nvidia GTX 1050 and just like that, a budget gaming PC was born. My brother ended up using it, and it was good for games that didn’t require a powerful CPU.

This is the box that started it all.

A year later, the dual core CPU was really showing its age. It could not even run GTA V, and the 4GB of RAM did not help the situation either. After checking the used PC parts market, I had essentially two choices:

get a quad core CPU (Haswell Intel i5) and upgrade the memory to 8GB
get a cheap PC and move over some parts, such as the GPU, SSD etc.

Performing the upgrade

I happened to find a PC that cost less than the CPU + RAM upgrade would have cost for the original setup. However, that PC was a complete horror show, and I only discovered that once I brought it home. Here’s a short list of things wrong with the PC:

The Nvidia GT 210 that was inside it was missing the L-bracket, and the heatsink had been cut, and another smaller heatsink was brutally screwed on top of the existing one. That’s definitely how cooling works, right?
The Windows 10 installation that it came with was probably a pirated Russian version of it. I couldn’t confirm it because the installation bluescreened on the first boot and everything was in Russian.
It had some random stickers on it, which was fine, but it took some force and alcohol to get it off properly.
One of the memory sticks inside it threw a lot of errors in memtest. Might explain that BSOD I got.
The HDD was barely held in place with a wood screw. Yes, a wood screw.
One of the PC fans was not connected. The end of the connector was just cut off and the exposed cables were wrapped over another power cable. I don’t know how that could have ever worked.

This GPU was held in place with a piece of wire.

Oh, it gets worse.

I don't know if I should be mad as this is clearly a fantastic demonstration of the creativity that some people have.

It goes to show that you really need to do your homework when buying a suspiciously cheap PC. In my case it was fine as I was going to replace a lot of the parts anyway and it ended up being a good gaming PC, capable of running GTA V and streaming CSGO on Twitch. A less experienced buyer would have been absolutely screwed though.

After moving the good parts over, I was left with a weak but functional PC. I didn’t really have a good use for it so I considered selling it. But then I thought about it and decided to try to build a good gaming PC out of it, since I didn’t have a competent machine at the time and I needed something to drive a big TV.

The plan

The case and the PSU this setup came with were not that great. I’d hate it if I built a good PC and the shoddy PSU fried it.

After some careful consideration, I decided to get the following:

Case: Fractal Design Meshify Mini C with dark tempered glass. I would have preferred a plain version of this case, but the tempered glass doesn’t look half bad either, just can’t drop that panel. Oh, and it looks good as well.
PSU: Seasonic Core 500W 80+ Gold. They make good stuff.
Storage: 2x 1TB Crucial BX500 SSD. Have to keep the games somewhere and an SSD-based RAID0 setup made the most sense to me.

The core of the system would be the motherboard, CPU and 8GB of RAM that I pieced together from leftover sticks. All I needed to do now was to simply upgrade the CPU and get myself a GPU, right?

CPU upgrade

The Lenovo Sharkbay platform supports Haswell CPU-s, which includes Intel Core 4000 series CPU-s. I checked used CPU listings and opted for an Intel i5 4460. 4 cores, 4 threads at a reasonable clock speed, that should play all the games I have with reasonable performance.

The CPU arrived, and with it came trouble. Because I started following the hardware world in more detail in 2016, I was not aware of one crucial detail: Haswell vs Haswell Refresh. Turns out that the CPU I bought was part of the Haswell Refresh series, meaning that you either needed a motherboard that supported this CPU, or a motherboard that offered a BIOS update which adds support for the upgraded CPU-s. This motherboard from the Lenovo prebuilt PC turned out to be neither of them. The fans spun up and kept spinning at max speed, but the PC did not POST.

Frustrated and disappointed, I decided to double down. I got myself another used CPU and was luckily able to sell the existing CPU for the same price I got it for. This time, the CPU was the Intel i5 4440, same specs, but a year older and a small drop in clock speeds. Plug it in, and it works like a charm.

Proprietary connections

Once the new parts arrived, I started building. I knew that the Lenovo machine had some non-standard front panel connectors that would not work well with more standard cases. I got myself a PCIe USB 3.0 internal header adaptor to enable the front USB ports and left the audio ports disconnected since I had no need for them.

PCIe to USB 3.0 internal header adapter, and dust.

Finding out which pins are for the power button was also a relatively simple process. I grabbed my flathead screwdriver and tried shorting pairs of pins in the front panel connector area. Once I found the one that turned the machine on, I connected the PC power button connector there. I didn’t care for the other connectors, so I left those disconnected as well.

These are the two pins that you need to short to turn this machine on.

Adding the GPU

To make this machine into a proper gaming PC, I needed a GPU. At first, I used an AMD RX 560 that I had, but since I borrowed it from my main PC and the Nvidia GT 710 was a horrible GPU for even the most basic tasks, I had to put that one back into the workstation.

The GPU market of mid-2021 is not that great. Any decent GPU from the last 5 years is completely overpriced, and even cards like AMD RX 480 - a 5 year old card at this point - sell for the original price from 2016. Somehow I managed to stumble upon an AMD RX 570 4GB model with a somewhat acceptable price, so I pulled the trigger on that one. That card is powerful enough to keep me busy until the prices for newer GPU-s become reasonable on the used market.

I plug the GPU in, and the TV displays nothing but a black screen. Tried the Nvidia GT 710 and the integrated graphics, all good. Tried the RX 570 as a secondary GPU, and the PC gets past the POST screen. Odd.

After some searching and digging, I saw some forum posts detailing how the CSM (compatibility support module) setting on the motherboard could fix the issue. I went into the BIOS while running off of integrated graphics, enabled CSM, plugged the RX 570 in, booted, and it actually worked now!

What I discovered after using the GPU for a while was that it got quite loud under load, even though the GPU itself wasn’t that dusty. While cleaning it, I noticed that the thermal paste seems to have never been changed, and it showed: the thermal paste was rock solid. I had to take a plastic prying tool to get rid of the thermal paste without breaking the GPU. A repaste and a reassembly later, the GPU was whisper-quiet even under a full FurMark test.

Cooling

The Lenovo prebuilt game with a surprisingly adequate CPU cooler. However, the fan was not having a good time and made more noise than I’d like. I decided to splurge a bit and got for a beQuiet Pure Rock Slim 2 as I had a good experience running that cooler on a PC I built years earlier.

When installing it, I noticed one critical flaw: the CPU cooler mounting holes were non-standard. They were simply too big. If I had the patience, I could have probably added some washers or other available solutions to fix it. Anyway, I ended up just replacing the fan on the original cooler with the one from the beQuiet cooler. I could probably use the CPU cooler in another project, so no big loss there.

I also noticed that the stock fan curve (or lack of it) means that the case fans run pretty loud by default. To fix this, I used Noctua low noise adapters that I happened to have from previous projects. It did fix the issue, but I still wish that I had more granular control over the fan speeds.

How well does it work?

This PC has mainly two uses: watching media and playing games. To keep it short: yes, it works well for these use cases. I value smooth performance more than a crystal clear image, which results in me running all of my games under 1080p and medium-high settings.

I don’t consider this particular setup to be in any way practical. The cost of all the parts is actually quite big once you add it all up. The setup does do everything that I need it for and that’s what makes it worth it for me. As an additional bonus, I also get the satisfaction of prolonging the life of the used parts (CPU, RAM, motherboard).

I’ll try to see how long I can keep using this setup. In case the CPU and/or RAM become the limiting factors, I’ll just replace that part of the system with something reasonable and carry the rest of the system over.

Gallery

Here’s what the setup looks like.

The tinted glass does a great job at hiding the green PCB.

Here's what it looks like on the inside.

The same, but now at a pretty angle!

AMD Ryzen 7 5700G: first impressions on Fedora 34

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Sat, 07 Aug 2021 07:00:00 +0300

Introduction

I’ve covered my self-hosting setup in a previous post. It has been a couple of months and the setup has mostly been fine. Sure, the power usage was pretty big under load, and yes, my internet did cut out every time I turned the VM on with all the services starting, but other than that, it has been rock-solid.

However, I soon realized that in most situations, the 12 CPU cores is a bit overkill. Performance was fantastic, but due to the case and the cooling solution that I’m using, the fans did have to ramp up to cool the CPU down. Due to the ongoing GPU drought driven by yet another cryptocurrency bubble, it also felt a bit wasteful to use an AMD RX 560 just to drive two 1080p monitors.

It just so happened that around this time, AMD released the Ryzen 5000 G-series APU-s to retail. Yes, you could buy the 4000 series “OEM-only” APU-s from certain online retailers as well, but with this release, it’s 100% official. The release date rolled around, and the reviews were good. A day later, the CPU was also listed at a local reseller and was actually in stock. After checking the street price for a used Ryzen 9 3900X and the Radeon RX 560, I decided to go for it, as it would be very close to a free “upgrade”.

CPU retail packaging. Holds the APU and a small CPU cooler as well.

Prep work

Before performing the purchase, I confirmed that the motherboard that I use (Gigabyte B450 I AORUS PRO WIFI) supports this CPU after a BIOS upgrade. B450 chipset lives in a bit of a gray area where the CPU support is determined by the motherboard manufacturer. No updates? No CPU support for you.

From my previous BIOS updating experiences, I remember that this particular motherboard had some buggy BIOS implementations. I ran into two issues:

PC would get stuck on POST if you reboot. It’s annoying when making changes to UEFI settings, but with my use case this is not that big of a deal.
If you remove a boot option, then it is impossible to get it back. Seriously, even a CMOS reset did not help with this one.

This time around, I decided to update the versions one by one to follow the happy path and not run into other types of issues. The bug did resurface right after the first version that I tried, so that was a bit disappointing.

Now would be the perfect time for a surprise power outage or a cat intervention.

Once I was all up-to-date, I carefully removed the old CPU and plugged in the new one.

There's something very satisfying and beautiful about computer hardware. Just look at it!

Hold on, why would you intentionally downgrade your CPU?

It all came down to “use the right tool for the job”. The Ryzen 9 3900X is a good CPU, no doubts about that, but during my work, I kept an eye on my CPU usage quite often and found that even in the worst-case scenarios, such as IntelliJ IDEA performing indexing, I wasn’t using the CPU to the full extent. It’s great for those scenarios, but I wasn’t hitting them as often as I would have thought.

It also did not help that the CPU did not have an integrated GPU. When all you have to run is a basic desktop setup with 60Hz 1080p monitors, having a dedicated GPU to run them sounds so wasteful. This increased the power use of the whole setup and the GPU could be utilized better elsewhere.

The Ryzen 7 5700G is a downgrade in multi-core performance, mainly due to 4 fewer cores and only 25% of the L3 cache, but according to various reviews and benchmarks, you could actually expect a small single-core performance uplift. Nothing groundbreaking, but good enough for my use case.

Power usage

Because I don’t change my CPU-s too often, I thought I’d take the time to run some benchmarks and measure the power usage of the whole setup. This includes everything: the PC (with two spinning hard drives), standing desk, two 1080p monitors and peripherals.

Please note that these benchmarks are not very scientific and are simply my observations of the change you can expect when performing a switch like this on your own setup. I did my best to make sure that the running conditions are as close as possible by stopping the service VM, waiting until the background CPU usage was normal, running the memory at the same spec etc.

AMD Ryzen 9 3900X + Radeon RX 560

Idle, sitting on desktop: ~139W
Full synthetic CPU load using stress: ~253W

AMD Ryzen 7 5700G + integrated Vega 8 GPU

Idle, sitting on desktop: ~99W (generally between 85-100W)
Full synthetic CPU load using stress: ~166W

As expected, the change of the CPU and removal of the dedicated GPU results in a big drop when it comes to the power usage of the whole workstation setup. To go even lower, I’d have to switch out the core of the PC to something like the ASRock Deskmini X300 and replace the hard drives with two big SSD-s. I’d go for that as soon as bulk SSD storage became a viable option. Unfortunately, this is not the case, yet.

Performance

Reputable review sites, such as AnandTech, have already done a lot of benchmarks with better testing methodologies, so I instead went ahead and tested out something that I do quite often at work: running tests in poorly written Java projects!

For this, I have two Java projects that have their own test suites. Project A has 900 or so tests, which are mainly parallel in nature. Project B has 2000+ tests, where a good number of them are dependent on a single thread to do all the work.

To run the tests, I just call the “Run all tests” function in IntelliJ and let it do the work of running the tests and measuring the elapsed time. And as mentioned before: this is not scientific, there are probably some errors in here, but the actions performed here are in my opinion a pretty good reflection of day-to-day work, and that’s what matters the most to me. Tests were run twice with a short break between the test runs to let the CPU cool down a bit.

Project A: AMD Ryzen 9 3900X

Test run 1: 10.396s.

Test run 2: 10.494s.

Project A: AMD Ryzen 7 5700G

Test run 1: 15.561s.

Test run 2: 15.690s.

CPU usage on the Ryzen 9 3900X during the test run.

Project B: AMD Ryzen 9 3900X

Test run 1: 364s.

Test run 2: 365s.

Project B: AMD Ryzen 7 5700G

Test run 1: 471s.

Test run 2: 480s.

The performance hit was somewhat expected for project A due to the parallel nature of the tests. However, for project B I expected better results than that due to the alleged improvements in single-thread CPU performance.

The Linux experience

I expected the APU to work as expected, with perhaps some small issues popping up here and there. I’m currently running Fedora 34 with all the latest updates installed and the kernel version at 5.13.6, which should be the best option when trying to get new hardware working on Linux.

The CPU portion of the APU is working relatively well for a CPU that was released just a couple of days ago. I have noticed that for some reason this CPU is constantly clocked at around 4.2GHz on all cores at all times, even if there isn’t too much going on in the system. sensors, the program I use to monitor system temperatures, is also having a small hiccup, as I cannot see the CPU temperatures at all, only the integrated GPU temperatures show up even after running sensors-detect.

nvme-pci-0900
Adapter: PCI adapter
Composite:    +47.9°C  (low  = -40.1°C, high = +89.8°C)
                       (crit = +93.8°C)
Sensor 2:     +47.9°C  (low  = -40.1°C, high = +89.8°C)

iwlwifi_1-virtual-0
Adapter: Virtual device
temp1:        +48.0°C  

amdgpu-pci-0a00
Adapter: PCI adapter
vddgfx:        1.36 V  
vddnb:       999.00 mV 
edge:         +51.0°C  
power1:        6.00 mW

The GPU seems to be a bit rough around the corners. I’m using the GNOME desktop environment, which means that I’ve learned to endure laggy user interfaces. I default to using Wayland, but I sometimes also use Xorg for those situations where the application I’m trying to run does not work properly under Wayland (looking at you, Slack and your screen sharing functionality…).

The integrated Vega 8 graphics are almost as fast as the dedicated RX 560 GPU under normal light desktop workloads. Once you run something very heavy, like Furmark, the GPU performance tanks and the desktop becomes almost unusable. This is not the case when running some WebGL workloads though, so it could be an isolated case.

The most serious issue I’ve managed to find is that Firefox has some issues with this GPU once you put enough load on the GPU. You may start to notice flickering and UI elements blinking when moving around. A good test case for this is to run Netdata in live mode and have a 1080p Youtube video in another window. I haven’t seen anything like that since running the Nvidia GT 710 with Nouveau drivers under Fedora 34. At least it does not crash the browser or the Wayland session, so it’s not a complete disaster. Still, pretty annoying, so I hope this gets resolved eventually. If I had to guess: the issue is probably related to the Webrender implementation that makes use of the GPU to speed up the browsing experience.

Closing thoughts

Overall, I’m happy that the APU actually works. I’ve had a couple of bad experiences in the past where the Ryzen 3 2200G had stability issues and had to be replaced, so this experience is already a step up from that. The PC runs cooler and quieter, the desktop experience is good enough for me and the CPU performance is acceptable.

I’m sure that the issues that I’ve had with this APU under Linux will be fixed eventually. Each kernel update usually brings lots of improvements to new hardware and I’m sure that this APU is no exception.

As for the BIOS issues, I’m happy that AMD and Gigabyte opted to support the B450 chipset well into 2021, but I’m sure that this choice has come at the cost of QA. If I had to choose between buggy support and having to buy a new motherboard for my new, shiny CPU, then I’d still pick the first one.

Update (2021-08-18)

I’ve had this CPU for a while now, and there have been a couple of kernel updates in the meantime. Here are my notes.

The idle power usage has recently been in between 65-70W for the whole setup. I’ve measured the desktop itself to take up 38W of that.

Although my motherboard does not seem to expose the cTDP option or have ECO mode available, I still managed to limit my TDP to 45W using this trick that /u/The-Stilt mentioned over at Reddit:

The potential lack of the ECO-mode does not prevent you from using it, 
since it is nothing but a preset of PPT/TDC/EDC settings.

Eco-Mode (95W): PPT = 128W, TDC = 80A, EDC = 125A

Eco-Mode (65W): PPT = 88W, TDC = 60A, EDC = 90A

Eco-Mode (45W): PPT = 61W, TDC = 45A, EDC = 65A

This had a great effect on the full load of the system: 3.8GHz all-core while the whole setup consumed 113W.

Precision Boost Overdrive settings that should result in a nice efficiency win. Target = 45W.

Update (2021-08-20)

Quick update on the GPU situation: Firefox seems to be fine now, I could not get the flickering to occur again even under worst case scenarios.

Notes and stuff

Here are the outputs for a couple of commands that I could think of.

IOMMU groups

Script taken from Arch wiki.

shopt -s nullglob
for g in `find /sys/kernel/iommu_groups/* -maxdepth 0 -type d | sort -V`; do
    echo "IOMMU Group ${g##*/}:"
    for d in $g/devices/*; do
        echo -e "\t$(lspci -nns ${d##*/})"
    done;
done;

IOMMU Group 0:
        00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge [1022:1632]
IOMMU Group 1:
        00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge [1022:1632]
IOMMU Group 2:
        00:02.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne PCIe GPP Bridge [1022:1634]
IOMMU Group 3:
        00:02.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne PCIe GPP Bridge [1022:1634]
IOMMU Group 4:
        00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge [1022:1632]
        00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus [1022:1635]
        00:08.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus [1022:1635]
        0a:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Cezanne [1002:1638] (rev c8)
        0a:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Renoir Radeon High Definition Audio Controller [1002:1637]
        0a:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) Platform Security Processor [1022:15df]
        0a:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne USB 3.1 [1022:1639]
        0a:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne USB 3.1 [1022:1639]
        0a:00.6 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) HD Audio Controller [1022:15e3]
        0b:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 81)
        0b:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 81)
IOMMU Group 5:
        00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 51)
        00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 6:
        00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:166a]
        00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:166b]
        00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:166c]
        00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:166d]
        00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:166e]
        00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:166f]
        00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1670]
        00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1671]
IOMMU Group 7:
        01:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset USB 3.1 XHCI Controller [1022:43d5] (rev 01)
        01:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset SATA Controller [1022:43c8] (rev 01)
        01:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Bridge [1022:43c6] (rev 01)
        02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
        02:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
        02:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
        02:05.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
        02:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
        02:07.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
        06:00.0 Network controller [0280]: Intel Corporation Wireless-AC 9260 [8086:2526] (rev 29)
        08:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
IOMMU Group 8:
        09:00.0 Non-Volatile memory controller [0108]: Kingston Technology Company, Inc. U-SNS8154P3 NVMe SSD [2646:5008] (rev 01)

# lscpu
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   48 bits physical, 48 bits virtual
CPU(s):                          16
On-line CPU(s) list:             0-15
Thread(s) per core:              2
Core(s) per socket:              8
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       AuthenticAMD
CPU family:                      25
Model:                           80
Model name:                      AMD Ryzen 7 5700G with Radeon Graphics
Stepping:                        0
Frequency boost:                 enabled
CPU MHz:                         3800.000
CPU max MHz:                     4672,0698
CPU min MHz:                     1400,0000
BogoMIPS:                        7585.94
Virtualization:                  AMD-V
L1d cache:                       256 KiB
L1i cache:                       256 KiB
L2 cache:                        4 MiB
L3 cache:                        16 MiB
NUMA node0 CPU(s):               0-15
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Full AMD retpoline, IBPB conditional, IBRS_FW, STIBP always-on, RSB filling
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid ap
                                 erfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topo
                                 ext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a rdseed adx smap clflushopt clwb sha_ni 
                                 xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists paus
                                 efilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip pku ospke vaes vpclmulqdq rdpid overflow_recov succor smca fsrm

# lspci -nnk
00:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne Root Complex [1022:1630]
	Subsystem: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne Root Complex [1022:1630]
00:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne IOMMU [1022:1631]
	Subsystem: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne IOMMU [1022:1631]
00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge [1022:1632]
00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge [1022:1632]
00:02.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne PCIe GPP Bridge [1022:1634]
	Kernel driver in use: pcieport
00:02.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne PCIe GPP Bridge [1022:1634]
	Kernel driver in use: pcieport
00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge [1022:1632]
00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus [1022:1635]
	Kernel driver in use: pcieport
00:08.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus [1022:1635]
	Kernel driver in use: pcieport
00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 51)
	Subsystem: Gigabyte Technology Co., Ltd Device [1458:5001]
	Kernel modules: i2c_piix4, sp5100_tco
00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
	Subsystem: Gigabyte Technology Co., Ltd Device [1458:5001]
00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:166a]
00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:166b]
00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:166c]
00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:166d]
00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:166e]
00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:166f]
00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1670]
00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1671]
01:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset USB 3.1 XHCI Controller [1022:43d5] (rev 01)
	Subsystem: ASMedia Technology Inc. Device [1b21:1142]
	Kernel driver in use: xhci_hcd
01:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset SATA Controller [1022:43c8] (rev 01)
	Subsystem: ASMedia Technology Inc. Device [1b21:1062]
	Kernel driver in use: ahci
01:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Bridge [1022:43c6] (rev 01)
	Kernel driver in use: pcieport
02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
	Kernel driver in use: pcieport
02:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
	Kernel driver in use: pcieport
02:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
	Kernel driver in use: pcieport
02:05.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
	Kernel driver in use: pcieport
02:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
	Kernel driver in use: pcieport
02:07.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
	Kernel driver in use: pcieport
06:00.0 Network controller [0280]: Intel Corporation Wireless-AC 9260 [8086:2526] (rev 29)
	Subsystem: Intel Corporation Device [8086:0014]
	Kernel driver in use: iwlwifi
	Kernel modules: iwlwifi
08:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
	Subsystem: Gigabyte Technology Co., Ltd Device [1458:e000]
	Kernel driver in use: igb
	Kernel modules: igb
09:00.0 Non-Volatile memory controller [0108]: Kingston Technology Company, Inc. U-SNS8154P3 NVMe SSD [2646:5008] (rev 01)
	Subsystem: Kingston Technology Company, Inc. U-SNS8154P3 NVMe SSD [2646:5008]
	Kernel driver in use: nvme
	Kernel modules: nvme
0a:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Cezanne [1002:1638] (rev c8)
	Subsystem: Gigabyte Technology Co., Ltd Device [1458:d000]
	Kernel driver in use: amdgpu
	Kernel modules: amdgpu
0a:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Renoir Radeon High Definition Audio Controller [1002:1637]
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Renoir Radeon High Definition Audio Controller [1002:1637]
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel
0a:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) Platform Security Processor [1022:15df]
	Subsystem: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) Platform Security Processor [1022:15df]
	Kernel driver in use: ccp
	Kernel modules: ccp
0a:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne USB 3.1 [1022:1639]
	Subsystem: Gigabyte Technology Co., Ltd Device [1458:5007]
	Kernel driver in use: xhci_hcd
0a:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne USB 3.1 [1022:1639]
	Subsystem: Gigabyte Technology Co., Ltd Device [1458:5007]
	Kernel driver in use: xhci_hcd
0a:00.6 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) HD Audio Controller [1022:15e3]
	Subsystem: Gigabyte Technology Co., Ltd Device [1458:a0c3]
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel
0b:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 81)
	Subsystem: Gigabyte Technology Co., Ltd Device [1458:b002]
	Kernel driver in use: ahci
0b:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 81)
	Subsystem: Gigabyte Technology Co., Ltd Device [1458:b002]
	Kernel driver in use: ahci

How to mess up a simple ThinkPad X230 BIOS flash and how to recover from it

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Mon, 28 Jun 2021 07:00:00 +0300

Background

I have a Lenovo ThinkPad X230. It’s a small and good laptop that I’ve used as my main laptop and a server as well.

A couple of years ago I messed around with flashing alternative BIOS implementations on the X230, such as the skulls project, which made installing prebuilt coreboot images very simple. Of course, before doing any testing, I made backups of the stock BIOS implementation (ooh, foreshadowing!).

Skulls was nice, it booted fast, but changing the boot order likely required modifications and rebuilding the coreboot image (I wanted to set the mSATA SSD as the first bootable device). It was also not UEFI-compatible, which could have probably been solved with something like TianoCore, but I didn’t feel like messing with that too much. Eventually I flashed the original UEFI implementation back and went on with my life.

Then I stumbled upon the 1vyrain project project, which allows you to “jailbreak” your BIOS. This modification also enables a lot more bells and whistles in the BIOS under the Advanced menu option, plus it also gets rid of the WiFi adaptor whitelist that Lenovo has put in place, allowing you to now use any adaptor that’s compatible with the form factor.

The mod was very easy to install and had no issues, until I got myself a DDR3 SODIMM RAM kit. One of the sticks was not working properly, showing up with graphical glitches and failing a memtest run. To troubleshoot it, I decided to change the speed at which the memory runs from DDR3-1600 to DDR3-1333. It booted, and still had the issues.

Then I changed it to DDR3-1066.

Black screen. Uh-oh.

Initial troubleshooting

Thinking that this is just a simple issue that I could get fixed by clearing the BIOS settings and starting fresh, I looked up some basic instructions on doing so.

Things I tried:

remove the battery (even if it is completely dead, like mine), and hold the power button for 60 seconds, then attach the battery and turn it on again.
remove both the battery and disconnect the internal CMOS battery that sits under the palmrest and hold the power button for 60 seconds again.
disconnect all the batteries and leave the laptop sitting on a shelf, then try again.
try “the secret ThinkPad power button combo”:

… unplug the AC adapter and take out the battery. Then, you push the power button 10 times in a row at one second intervals. Next, you push and hold the power button for 30 seconds. Then you put the battery back in and push the power button… and she lives. The computer came back, good as ever.

After all of that, the laptop was still as good as dead. You could see the light turn on, the fan starts spinning, but nothing shows up on the screen. At that point I knew that the BIOS reflash was probably my only choice.

Flashing the BIOS

To fix the issue, I had to flash the BIOS externally. Luckily I still had the original BIOS backup images available.

With the help of a good friend, we set everything up and did some tests to ensure that the connection is good and that read/write operations on the chips return consistent results.

The chips that we're going to flash sit behind the left side of the palmrest, making them quite easy to access.

Left one is generally referred to as the bottom chip and the right one is the top one.

Once we had specified the correct chip type by looking at what was written on the chip package itself, we flashed both the top and bottom chips with the original images that were once made as a backup. After reading the chips back and comparing checksums using md5sum, it all looked good.

We disconnected the SOIC clip, attached the power cable and tried turning the laptop on. Power button light comes up for 1 second and then turns off. Not a good sign.

We flashed both chips again and made sure that everything is in order. No dice, the symptoms are the same.

A closer look

It didn’t take too long for me to start thinking about what I could use all these extra laptop parts for. After all, the screen was good, it had 16GB DDR3 RAM, two SSD-s, a screen assembly in good condition etc. I was stopped in my tracks when my friend noticed something.

Find the missing resistor.

It turns out that during this flashing process, we (probably me) had managed to get rid of a resistor that sat next to the top flash chip. Whoops. Turns out that it was an important one, because over at Reddit, someone else also ran into this issue and had similar symptoms .

To resolve this, my friend had the idea of putting a 10K resistor in there to see if it fixes things. The issue was that the only one available right from the get-go was a very big one, so he had to be clever with it. He ended up positioning it upright with a piece of wire handling the other side.

Isn't it beautiful?

This, however, didn’t improve the situation much.

Schematics to the rescue

To understand the role of the missing resistor better, we started looking for schematics online. I managed to find an OK looking one from a random forum and started browsing through it. The keyword of the day was “SPI” and when searching for it, we found some promising results. My friend found the resistor and found that we could just short it, since the resistor used to be between two wires. And that he did.

The missing resistor, as shown on the schematic.

The big short.

We attempted to power on the machine again and this time the error sequence was different as the laptop went into some sort of a party mode with lights flashing. The usual troubleshooting steps didn’t work that well here, either, so it’s possible that the stock BIOS image did not like something here.

What we did have was the option of flashing the known-good skulls image on. I handled that part quite OK, since all the commands were in my bash history anyway. The excitement, however, got the best of me and I managed to repeat the original mistake and I bumped the soldered piece of wire off of the board, putting us back to the original issue.

After another session of carefully connecting the two wires, we repeated the steps and after the flash, we finally saw the Skulls splash screen.

It works!

Picture taken after the fact, because I forgot to do it initially.

After booting into a Fedora 34 liveusb environment with iomem=relaxed, I grabbed the latest release of skulls ( released on 4/20 🔥) and used the internal flashing method to update.

Reboot, and we saw party mode again. Having used skulls in the past, I knew that this was likely down to skulls (or coreboot) being very picky about RAM. The solution here is to reseat the RAM modules and turn it on again. Why does that work? Not sure, but it did.

To make this solution more resistant to my unintentional attempts of breaking things, I added a glob of hot glue next to the top chip to make sure that I don’t knock anything off of the board again.

Testing

Once the laptop was assembled again, I did some testing to see if everything still works as intended.

Fedora 34 liveusb boot resulted in a black screen once, but on the second try, it worked again.

Windows 10 installed fine on the machine and while some functions don’t work as well (TrackPoint and touchpad being the prime examples here), it was still usable. Unfortunately I did not manage to write down the original Windows 10 key that is embedded to the stock BIOS image, so I could not activate it. But hey, at least it works.

Windows 10 also works.

Lessons learned

Even though this adventure started with a self-inflicted wound, I still consider it a success, as I learned a couple of things.

It’s very easy to get confused when reviewing how the wires should be set up on the SOIC test clip and the device that’s actually performing the flashing.
If a piece of software exposes a bunch of knobs and levers and warns you about catastrophic failures that might occur as a result, then trust it and don’t mess with things that you have no clue about. With this example case, though, I didn’t expect a memory speed change to result in such a catastrophic failure in the first place…
Developers that know hardware are worth their weight in gold.
Access to schematics is very valuable when fixing hardware.

And on the final note, I would like to give a shout-out to Arti Zirk, the guy who helped fix the mess I created. If you need someone who is well-versed in anything related to software development, Linux and embedded systems, then he is your guy.

Genius at work.

flashrom references

This section has some commands that might be useful to you. Unless you’re using the exact same flasher, you will probably need to specify a different programmer with the -p flag.

The names and extensions for the input-output files are arbitrary. A file is a file.

Reading/writing the top chip.

Note that I had to specify the chip type, otherwise flashrom would complain.

The top chip is 4MB and the bottom chip is 8MB. This should also be reflected in any output that you get.

--verify option is also useful for checking if the results are OK. I also recommend reading the image from the chip after a write and comparing its checksum using md5sum, just to be extra safe.

# Read the existing image on the chip
flashrom -p ft2232_spi:type=2232H,port=B,divisor=4 -r top.bin -c MX25L3206E/MX25L3208E 

# Write the specified image to the chip
flashrom -p ft2232_spi:type=2232H,port=B,divisor=4 -w top_backup.bin -c MX25L3206E/MX25L3208E

Reading/writing the bottom chip

# Read
flashrom -p ft2232_spi:type=2232H,port=B,divisor=4 -r bottom.bin -c MX25L6406E/MX25L6408E

# Write
flashrom -p ft2232_spi:type=2232H,port=B,divisor=4 -w bottom_backup.bin -c MX25L6406E/MX25L6408E

I've reached the self-hosting endgame

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Sun, 23 May 2021 07:00:00 +0300

Setup

After I had a perfectly functional, quiet and performant server up and running, I fell victim to my imagination and completely changed my self-hosting setup. Again.

Here it is, in all its glory:

Find the server.

Well, at least that’s what’s on the table.

There it is.

This machine is a true all-in-one: it hosts all my services in a VM while also acting as my main workstation.

Specs

CPU: AMD Ryzen 9 3900X. 12 cores and 24 threads in a consumer platform is absolutely bonkers.

RAM: 32GB DDR4 @ 3200MHz

Motherboard: Gigabyte B450 I Aorus PRO WIFI. It’s OK, but oh man are the newer BIOS releases buggy. At least it gives 0 hoots about flashing an older BIOS version.

GPU: AMD RX 560. It gets the job done.

HDD: 2x 12TB WD white-label (WD120EDAZ) drives, shucked from WD My Book external drives.

SSD: 240GB Kingston A1000 NVMe for OS, 2x1TB Samsung 870 EVO SATA SSD for fast solid state storage

PSU: Seasonic Focus something-something. Doesn’t really matter, it’s reliable and has plenty of power.

Cooling: Noctua NH-D9L on the CPU, 200mm fan at the front and 2x80mm beQuiet fans at the back to keep the system cool.

Case: ThermalTake Core V1. Not the smallest case, but super simple to work in!

Why?

Before setting this up, I had three machines running:

a laptop that was always on so that there won’t be any conflicts with Syncthing.
a server with server workloads
a desktop that I mainly used for work

At any point in time, I had at least two machines running and the desktop was fired up for a good part of the day due to work. Since trying to reduce my electricity usage was one of my goals with going low-power on my server, this sounded a bit wasteful. At this point I got the brilliant idea of trying to combine this into one machine that does it all, but before that, I had some concerns. The hardware was simple: just shuck the 12TB drives in there and get some big SSD-s and partition them, but the software part required some thinking and testing.

The software

To keep the work to a minimum, I found that the easiest way to move my setup was to just put the existing server installation into a VM. I’ve had plenty of experience with VM-s so this was not a difficult task, just install libvirt and all the dependencies will be settled automatically.

The problem is the networking. If I want the VM to show up as a separate machine on the network, I need to add a virtual network interface as a macvtap device, which means that the desktop and the VM can both connect to the internet and to other devices on the local network, but the desktop cannot reach the VM directly. There are other solutions, such as the default NAT-based solution, but with that, forwarding ports is quite tricky to someone not familiar with networking.

To overcome this limitation, I came up with a pretty simple solution: add another network interface to the VM! This time, the interface was set up as the NAT-based solution. To make this work, I had to make sure that the Debian 10 install was configured to use that interface as well, which meant a little copy-pasting in /etc/network/interfaces. To make the host connect to this machine using the hostname, I also added an entry to /etc/hosts on the host machine to route all requests to the hostname via this second virtual network interface.

With this, the main obstacle was resolved and the host machine had no issues connecting to the VM.

The storage

Since this machine was also used as a workstation, I had to be a bit clever with my storage setup. The bulk storage in the form of two 12TB hard drives was 100% managed by the VM.

The 1TB SSD-s were partitioned as such:

2x 500GB partitions for my workstation running in BTRFS mirror, since I don’t trust ZFS updates on Fedora.
2x 250GB partitions for fast SSD-based ZFS mirror in the VM
2x 250GB free space for overprovisioning.

In the event that something does go wrong, it should not have too much of a negative impact either on the host or the VM. If the drive quality does become a concern, there is always the option to upgrade to PRO series drives that should have a much better endurance rating.

Pros and cons

With that, the setup was up and running. The VM initially had assigned 6 CPU cores and 8GB of RAM, but lately I’ve decided to assign 12 cores and 16GB of RAM to make the most out of the available resources.

So, how well does it work?

Quite well, in fact. There were some hiccups, though:

the latest BIOS update for the motherboard was very buggy, which resulted in the ability to permanently get rid of boot devices in the boot device selection menu and the annoying issue of getting stuck in POST when rebooting the machine
the VM would sometimes have issues with networking, which I attempted to fix by treating the NIC-s within the VM as non-hotpluggable devices. This seems to have done the trick, as I have not had any issues since.

With this setup, there are some tradeoffs that you should be aware of.

Pros:

Everything runs in one machine with reasonable power usage : ~70 W when idle and monitors off, ~120 W with monitors on and a light workload
Optimal utilization of resources. Instead of having to upgrade my workstation or server separately, I can just share the resources and modify limits any time I want. Should I upgrade to 64GB of RAM in the future, I can make some more available on the server by changing only one configuration parameter in the VM. The same goes for storage. If I were to upgrade to something bigger and faster, then it will benefit both the workstation and the server workloads.
Easier troubleshooting. In case things go wrong with the server, I don’t need to grab an external monitor and hook it up to the server, the console is accessible via virt-manager on the host.

Cons:

All eggs in one basket. If your host OS breaks something, then this will affect your server workloads as well.
Noise. While not so bad with noise-cancelling headphones, it can still be annoying when you have hard drives running at your desk.
Your server workload is dependent on your host OS booting up first, which will require manual input from you if your host installation is encrypted.

Conclusion

This setup worked out better than I expected, with no further issues occurring over a couple of weeks. The availability issues during restarts are not a concern for me, as this is a hobby to me, and the flexibility and ridiculous amounts of power this setup has outweighs all the cons.

Let’s see how long this setup lasts.

2021-06-28 update

It still works like a charm!

Tech tip: eliminate HDD humming noise

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Fri, 02 Apr 2021 07:00:00 +0300

Anyone that has bought themselves external WD drives from the Elements/My Book/Easystore series are probably familiar with the acoustic characteristics of the drives. The drives have a loud hum caused by WD running the drives at 7200rpm while claiming the drives to be “5400rpm-class” and the clacking of the read-write heads is audible as well. In a small space, such as an apartment, the hum is maddening, especially when you have more than one drive running at the same time.

After running such a setup for months, enough was enough. I bought some sound dampening foam and used that to try to limit the noise that my setup was making. However, that didn’t do much and the before/after noise comparisons didn’t have much of a difference. As a result of this testing, I did have various pieces of acoustic foam left, and after I noticed that my Lenovo M73 Tiny PC had a similar shape to the WD My Book 12 TB hard drives that I got recently, I had an idea.

Minimum viable server, now with 90% less noise!

I used four pieces of foam and placed them so that I could stack the hard drives on top. This resulted in the humming noise being completely eliminated. The sound of hard drive read-write heads is still there, but it is so much less audible now. After this change, I have started hearing the cooling fan more than the hard drives themselves.

This setup could probably be improved by switching around the setup so that the hard drives were at the bottom with the bottom drive having additional foam or rubber feet below it to support it better. For the time being, I’m perfectly happy with this arrangement, as long as it doesn’t fall over.

2021-04-23 update

I have now added a fan to the setup!

Pretty cool, eh?

The fan itself is powered by one of the USB ports with the help of a spare sacrificial USB cable and a Noctua omni-join kit that I had left over.

The drive temperatures are now reading around 42C and 46C.

Whacky setups: seedbox on a wall

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Sat, 20 Mar 2021 12:00:00 +0200

The Orange Pi Zero is one hell of an SBC. It has served as a Wi-Fi access point for months without issues and as a testbed for playing around with MySQL.

I wanted it to be a useful device again, but had trouble finding an use case for it due to its hardware limitations. After messing around with my amateur archival work, I had accumulated some data that might be worth keeping around. Due to my network being quite unreliable and slow, I decided to go with torrents as the main distribution method. I could have just hosted the torrents on my main server box, which is what I later did anyway, but then I realized that hosting torrents is a workload that a weak machine could handle just as well.

I took a board that was left over from a shelf, grabbed some twine and nails and banged together this monstrosity:

The setup in all its glory.

This setup originally had two 4TB hard drives. Unfortunately one of them was too faulty for even BTRFS to handle.

This brave little machine runs on Armbian and hosts one Transmission instance. This instance is a mirror for some of the data that I collect as part of my archival effort. The OS lives on a 120GB Crucial SSD with the help of the nand-sata-install script that Armbian provides, and the data lives on one 4TB Seagate hard drive that is somehow still working.

The board is pressed against the metal part of an AMD AM4 socket stock cooler with the help of a thick 5mm thermal pad and solid copper wires taken from an electrical cable. This looks very janky, but the cooling performance is absolutely fantastic: 30C when idling and not more than 50-60C under the heaviest workloads.

The total power consumption of this setup is typically between the 5-10W range, which isn’t the lowest, but still acceptable. The CPU is a 32-bit one, so let’s see if this board can make it to 2038.

All in all, it was just another server on the wall.

2021-04-02 update

The 4TB Seagate HDD decided to start a career in experimental music production and started clicking and not showing up when connected to a PC. And thus, the seedbox on a wall has been put on pause for now.

How to start your self-hosting adventure: a high-level overview

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Wed, 17 Mar 2021 15:00:00 +0200

Reddit is a great starting point for getting new ideas for your homelab: racks full of machines in /r/homelab, storage measured in terabytes (or even petabytes) over at /r/datahoarder, all the different services that people host over at /r/selfhosted. This can be a bit overwhelming for someone just starting out in this area, which is why I decided to write a small guide on how to start off on your self-hosting adventure.

Why self-host in the first place?

Hobbyists and professionals have a lot of different reasons for self-hosting services. These include (but are not limited to):

it’s an excuse to play around with computer hardware and rationalize the purchase of thousands of dollars of hardware
reduce reliance on “big tech”
the services solve an actual problem for the user (home automation, secure file storage, backups etc.)
learning new tech and solutions that otherwise have not come up during their day-to-day work

Regardless of the specifics, the main part of the self-hosting experience is to just have fun and work on challenges that you find interesting.

Step 0: find a problem to solve

One of the best motivators for setting up your homelab/self-hosting setup is to solve a real problem that you have.

Do you want to automate your home with HomeAssistant?

Or do you want your family to be able to back up their valuable photos and videos to your server using Nextcloud?

Or do you want to share your media collection with your family and friends using Jellyfin?

It doesn’t really matter what solution you end up using or how you do it, as long as it is a secure solution and that solves your original problem without creating 10 additional ones.

Step 1: find a place for running your software

Once you have a problem that you are going to solve, you will need to also find a home for your project to live in. For this, you have a lot of different options:

using a virtual machine at a cloud service provider (DigitalOcean, Linode etc.). Performance is not the best and beefier configurations can be quite expensive, but this is a reliable option with a fast network.
renting a server from a service provider (example: Hetzner). More performant than a virtual machine at most cloud service providers, but probably more expensive as well. Usually comes with a fast network.
an old laptop that you have collecting dust on your shelf. Not the fastest or the quietest option, but it does not use a lot of power.
an old desktop PC. Probably plenty fast for any task that you can throw at it, but its power usage is higher compared to laptops.
a tiny desktop PC. Low power usage, but can pack quite a punch. For inspiration, check out Project TinyMiniMicro by ServeTheHome.
a single-board computer, such as a Raspberry Pi or alternatives. Plenty of performance for most workloads, uses very little power (5-15W generally) and silent as well.
a second-hand rack-mounted server. Probably quite performant, but uses a lot of power and is very noisy.
a NAS box from one of the more popular providers (Synology, QNAP, TerraMaster etc.). Useful for when you need a lot of storage on your machine or something that just works.

Your choice will likely be affected by other factors as well:

your home internet connection. If your ISP does not allow opening ports or if your upload/download speeds are not that great, then using a cloud service provider or renting a server might be a better option for you.
living in a small apartment will likely rule out noisy solutions, such as full-blown rack-mounted servers or powerful desktop PC-s.
if all you want to do is local testing, then a virtual machine on your already existing desktop/laptop PC might be a better fit, as it does not require you to buy any new/used hardware.
reliability and price of electricity. In some countries, running a server in your home might result in a huge power bill or might not be possible at all due to the power grid being unreliable.

The workloads that you plan on running will also have an effect on your chosen approach. You will need to take into account the resources that your workload requires. It does not make much sense to get a rack-mount server for hosting your Wordpress blog if a Raspberry Pi can do the same job just as well. On the other hand, running a CPU-heavy service like Jellyfin on a Raspberry Pi will be a painful experience when compared to running it on a desktop PC or an enterprise-grade server. If your service will be used by a high number of users, then you might want to opt for a more powerful machine.

Some solutions may also benefit from faster, SSD-based storage. Loading a lot of smaller files or running a big database off of hard drives will be noticeably slower than running them off of fast SSD-s.

There is no one size fits all solution. It’s better to do a little bit of initial research and use the right tool for the job. If in doubt, ask for help from someone more experienced.

Step 2: find the software solution that fits your use case the best

Just like with hardware, there is also a lot of choice when it comes to software.

Some solutions are pretty low-level and require setting up the OS and the services manually. This is not very user-friendly, but it offers the most potential for learning about your system and how to manage it. This approach will likely require you to learn new things and pay more attention to automating common tasks (updates, monitoring, backups) and making sure that everything is secure (no weak passwords, services properly isolated, regular and frequent updates etc.). If something goes wrong, then you will need to fix it yourself, but at least you have all the tools and knowledge to do so.

Example: Debian with ZFS for your main storage and services running using Docker or Podman (my personal preference).

There are also solutions that cater to people who just want to get work done. These solutions usually provide the user with a web-based user interface that can manage the system and any services that run on it. Do you want to set up your very own Wordpress blog? Just click this button and you will soon be set up with one without having to even know what the hell a “Docker” is. The downside of this approach is that when things go wrong, you will still need to familiarize yourself with the inner workings of the system. Troubleshooting issues with no preparation and learning new things under stressful circumstances is not very fun.

Cloud service providers have also started offering one-click solutions to setting up the more popular services. your very own Minecraft server is just a click away!

Examples:

a NAS box from Synology that is aimed towards consumers
VM provided and set up by Linode using a one-click solution in their web UI.

And then there are solutions that fit in somewhere in-between. Solutions like TrueNAS (formerly FreeNAS) and OpenMediaVault are made for users that can probably set things up themselves as well, but are just too lazy to do it. Using these solutions is not in any way worse than the alternatives. If it solves a problem and fits your use case, then go ahead and use it! Just keep in mind that you might need to peek under the hood when there are issues where the GUI cannot help you out.

As always, if in doubt, ask for assistance. It is perfectly fine for you to pick one of the user-friendlier options first and dig deeper once you feel more comfortable. Self-hosting is not a race, you can do everything at your own pace.

Step 3: managing your setup

Once you have everything up and running, there are some topics that you really should pay attention to in order to avoid big problems down the line.

Backups

If the only copy of your files lives on your NAS and something should happen to them (accidental delete, hard drives suddenly die, you get hit by ransomware etc.), then you are in a world of hurt. Having backups is a mandatory part of your self-hosting adventure. A mix of automated and manual backups is better than having no backups at all.

Some ideas for backups:

external hard drive attached to your server that receives automated backups that run on an schedule. Just make sure that any failures during the backup will be communicated to you (over e-mail, for example). It’s also a good idea to occasionally check if the last backup was successful or not. Downside of this approach is that since this drive is still connected to the server, it is subject to other events that can shred your data (ransomware, power surges etc.).
offline backups that you will manually run on a regular schedule. Example: copying all files from the server to an external hard drive on the 1st day of each month. If you accidentally delete all files on your server or a piece of malware does this for you, or if your server is toast due to a power surge, you will still have a copy of the data that cannot be affected by those kinds of issues.
snapshots. While technically not a backup, snapshots do help against accidental deletes. Depending on your setup, you might be able to retrieve files deleted a week ago if you keep snapshots of your data for the last 30 days.
a hard drive at your friends house. Helps protect against the worst case scenario where your house burns down.
(encrypted) backups to cloud storage providers, such as Backblaze, OneDrive or Google Drive. It will require a monthly subscription, but at least you will likely have a copy of the file somewhere. Just keep in mind that if you should get locked out of the service or fail to pay for it, then you might not be able to access the data hosted at that service.

Security

If you are exposing a service to the wild wild west that is the internet, then you will be a target. Automated scanners will poke your services and if they find a vulnerability, they will use it. To prevent the worst, consider the following:

keep everything up to date. Don’t forget that for some updates to apply (example: kernel updates), you need to restart the machine. For other services (example: ssh), they can be restarted without restarting the whole machine. Regular updates will help prevent most security issues.
only expose services that you absolutely need to expose. With something like a web server, you really don’t have a lot of choice here, but with services aimed towards a known set of users, you can limit the exposed surface area by using a VPN or enforcing other restrictions (whitelisted IP addresses, fail2ban to block most intrusion attempts, etc.).
make sure that your configuration is correct. A lot of intrusions are caused by configuration issues that usually end up with a service being exposed to the web without any passwords or other protections set up. It’s a good idea to check if you are accidentally exposing too much by checking what others are seeing when they try to connect to your service.
try to set up your services so that you are prepared for an intrusion. If the attacker is able to take control of service X, but it is completely isolated from other services and devices on the network, then the damage they can do is likely limited.

Monitoring

Your setup will likely have issues. Hard drives might die, or the CPU might be overheating due to years of dust accumulation, or one service might be using up all the available resources, causing others to slow down as well. A better understanding of what your system is doing will help out a lot here.

Cloud hosting service providers usually provide at least a basic level overview of your resource usage: CPU, RAM, disk, network etc. If you are running your own machine, then you will need to figure this out on your own.

To understand what your system is doing, a tool like Netdata is a good starting point. If your server slows down, then you can most likely pinpoint the reason by looking at the resource indicators.

It is not realistic for you to keep an eye on all the indicators all the time, so having some kind of alerting set up is also a great idea. Events, such as a hard drive starting to throw errors, the CPU usage being at 100% for more than an hour, or a service not being reachable, are all something that you might be interested in hearing about.

You don’t have to monitor everything and have alerts set up to catch everything, but it is a great starting point for responding to issues proactively. Ending up with all your data lost because you did not hear about hard drive failures is not a great situation to be in.

What next?

If you have reached this point, then you are probably aware of most of the topics that will come up during your self-hosting adventure. You will encounter issues that will be specific to the approach that you have chosen at the hardware and software steps, but that’s perfectly normal. Most of these issues have solutions that are one search away, and if you have trouble figuring something out, then feel free to ask for help!

If you need new ideas, then check out the subreddits mentioned at the start of this post, you might find some gems in there! Do keep in mind that you will likely end up in an endless loop of building something, being happy with it for a week and finding something else to try out. It’s really fun, though.

My server setups throughout the years

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Wed, 17 Mar 2021 10:00:00 +0200

My self-hosting adventure started during my university days. Once I actually had money to buy myself a decent used laptop (ThinkPad T430) that had more than 3GB of RAM that my ThinkPad T60 had, I could use some of my older hardware as testbeds for any of the stupid ideas that I had. For someone that liked to learn more about Linux and wanted any excuse to play around with hardware, something like a RAID0 setup over 4 80GB Samsung hard drives on an old PC setup sounded great.

I also had a small collection of data that I had on an 1TB external hard drive, and I wanted to also have backups of the data that lived on my laptop. This was the main motivation for setting up a machine that would serve as a NAS and a testbed for anything that I wanted to try out.

The first box: AMD Athlon 64 X2 4200+ based PC

At this point in time, I didn’t really have any names for my systems. They were simply named by the model of the machine or whatever else I had in mind.

One of the first “server” builds that I built was based on some old parts I had lying around. The CPU was from an old family PC that we got in 2006, and to this date, it is the only surviving component from that build. The motherboard was from a place that collected old hardware and resold some of it and if I remember correctly, it was the abit KN9 Ultra. AM2 socket, DDR2, 6 SATA ports, all that good stuff. This was coupled with 4GB of DDR2 memory. Not much, but it is a start.

The heart of the system. It wasn't fast nor power efficient, but at least I had something.

The details regarding the hard drive situation are a bit hazy, but at least initially it hosted whatever drives I had around. At some point I managed to buy 8x 500GB hard drives to test out ZFS on a big array. This was mainly inspired by the glorious 32TB server build in an NZXT S340 case that someone showed off in Reddit. I’m still surprised that it was acceptable for me to be in the same room as that monstrosity.

This setup ran well for a while and there weren’t any hardware issues with this setup. Drives were OK, speed was acceptable, and ZFS did its job well, since this was all a big RAID-Z3 array. I even removed two drives temporarily to put in a GPU at one point and ZFS did not complain. After putting the drives back in, the drives were resilvered and all was good.

The server box running a more sensible setup of less than 8 hard drives.

The engoodening of my server: AMD Ryzen

2017-2018 was wild in the world of CPU-s. AMD came in with some well-designed CPU-s that provided a much better value proposition than the competing Intel CPU-s at the time. The value was insane: double the cores for the same price! It didn’t take much for me to be convinced that the next server build will be an AMD Ryzen based machine, and as soon as I had the resources, I got myself parts for doing that upgrade.

Picture taken moments before a lot of fun was had.

The build wasn’t without hiccups, though. I discovered that cats really like playing around with screws and walking on all the expensive computer parts that took up half of the free space in the apartment. Nevertheless, I managed to complete the build.

How to migrate from an 8x HDD setup to something more sensible.

New parts, new drives.

Laptop-as-a-server

Due to reasons that have no rational explanations, I started looking into switching out this perfectly working server machine into something more power efficient. The CPU sat idle most of the time and I figured that I could use the machine somewhere else.

And thus, one of the stupidest server builds was born: a ThinkPad X230 serving as a server, with two 4TB WD Red hard drives attached over USB 3.0, running BTRFS RAID1. I do not have pictures of the setup, but this was pretty much it. I learned soon after that cats like climbing on things and that they have no understanding of the amount of force that is required to temporarily disconnect the USB 3 cable from the laptop. This is the part where BTRFS failed on me and I actually lost data in the array. Luckily I had working backups.

Not long after this incident, I started looking for alternatives.

Low power, but still somewhat usable?

My search criteria was quite simple:

low power (ruled out any desktop builds)
x86 based (no OS lock-in)
hosts at least 2 drives over SATA
reasonably priced

The closest match I could find was the TerraMaster F2-221. It has a low-power Intel CPU, standard UEFI implementation, fits two drives and has two SATA ports. The only part that it really didn’t fit under well was the reasonable cost aspect, but it was still much more affordable than the alternatives.

Around this time I also discovered the world of shucking. I got myself two 8TB WD MyBook drives and used them in this build. However, what I did not take into account was the noise. These 8TB drives were much louder than the 4TB WD Red drives that I used previously, and allegedly this is due to WD labelling these drives as “5400-rpm class”, even though they actually run at 7200 rpm. The TerraMaster NAS unit had no vibration dampeners or other methods to isolate the noise, so I had to stick it in a closet.

The TerraMaster build was otherwise great. I could upgrade the RAM to 10GB using the available DDR3 SODIMM slot on the board, and the services that ran on it were fine. This thing could even run Jellyfin, assuming that you utilized the integrated GPU to provide hardware acceleration for transcoding video.

However, there were some limitations for this setup:

OS lives on an USB drive. Generally fine, but depending on your setup you could pick between internal USB 2.0 ports or one of the USB 3.0 ports at the back.
When you have lots of small files and a database on the system as well, the lack of SSD base storage became quite painful. You could do it over the available USB 3.0 ports on the back, but that’s it.

Yes, that is an mSATA drive in an USB enclosure stuck inside the TerraMaster chassis.

Back to the desktop

Due to a combination of me getting two ITX builds and having workloads that are performance-sensitive (PostgreSQL, Nextcloud, Jellyfin), I decided to go back to where I started from and put the setup back into a desktop case. For that purpose, I picked the Thermaltake Core V1, mainly due to how easy it is to build in it and it having room for 2x8TB hard drives and 2x 2.5" SSD-s.

Clean and performant server setup.

For future historians: yes, even servers had RGB in 2020.

This box was very fast. Too fast, in fact. It also used quite a lot of power. 70-120W doesn’t sound like much, but when this box is supposed to run for 24 hours a day, this really adds up. I had reached my storage target, the performance was acceptable and everything was working, which meant that the next logical step is to change the setup and hit some limits again.

Quest for low power usage

At this point, I had made it a game for myself to try to go as low as possible with the power usage of my setup while also running my workloads at an acceptable performance level. From my previous meddling, I had a ThinkPad T430 that now boasts 16GB of RAM and a 4-core Intel i7 3820QM quad-core CPU, which is plenty of performance. This laptop also houses up to three SSD-s, which meant that having performant storage around over SATA was also not an issue. The main storage (2x 8TB) was attached over USB 3.0. Not the best solution, but ZFS has handled these kinds of setups very well in my past experience.

Sure, it's low power, but the cables are a mess.

Can we go even smaller?

Recently I have learned about tiny form factor machines produced by big OEM-s, such as Dell, HP and Lenovo. These machines are very compact, but can pack quite a punch with a desktop-level CPU and the ability to put in 16+GB of RAM while being powered by a laptop power supply. The only downside of these builds is that the internal storage can be limited to 1-2 drives (SATA or M.2 form factor) and not all USB ports might be USB 3.0.

Once I stumbled upon one offer for a Lenovo M73, I knew that I had to try this kind of setup out. Initially I was thinking of using such a machine for media center purposes, but after I accidentally configured one too many UEFI settings for my ThinkPad X230 in the advanced BIOS settings, I needed a replacement laptop. And thus, the ThinkPad T430 based server was replaced by the M73. I did have to limit the CPU power usage using intel_pstate driver and the USB 3.0 ports needed a bit of bending so that they aren’t loose, but other than that it has been smooth sailing.

This is where the story ends for now. The machine takes about 13-15W when idling, and with 4 8TB external hard drives attached, the typical power usage is between 40-55W. I am planning on replacing the 4 drives with 2 high-capacity drives, something like a WD Elements/MyBook 14TB once those go on sale.

The current setup in all its glory. Don't worry, the cable mess is temporary until I get a PSU that can supply all the drives at once.

Future plans

This is probably where the setup changes will stop for a while. I occasionally play around with some alternative setups to try to go even lower (Raspberry Pi 4), but so far this setup has proven to be the minimum viable server. It does everything that I need, the power usage is low and it has room for expandability with regards to services that I can run on it.

If I had to start from scratch, then I would probably opt for an AsRock Deskmini X300. It has the same power efficiency benefits, but you could install a more powerful CPU, a lot of RAM and up to 4 SSD-s, two of which can be NVMe drives.

The bulk storage would still be over USB 3.0, but that comes with its own benefits:

you can not go over a reasonable amount of drives, meaning that your wallet will be happy.
you do not need to shuck the drives, you can use them as-is and get warranty service without issues.

This has been my self-hosting journey throughout the years. I hope that it was at least somewhat interesting and maybe gave you some ideas to test out on your own.

My attempt at archiving nfscars.net

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Mon, 01 Mar 2021 09:00:00 +0300

Background

My mother once had a laptop: the Compaq Armada 1592DT. It came with Windows ME which later got “upgraded” to Windows 98 SE after I managed to completely screw up the OS, a whopping 96MB of RAM and a hard drive that probably wasn’t much larger than 1-3GB. It wasn’t powerful or anything, but some of my earliest memories of playing video games on that thing were related to Sports Car GT and Need for Speed III Hot Pursuit. Yes, both were demos, and the performance was probably in the 10-20FPS range, but it was still great for a younger me. That laptop eventually got replaced by a Windows 98 based desktop which wasn’t a lot better with its 64MB of RAM, but it could play RuneScape (barely) and also Need for Speed III.

Need for Speed III Hot Pursuit holds a very special place in my heart. It looked great at the time, the police chases were fun and challenging, the soundtrack was fantastic and the track featured in the demo (Rocky Pass) is still something I could probably draw up from memory, assuming that I could draw. At some point I discovered that it is possible to mod the game to include new cars and tracks, which resulted in ridiculous speeds, cars that could absolutely destroy the police cars, and lots and lots of game crashes. One of the sources for these mods was nfscars.net.

nfscars.net

Depending on when you will be reading this, nfscars.net is/was a website that hosts/hosted thousands cars, tracks and other mods for various Need for Speed titles. In 2020, I decided to check this site out again to see if it is still operational.

The last post referred to an “upcoming” NFS title that was already released a year ago.

The link to the forums resulted in an error page being shown.

Should the site suddenly disappear without any warning, all the content would be lost. Since I have a small datahoarding bug, I decided to take matters into my own hands and do my best to archive the contents of the site. That, and nostalgia for the old Need for Speed titles was what pushed me to start this archiving adventure.

Initial efforts

Before trying any fancy technical solutions to crawl the site and get all of the assets that way, I decided to start from the source and tried contacting the site owner to arrange the download of the assets. That did not work out at all, since the contact e-mails were either not active or the inboxes were full.

I signed up for the site and tried to message some user accounts which hopefully were related to someone who was in charge. No dice.

There is a small chatroom on the site named “Shoutbox”, so I tried asking about the site owners there. It seems that a couple of users are still occasionally chatting there, mainly about random topics and spammy users. Unfortunately none of the users there - even the moderators - could not assist with setting up a backup of the site.

With the domain (potentially) expiring on 2021-05-02, I felt like I had to act now, because the site was probably running on borrowed time.

Technical solutions

I checked a selection of solutions for performing a crawl on the site and decided on Heritrix, mainly due to this being backed by the Internet Archive. If they cannot get archiving right, then who can?

Setting up Heritrix turned out to be quite a hassle. There does not seem to be a quick guide that you can follow to get started in 5 minutes. After stumbling around, learning the config file format and understanding that the best way to configure crawls was to do it externally with a text editor, I was set up to launch my first crawl. Even then, I hit some roadblocks:

The crawl was slow. I understand that Heritrix is just trying to be polite here, but if you have millions of URL-s to crawl and the site might go down at any moment, then time really isn’t on your side.
Checkpointing is supported, but for some reason the default configuration does not enable it. If the machine that you are running this on should restart, then your crawl will be stopped and you cannot resume at that point, you can only start over.
Heritrix is based on Java, which means that theoretically it should run on anything that runs Java, such as an ARM CPU based SBC. However, I ran into weird java.nio related issues with symlinking on those platforms and I just could not get them resolved.
If you run too many crawls at once, then Heritrix is perfectly capable of killing your LAN by using up all the available connections. It managed to rack up 16000+ active connections, which hit the 16384 connection limit on my consumer-grade router. That was not fun to debug.
Automatically continuing crawls from a checkpoint on system startup requires you to use the Heritrix API, which isn’t too complicated, but I just wish that this was another feature that it supported out of the box.

After messing up a couple of times and trying again, I finally managed to complete a crawl of the site. This only took 6 months, where the crawl itself ran for 3 months. MONTHS!

What the hell is a WARC?

Heritrix collects the results of the crawl into a special format called WARC, which it then compresses, resulting in lots of little .warc.gz files. If you extract one of these, you will end up with a .warc file.

To extract files from a WARC file, you need one of these tools listed here. After trying a couple, I decided to go with warcat since it seemed to mostly work and it supported combining smaller WARC files into bigger ones and extracting the contents.

Depending on the size of the site, the extracting process might take a while. Crawls can contain thousands or even millions of small little files, and if you are using hard drives for storing these, then it will be slow. I have not done testing on an SSD, but it will probably be much faster due to its superior random I/O performance.

Once I had extracted the contents, I did some checking to see if the assets that I care about are present in the crawl. Did a file search for a modded car in Need for Speed III.

No results.

Okay, that’s odd. Tried it with another file name.

No results.

Well, what do we have? I checked the extracted contents using ncdu and found that while Heritrix managed to grab a lot of the webpages and the images from the galleries, it failed to download files, such as .zip and .rar files that contain the modifications themselves.

Getting the final pieces of the puzzle

I started investigating the download functionality of the site. On every page load, the download link would have a new identifier attached to it. Upon clicking on it, the download would start. If you wait too long, then the download link would expire.

Firefox has a handy feature where you can copy the request into a curl command. I did that, added some parameters to it so that it would use the filename that comes from the response header and made it save the file to disk. It worked!

Now I had to figure out how to collect all of these download links. The site has sections for each game, which is also reflected in the URL. However, I found that when you navigate to the page of a mod and change the numerical ID at the end, you could cycle through all the cars, tracks and other tools for all games, even if the URL was technically referring to one game in the series.

After bashing around for a while, I came up with this masterpiece:

#!/bin/bash
cd /your/download/destination || exit 1

for index in {1..18050}; do
  echo "Index at $index, starting."
  filePath=($(curl --silent https://www.nfscars.net/need-for-speed-hot-pursuit/1/files/view/$index/ -H 'Cookie: csrftoken=XXX; sessionid=YYY' | grep -oE "files/download/send/[0-9]+/[0-9a-zA-Z]+"))
  echo "File path: $filePath"
  echo "Making directory $index"
  mkdir -p "$index"
  echo "Going into directory $index"
  cd $index

  for path in "${filePath[@]}"; do
    echo "Starting download on path https://www.nfscars.net/$path"
    curl --silent "https://www.nfscars.net/$path" -H 'User-Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:85.0) Gecko/20100101 Firefox/85.0' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' -H 'Accept-Language: en-US,en;q=0.5' --compressed -H 'Connection: keep-alive' -H 'Referer: https://www.nfscars.net/need-for-speed-hot-pursuit/1/files/view/1/' -H 'Cookie: csrftoken=XXX; sessionid=YYY' -H 'Upgrade-Insecure-Requests: 1' -OJ
    echo "Download $path done."
  done

  cd /your/download/destination 
  echo "Back in home dir. Index $index end."
done

Not the prettiest script around, but it did the trick. In a day or so the assets were downloaded and you could use the index to connect these to a page that is contained in the crawl.

Results

All this effort would be for nothing if I didn’t share my results, so I decided to pack these up and serve them for anyone that wants to keep a backup of at least one part of Need for Speed history.

The archived content can be found on the Internet Archive. Here’s a newer crawl from 2021.

The torrent files are also mirrored here:

Pieces of history get lost all the time with sites going offline. I’m hoping that at least with this effort I have managed to do my part in preserving a tiny piece of it.

2021-03-15 update

The site seems to be down at the moment and has been for a couple of days. Oh no.

2021-03-20 update

Somehow the site is alive again.

2024-08-20 update

It’s dead.

However, I’m delighted to find people using this archive to get their favourite mods!

Database optimization adventures on low-end hardware

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Sat, 27 Feb 2021 15:00:00 +0200

I used to work on a short-term project a while ago where the goal was to visualize some metrics that were collected from a pretty fancy smart home setup. This data included power usage of various sections of the building, temperature sensors, water usage levels and more. The data itself was collected by a proprietary piece of software that sent this data to a MySQL database. My job was to understand the data, get useful values from the binary data and visualize it.

The hardware

While testing out this solution, I started with an Orange Pi Zero that a good friend had given to me. You might remember this board from a previous adventure. This board boasts a whopping 4 ARM 32-bit CPU cores running at 1000MHz, 100 Mbps networking and one USB 2.0 port. The slow USB port means that the read speeds for any storage you attach to it will be capped out at 40 MB/s.

With all of this in mind, I still went ahead and built this monstrosity. I took a 480 GB Kingston SSD that was lying in a box and connected it to the board.

I'm pretty sure that the PCB of the SSD itself is larger than the SBC itself.

The workload

This server had two jobs: collect the incoming data from sensors to a MariaDB instance and visualize it using a tool like Grafana. The first part was OK, since the amount of incoming data was not that high and the SBC could handle it. However, I hit some roadblocks when trying to visualize it.

The first hurdle was to convert the binary values into something usable. Going by the documentation, I could figure out how to show signed and unsigned integer values correctly by utilizing some SQL magic that did the conversions more-or-less correctly. However, with other data types, such as floating-point values of varying precision and other fun formats, I could not use the same methods to perform the conversion.

The second hurdle was performance. The incoming data was put into one table, which would periodically get split up on a month-by-month basis to avoid the table getting too big. Not a bad solution, but since I was planning on using Grafana, having all the data in one table was the only sensible solution if I wanted to show data for previous months as well. This meant that I had one big table with all the important data, plus some adjacent ones that provided details for the sensor/meter type, its value type (integer, floating point, etc.) and more. Performing SELECT statements with a couple of joins proved to be a headache: the queries would trigger a full table search, which was very painful on a system where the maximum read speed of the SSD is 40 MB/s. Understandably, lookups on a database that was already at a couple of gigabytes in size and growing would be slow even for shorter time spans (30 days).

These limitations meant that in its current state, the solution was pretty limited in functionality and very slow to use as well.

The solution

I soon learned that the company behind the hardware and software of this system provides a tool which can be used to connect to the database and read the collected data, with values converted properly and sensor/meter names provided as well. And to top it all off, this thing was written in Java!

I decided to dig around in the provided .jar file using IntelliJ IDEA and stumbled upon some interesting files. These files corresponded to different data types, and also contained the logic that was used to convert the binary data to usable values. I can only guess that this conversion was made with the goal of keeping the database size as small as possible, or to just avoid users from using this data for other purposes using tools made by other companies.

There were still the performance issues to resolve. I looked into optimizing MySQL/MariaDB tables and queries and stumbled upon some useful tips and tricks. While I cannot remember the exact details, I did end up testing the different DB engines (InnoDB vs whatever they had previously), configuring various cache sizes and limits, indexes on various column types, investigating query plans and the performance benefits of denormalization.

And then it clicked.

What if I wrote a small Java service that would handle the data conversion and put all the data into one table that holds only the relevant data and has the correct set of indexes set up? The service would periodically check the main table that collected all the data, select the results, convert the value into something that a human can understand, and then insert it into a new table that is optimized for read performance.

After hours of work, I had this Java service up and running on the underpowered server and started testing this solution. Queries that would previously take minutes to run, would now finish within seconds. Even queries with a time window of 90+ days would still finish within 5 seconds. This improvement in performance is further amplified by the fact that Grafana is able to show multiple graphs at the same time, resulting in multiple queries running at the same time.

Deployment

The hardware that I tested this solution on was pretty neat and used up very little power. The total cost with the SSD included was roughly in the 100-150 euro range. However, it was decided that the deployment would be done on a more conventional machine.

I went ahead and built a tiny PC based on the AsRock Deskmini A300. The build included a dual-core AMD Athlon APU (cut-down version of a Ryzen APU), 16 GB of DDR4 RAM because it cost just as much as an 8 GB DDR4 kit did at the time, one 120 GB SATA SSD for the OS, two 512GB Kingston A1000 NVMe SSD-s for the main storage (ZFS mirror) and a cheap 480 GB Kingston A400 SATA SSD for local backups. This whole build fit in a 500 euro budget and was a bit overkill for the purpose, but the performance that you could get out of this build was insane. And to top it all off, the server itself is very tiny and energy efficient, meaning that you could stick it pretty much anywhere, and it would still run happily while consuming less than 50 watts even under the most intense workloads.

Yup, that's the whole motherboard. Very small, but it can pack quite a punch.

The performance optimizations that I did were a must-have on the SBC that I did most of my testing on. On this new build, however, all the bottlenecks were lifted. The CPU probably had more processing power than 20 of the Orange Pi Zero boards combined, and the maximum read speeds on the crazy fast ZFS mirror were measured in gigabytes per second. Queries that took a second to run on the old setup were now done in the blink of an eye (or in other words, measured in milliseconds).

This deployment would probably last years without any intervention from anyone, assuming that the incoming data would not grow too fast. Unfortunately the solution itself did not find much use, so I could not test this theory out. The box itself is still happily running at the time of writing.

Lessons learned

The main reason that I am writing about this experience is the simple fact that it was a fantastic challenge to try to extract as much performance out of the Orange Pi Zero based server. Getting your queries from 30 seconds down to 0.3 seconds felt like a great achievement, plus I got to revisit some database related topics that I first learned about during my university days.

This experience also shows that even low-end hardware can achieve a lot if you are aware of its limitations and can work around them with a little bit of effort. After this project, I have played around with my homelab setup a lot and have finally settled on a solution that does not consume a lot of power, but can still run all the workloads that I want to run.

Things I've learned during my self-hosting adventure

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Mon, 08 Feb 2021 16:00:00 +0200

This is a list of lessons that I’ve learned while playing around with my computer setup during the last 6-7 years or so.

USB connected storage is a bad idea when you are using BTRFS.
USB connected storage is still a bad idea when you are using ZFS, but it is at least much more resilient.
USB drive enclosures are a total crapshoot when it comes to the quality. Some are good, some will have random issues.
SMR based hard drives can be unusably slow for certain workloads.
Desktop PC-s use quite a lot of power when sitting mostly idle (50+ watts).
You might be OK with Intel QuickSync for your transcoding needs, try it out before opting for a much more powerful hardware configuration.
Some hard drives are very, very loud. If you live in an apartment, keep this in mind, otherwise you will start trying to rationalise the purchase of 8TB SSD-s to make your setup quieter.
ZFS snapshotting with a tool like Sanoid is invaluable.
You will need to use your offline backups, don’t think that you can get away without having some.
Five nines uptime is perfectly doable in a homelab environment, if you are willing to move the decimal point.
It’s all fun and games until you have actual people using services on your server.
Who needs a fancy monitoring setup when you have your friends and family members yelling that the server is down?
VM-s are great for service isolation, but terrible for flexibility and efficiency.
Containers, if kept up to date, are an amazing way to manage your numerous services.
BTRFS is one of the few filesystem options on 32-bit ARM CPU based machines that can still provide data integrity benefits.
Some workloads, such as databases and services with lots of random I/O, require an SSD to perform well.
If your workload is very heavily dependent on random I/O, then the “pro” branded SSD-s are actually worth the money.
SSD power usages can vary wildly, some may require 5V@1A, some up to 5V@1.8A. This will be problematic for hosts that cannot supply much power through the USB ports, such as a Raspberry Pi.
Armbian is great for SBC-s that support it.
Try running your database on a slow machine first. Any optimizations you will make will be very noticeable and your database will fly once you move it to a more powerful machine.
Old laptops are a great way to start your self-hosting adventure, especially if you can limit the fan noise with a tool like thinkfan.
External hard drives, such as the WD Elements or WD MyBook, are a great way to build multi-terabyte storage setups without breaking the bank.
Raspberry Pi 4 specs sound good on paper, but its limited I/O capabilities prevent it from doing too many things at once.
When setting up your system, document every command and installed package for future reference. You never know when it may come in handy.
Configure your setup for when you get hacked.
Things will break at the most inconvenient time.
Some computers can be very temperamental. However, it might be easier to handle the quirks of one system rather than getting a replacement and learning about its quirks the hard way.
Some workloads, such as Heritrix, might not run well on all platforms, such as arm64 or armhf.
The original Raspberry Pi 1 B+ is just too slow for most useful workloads in 2021, even when overclocked.
Having a filesystem that can protect from data corruption is a must-have.
Only use ZFS SLOG and L2ARC devices if you are absolutely sure that your workloads will benefit from them. If your current ZFS ARC cache hit rate is already over 90+%, then it probably is not worth it.
Netdata is both beautiful and useful.
OS support cycles mean nothing and can be changed on a whim.
If you want a stable ZFS setup, use a distro that has a stable kernel version. Debian and Ubuntu are great for this purpose.
Update and restart as often as possible. It’s much easier to rule out what broke your setup if the amount of changes that have built up is small.
Automate the way you deploy configuration changes to your server.

How to fix ZFS pool not importing at boot

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Mon, 01 Feb 2021 08:45:27 +0200

Issue description

You are running a Linux-based machine with an install of ZFS on Linux. Everything seems to work correctly, but after restarting your machine, the ZFS pool is not visible. You can still import your pool manually using zpool import poolname or zpool import -a.

In my situation, this issue does not occur with Debian 10 on an x86-based machine, but it does occur on a Raspberry Pi 4 running Ubuntu Server 20.04.1 LTS.

Setup:

Raspberry Pi 4 (4GB)
Ubuntu Server 20.04.1
32GB microSD card for root filesystem
2x 8TB WD Elements external USB 3.0 drives (ZFS mirror)

Potential causes and solutions

When looking this issue up, there were many potential causes and solutions provided:

In my specific scenario, none of these seemed to do the trick.

Investigation

The ZFS pools should be automatically imported on boot by a service named zfs-import-cache.service. The cache file should contain information about the imported pools.

Once you have started your system and don’t see any ZFS pools available, try running systemctl status zfs-import-cache.service or journalctl -u zfs-import-cache.service. This should show you if the automatic import succeeded or not.

In my case, the failure shows up like this.

The pool tanker is a ZFS mirror that is supported by two 8TB WD Elements external hard drives. They are big, loud and require some time to spin up the platters before you can actually read data off of them. At this point in the investigation, a good friend recommended that I check what disks were recognized by the system at the time when this import service was starting up.

For that, you need to modify the systemd service by running systemctl edit zfs-import-cache.service and put these contents in there:

[Service]
ExecStartPre=/usr/bin/lsblk

This addition will run the lsblk command before the command specified in the systemd service is actually executed. The output of this command will be visible in the service logs. After making this addition, reboot your system and run journalctl -u zfs-import-cache.service again. You should be able to see all the connected drives on the system.

In my case, the system recognized the microSD card and an SSD that was connected over USB, but not the hard drives. This was a very clear indication that the system was not waiting until the drives had spun up before attempting to import the ZFS pool.

Hacky solution

To resolve this issue quickly and move on with more interesting things, you can use the same trick that we used for troubleshooting this issue, but instead of listing the connected hard drives, we can make the systemd service delay its execution by putting a sleep statement in there.

Edit the systemd service again with systemctl edit zfs-import-cache.service and replace the contents with this:

[Service]
ExecStartPre=/usr/bin/sleep 15

This will make the service sleep for 15 seconds before importing the ZFS pool(s). It could be 10, or 30, or whatever value works for you, 15 seconds was chosen here because it is more than enough time for the disks to spin up.

Proper solution (probably)

There are probably better solutions, such as messing around with udev rules or enforcing the delay somewhere else in the ZFS pool import chain of operations, but I have not investigated them at this point. Feel free to contact me with a better solution, and I will update the post accordingly.

Quest towards energy efficiency part 1: the water heater

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Mon, 25 Jan 2021 15:00:00 +0300

While listening to the SelfHosted podcast, one of the hosts brought up their peculiar living arrangement multiple times. Essentially, he and his family are living in an RV that runs off of batteries, has solar panels, utilizes Home Assistant for automation, supports multiple mobile networking providers and also has some Raspberry Pi-s running in a homelab. In such an environment, low power usage is critical.

That got me thinking: how much electricity am I currently using and how low can I go?

Could lowering my overall power usage help transition to an off-grid setup in the future?

Initial measurements

Before making any changes, it is important to make some measurements. After all, if you don’t get an overview of the current situation, then how are you going to do comparisons? Your local power company is measuring the total power usage for you, so you might as well utilize that to get an overview of daily and monthly power usage.

In my situation, the monthly power use would be between 200-250 kWh and daily power usage would jump between 6 kWh and 13 kWh, depending on the day. I am not sure if that is normal power usage or not, but at least it’s a starting point.

Example daily power usage graph. Blue parts indicate nighttime pricing (cheaper).

What’s using all this power?

There are many candidates for appliances that use the most power: water heater, refrigerator, electrical oven, electrical kettle etc. With some of them, it’s quite easy to reduce the power usage: just use it less! With others, it might not be as straightforward.

For example, with a refrigerator, you cannot really use it less. Sure, you can try to open the door fewer times so that it has to do less work, but you really shouldn’t try to limit its operating window, as that can result in food going bad. It will probably help if you got yourself a refrigerator that has a better efficiency rating, but that’s it.

However, with a water heater, you are free to limit its use significantly. After all, the worst thing that can happen is that you might run out of hot water while showering or washing the dishes. Today we will thus focus on an old water heater and try to see how low we can go.

Reducing the total power usage of a water heater

The way that a water heater works is quite simple: if the water temperature is below a threshold, heat it. This is usually controlled with a dial on the water heater itself. After you have used hot water for a while, you will notice the water heater kicking in and doing its thing.

A water heater can use quite a lot of power. In my case, it uses 2000W while heating the water. This result was measured by using an electronic power usage meter that you plug between the appliance and the wall socket. From the data provided by the power company, the days with showers or other activities that used hot water spiked up to 13 kWh, which is quite a lot when compared to the daily average of 8 kWh.

Water heaters also have an insulation layer to prevent the water from losing heat. Once the water is heated up to the desired temperature, it should be able to keep it at that temperature for quite some time without having to heat it up again.

With that in mind, I tried a simple approach: buy a mechanical timer, set it to run at certain times throughout the day, test out the results and find the sweet spot. After all, why would I want it to run during the night, when nobody is using hot water anyway?

Initial testing

I picked a sensible starting point: run for about 1-2 hours in the morning and again some time in the afternoon (16-18). By starting with that, I should be able to avoid situations where I run out of hot water at the most inconvenient time. This worked well, but due to the water heater potentially running for up to 4 hours, it would still mean a maximum of 8 kWh power use for the water heater alone.

During the next week, I experimented with shorter values. I limited the evening heat cycle to 30 minutes and went as low as 45 minutes in the morning heat cycle. At that point the water didn’t heat up enough in the mornings and the boost in the afternoon didn’t really resolve the problem, either, which meant that I found the low point.

With more recent tests, I have found that for our use case it is perfectly viable to set the timer to run for 90 minutes in the morning and have hot (or warm) water for the rest of the day, while also allowing for comfortable showers in the morning and afternoon. Just make sure to set the temperature setting to the highest value so that it does not stop heating the water while the water heater is allowed to run.

Results

While looking at daily power reports, a couple of things become clear:

the spikes have now been replaced with a more stable usage
it is possible to limit the water heater usage down to 3 kWh per day without giving up too much convenience
a mechanical timer allows me to use power while it is cheap (nighttime)
with water heater use concentrated on one specific time slot, it is easier to identify other appliances with high power usage
you will suddenly find motivation to shower as fast as possible, because who knows how much hot water you have left?

Not very hard to guess when the water heater is doing its job.

Is it worth it?

Not from a financial standpoint, definitely. However, it does give a good feeling when you are living in a country that is relying on one of the most polluting energy sources.

To be quite fair, at this point it has become more of a game for me than anything else. It’s fun to find out the limits and try to push them as far as you can go.

Are there better solutions?

Probably. My solution here is cheap (the mechanical timer only cost 4 euros), but there could be smarter solutions that learned your daily habits and tried to optimize the water heating process using that. However, predicting when you need the hot water becomes much more difficult if you deviate from your usual routine. Your children found the wonders of playing in a puddle and they need to shower at an unusual time? Well, there goes all that data collection and machine learning effort.

Alternatively, you could risk getting electrocuted and get an attachment that will heat up water on-demand. The downside with that option is the hassle and upfront investment of having your kitchen sink and shower set up like that.

What next?

Now that one of the big ticket items is handled, I can shift focus to my computing setup.

What can you do to have a more efficient homelab?

Do you need to sacrifice anything in your attempt to balance data hoarding and low power use cravings?

How much power is your desktop PC using while doing nothing?

Stay tuned for future installments of my quest towards energy efficiency!

2022-01-18 update

At some point I stopped using this trick because while I was paying a bit less on my electricity bills, I still consumed a lot of power. On a 35 EUR power bill, that would have meant about a 5 EUR reduction in cost. You still pay to have those electrons moved to your home. Using the water heater on a lower temperature setting throughout the day used less power in total due to having less water to heat up and the hot water usage being a bit unpredictable.

However, with the current state of the energy market, the prices have been multiplied. I’ve gone back to this setup to take advantage of lower rates during nighttime, which tend to be 2-4x cheaper. At these prices, there’s a real benefit to trying to use the off-peak hours. Let’s see how well this works out.

How to make digital copies of your old video tapes

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Sun, 27 Dec 2020 18:00:00 +0300

Introduction

This is a short overview of one way to take your old video tapes and make digital copies of their contents.

To get started, you will need the following:

old video tapes that you want to make copies of (VHS and its variants)
a device that can read analog AV signals and convert them to digital signals
a device that can play back the tapes (VHS player, camcorder etc.)
- in this example, we will be using a camcorder that has video playback capabilities.
a computer with enough free space to store the digital copies
- for VHS and H.264+MP3, this means roughly 1.5GB of space for 1 hour of footage
VLC media player
time and patience

In this example, we will be using the Nedis VGRRU100BK. Not because it is any good, it was just something we had access to.

Nedis VGRRU100BK - front.

Nedis VGRRU100BK - back.

The process

Step 1: setting up the physical devices

Set up your video playback device so that it can play the cassettes that you are planning to record.

Connect the AV output cables (composite or S-video) to the input of the USB device. For this specific device, it means connecting the appropriate composite AV cables together and connecting them to the USB stick using a proprietary connector.

The end result might look something like this:

Overview of the setup used in this example.

The USB capture device connected to the PC.

We are using a camcorder to play back the video in this case.

Video outputs on the camcorder. Yes, that is mono sound.

Video output -> USB capture card input.

Step 2: testing the initial display output

In this example, we will be using VLC to play and record the videos. We are using Linux here, but the process should be very similar on Windows.

Open up VLC.

In the top-left corner, click on Media and select Open Capture device....

You should see a screen similar to this one:

Your computer probably has multiple options for selecting audio and video devices, since microphones and webcams will likely show up here. If you are unsure which one is the correct one, go through each one-by-one until you find the options that work.

In this example, our video capture device is named /dev/video2 and the audio feed for the capture device is hw:4,0.

In the bottom-left corner, you have the option to play back the video signal directly in VLC. You can use this to check which video and audio device is the correct one.

If all goes well, you should be able to see the output in VLC:

Camcorder output being streamed to VLC using the USB capture device.

Step 3: recording

Once you have found the correct configuration, you can click on the arrow next to the Play button and select the Convert option.

In this view, you can configure deinterlacing, select the video profile, and set where the resulting video file will be saved.

For our purposes, we have chosen to deinterlace the video, since it provided a higher quality output, and we have stuck with MP4 as the output format because it is an acceptable default option for us.

Displaying the output while converting the video did not work for us and that might be related to how VLC accesses the video feed in Linux (only one program can read the stream at a time, it seems).

You can now click Start in VLC to start recording the video. You will also need to hit Play on your video playback device.

Wait until the VHS tape has finished playing and then hit the stop button in VLC.

Step 4: verify results

You can now open the resulting video file and check the results.

If you have additional VHS tapes to record, then repeat step 3.

Troubleshooting notes

There were some challenges to get this setup working.

Not able to read the output from the USB capture device

Being able to read the video signal from the USB capture device turned out to be a headache. The software that comes with the USB capture device (Arcsoft ShowBiz) had issues displaying the video signal and was generally unreliable and crashing.

Other software, like OpenShot and Kdenlive video editors, could not recognize the video capture device.

Since we found that this USB capture device emulates an USB webcam, we tried viewing its signal using a browser or a camera app in the OS (Cheese in Linux, Camera in Windows), but that did not work out either.

We eventually found success with VLC and then chose it for performing our video recording effort. It has multiple advantages over the software shipped with the capture device itself:

it just works
free and open source, meaning that you can do your recording on different platforms
no need for a license key
up to date (Arcsoft ShowBiz installation files were unchanged since 2012)

VLC truly is a Swiss army knife for media playback and recording.

VHS playback troubles

We initially tried the recording process with a proper VHS player, but while recording the footage and reviewing it later, we noticed that at random points the VHS player had fast-forwarded over portions of the video.

The camcorder did not exhibit this behaviour, so we went with that. However, with the camcorder we could only output mono sound, which means that the resulting video file will only have one sound channel present. This will be especially noticeable when listening to the recorded footage using headphones. To fix this, we used HandBrake to copy the single audio channel to both channels, which resolved that issue for us.

What running out of SATA ports looks like

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Sat, 29 Aug 2020 17:21:21 +0300

One day I stumbled upon a Craft Computing video about his new server build, and on the parts list was a Sedna PCI Express 2x SATA adapter card:

The card in question.

There are different designs out there, including a 4x SATA card, which is bonkers.

Anyway, I thought that I might as well give this a shot, since I have two 4TB 2.5" 15mm Seagate hard drives running, with one of them being in the process of accumulating bad sectors.

Adapter card with two drives installed (front).

Adapter card with two drives installed (back).

The installation procedure itself is simple, just take your drives, screw them in and plug this into a PCIe slot.

This is what it looks like installed into my server.

Alternate angle.

As you can see, it did not kill the system.

So, how well does it work?

Quite well, actually.

The PCIe 2.0 x1 link does mean that your speeds will be capped at around 500MB/s, but with current hard drives you are unlikely to reach this limit. SSD-s, on the other hand, will be bottlenecked in sequential read/write workloads.

lspci -nnk output:

01:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 01)
        Subsystem: ASMedia Technology Inc. Device [1b21:1060]
        Kernel driver in use: ahci
        Kernel modules: ahci

The drives are individually addressable and show up as normal drives (like /dev/sda). S.M.A.R.T info is functional and readable.

This adapter has been running in my system for over half a week now and so far there haven’t been any stability issues whatsoever.

Should I get this?

This adapter is a good fit for the following use cases:

there are no more HDD/SSD drive mounts in your system left and you do not want to use double-sided tape as one
you have run out of SATA ports and don’t mind using up a PCIe slot
you want to install additional SATA drives in your system, but also would prefer to keep the amount of cabling in your system at a minimum
flexing on your friends with your fancy new 8/10 TB storage card

I would recommend looking into alternative solutions if your use case matches one of these:

using this adapter with SSD-s on sequential read/write workloads that exceed 500+MB/s
adding more than two SATA drives into your system (get a simple PCIe -> SATA adapter that only provides the data connectors instead)

Running on fumes

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Sat, 29 Aug 2020 16:42:19 +0300

After a run-of-the-mill Windows BSOD, I was redirected to the UEFI settings and was presented with this fun little bug:

0GB ought to be enough for anybody.

Stupid project ideas: the War Machine

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Sat, 22 Aug 2020 21:24:08 +0300

LinusTechTips has some pretty great videos about building insane PC and server setups, such as:

I have the ideas, but not the budget of LinusTechTips, so writing them down and hoping that one day I can test this out is the best I can do for now.

The project itself

The main idea is to have a laptop that has ridiculous amounts of storage and the performance to back it up. The idea started gaining traction in my head after I realized that my Lenovo ThinkPad T430 can house up to 3 SSD-s, which makes it a perfect candidate for a system with a ZFS mirror configuration. It also helps that it is powered by a quad-core CPU and 16GB of RAM.

Additional points:

low power, should stay below 60-90W even under the worst case scenarios
portable, just plug in the power and an ethernet cable and you are good to go
built-in UPS in the form of the laptop battery
well built, can survive a fall or two (already has done so, but that’s a story for another time)
includes a screen and a keyboard for those times where you really screwed something up and need to fix it fast
supports 1x mSATA SSD (great for / partition) and 2x SATA HDD/SSD (one in the usual drive bay, the other in the HDD caddy that is replacing the optical drive)

Storage

With SSD prices having dropped quite low and capacities increasing with time, it might not be completely unreasonable to get SSD-s with capacities reaching multiple terabytes. At the time of writing, Samsung, for example, is shipping SATA SSD-s with capacities up to 8TB, which is the same size as the main hard drives in my server right now.

However, these SSD-s are also ridiculously expensive, with the 4 TB model costing ~400 euros and the 8 TB costing almost 800 euros.

The capacity isn't the only thing that is big about this SSD.

Hard drives are generally cheaper per TB, but in this case we don’t actually have much better options. 2.5" 9mm hard drives seem to go up to 2TB at most. You will also have to take into account the fact that hard drives do not like impacts, such as falls, and they are much slower than SSD-s in most workloads.

Cooling

Since I have fitted this T430 with an Intel i7 3820QM CPU with a 45 watt TDP, cooling is an issue. Liquid metal, beefier heatsinks, fresh thermal paste, I have tried it all, but the only solution that has actually worked is disabling the turbo boost feature. There is also this other solution I created that automatically throttles the CPU at a chosen temperature limit using Intel p-state driver, but that solution requires a rewrite.

With poor cooling comes great noise, so ideally this server should run in a place that does not bother anyone with its high-pitched fan noise.

Performance

With a quad core CPU, this system can do quite a lot. Running Nextcloud, PostgreSQL, Jellyfin or any other service should be a breeze, especially since the main storage is SSD based. The amount of RAM is also perfectly suitable for doing quite a lot. Run a couple of VM-s, transcode your media, it can do it all (with reasonable expectations, though).

Portability

One of the best features of this setup is that you can just grab it and go, should the need arise. Its low power requirement makes it a much more suitable candidate for powering it in situations where the electrical grid is down and all you have is a generator or some form of renewable energy. In addition to that, everything you need to work with the machine is already there, just open up the laptop and start working using the built-in keyboard and screen.

The good build quality of the laptop also means that you do not need to worry much about putting it in your backpack and throwing it around. Sure, the plastic might break in some places, and if the impact is strong enough then it can still break, but it is less likely to break when compared to alternative thin and light types of laptops.

Practicality

It’s not all fun and games, though. This setup does have quite a few downsides:

no way to upgrade CPU and RAM any further. What you have is what you will have to live with.
it’s expensive simply due to the fact that the SSD-s get very pricy if you require high capacities, hard drives are still more economical for storing huge amounts of data
the cooling issue will remain unsolveable, unless you invest time and resources into designing a custom solution

Conclusion

Do I still want to build this?

Yes.

Can I build this?

Not unless a delivery truck full of 8TB SSD-s happens to crash in front of me.

Or someone sponsors me. Take a hint, WD/Seagate.

Disk is OK

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Sat, 22 Aug 2020 07:29:19 +0300

Maybe I am unlucky with computer hardware.

Maybe I have the opposite of the golden touch of Midas.

Or maybe it’s just the fact that I use my hardware for things they were never meant to be used for.

Anyway, I have acquired two Seagate 4TB external hard drives for doing some testing with ZFS. I shucked them (took them out of their enclosure) and started using them as internal drives in many different configurations.

Funnily enough, even though the two external drives had different branding (Backup Plus, Expansion) and enclosure design, the hard drives in them were identical.

Some info about the drives:

Model: ST4000LM024-2AN17V
Recording type: SMR (shingled magnetic recording)
Form factor: 2.5"
Interface: SATA
Thickness: 15 mm

Before I get to the stupid stuff, I would like to mention what these drives are good for, and where they suffer heavily.

Good:

WORM (write once, read many) type of workloads, such as a Steam library, your media collection etc.

Bad:

random read/write workloads (performance is absolutely rubbish and makes the system unusable)
write heavy workloads (SMR requires some clean-up activities that are managed by the drive, and you cannot do anything while those take place)

The stupid stuff

I have done some pretty dumb things with these, such as using them in an InWin Chopin computer case and letting them hit 60C under load. Fun fact: these drives actually fit almost perfectly in the case, as long as you bend one small metal section for one of the drive cages. Not so fun fact: they will overheat if you have a high CPU load and you cannot do much about that.

Currently these drives are sitting in my main desktop. Initially I tried using them in a mirrored configuration under Windows, because having your whole Steam library downloaded with disk space to spare is pretty cool. Unfortunately, one of the disks started having issues and it got kicked out of the mirror. It kept doing this every time it accumulated bad sectors.

Now these two sit in a ZFS mirror configuration inside a VM, acting as some sort of a temporary scratch disk. Another free fun fact: kvm (or qemu, or libvirt, one of those) will pause the VM if it detects that one of your disks has encountered I/O errors, and seems that the only “fix” is to force a restart of the VM (could not unpause the VM).

It has become some sort of a fun pastime for me to keep an eye on one of the failing drives and the status of the ZFS mirror.

Will ZFS detect some errors?

How large can the bad sector count get before the drive dies?

When will it actually die?

Why is S.M.A.R.T still reporting that the disk is OK?

So far, so good.

2020-08-17. Nope, not OK.

2020-08-20. Still not OK.

2020-08-22. OK, that's pretty bad.

2020-08-29. Moved this to a proper server box.

2020-09-09. He's dead, Jim.

2020-09-12. One restart later: it's back!

2020-09-22. More errors, more work for ZFS.

Expect this list to grow with time.

How I fixed one hardware issue with another one

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Tue, 11 Aug 2020 19:08:14 +0300

I happen to run on an ASRock Fatal1ty B450 Gaming-ITX/ac motherboard that used to run fine, but ever since I installed one UEFI update, it has had this one annoying issue: whenever you reboot, it gets stuck at POST. Cold boots work fine and the shutdown-and-push-the-power-button cycle also does the trick, but it’s not something I want to do all the time, especially when the machine is out of reach.

Before you say it: no, warranty service was not an option here, especially since this motherboard has suffered a fall once and it has this slight curvature in it. No, this separate incident was after this freezing issue started occurring.

Initial troubleshooting

I went through most of the standard hardware troubleshooting procedures:

clear CMOS
re-seat RAM
try a different CPU
disable features in UEFI, reduce PCIe speeds
disconnect storage devices

No luck, issue was still there. The POST code was stuck at A0. When taking a look at the ASRock Dr. Debug POST codes list, I saw that A0 falls under

Problem related to IDE or SATA devices. Please re-install IDE and SATA devices. If the problem still exists, please clear CMOS and try removing all SATA devices.

Since I had just done that procedure, and the issue was still there, I was stumped.

Wake-on-LAN

I had tried wake-on-LAN many years ago and was successful with it, so I thought that perhaps this could be an option. After all, I really did not want to get a new motherboard if I could just work around this issue.

I checked the ASRock UEFI for various wake-on-LAN related settings and found a couple of them. Oddly enough, the Wake on LAN setting itself only became available after I enabled booting from LAN in a completely unrelated UEFI section. In addition to the Wake on LAN setting, I also made sure to enable waking up from PS2 and PCIe devices, just in case.

After a quick Google search I found the Arch Linux Wiki page on this topic and saw that I needed to make sure that my network adapter was configured properly for wake-on-LAN to work.

[root@my-machine]# ethtool enp9s0 | grep Wake-on
	Supports Wake-on: pumbg
	Wake-on: g

Super, everything should work then!

I set up an always-on machine to send the wake-on-lan magic packets using wol every minute so that the machine eventually wakes up again. It’s configured, I shut down the problematic machine and wait.

Nothing happens.

Huh, perhaps I missed something.

No, everything was configured properly and the magic packet setting g was on by default.

I checked the wiki page again and two things caught my eye:

Note: Setting one of u, m or b along with g might also be necessary to enable the feature.

It is known that some motherboards are affected by a bug that can cause immediate or random wake-up after a shutdown whenever the BIOS WoL feature is enabled

Well, that’s interesting.

I added * * * * * ethtool -s enp9s0 wol pumbg to my crontab and shut my machine down.

Power supply does a click. Silence. Another click, ethernet port LED-s light up and my machine is powered on again. Success!

As it turns out, I did not even need to enable sending wake-on-LAN magic packets from another machine, the two motherboard issues managed to cancel each other out.

This page looks better in the app

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Sun, 02 Aug 2020 09:33:41 +0300

You’re a web developer at a social media company that has recently made a big push for modernizing their frontend for the mobile-first era. It has taken a lot of time and effort from many people. Countless challenges, arguments, testing.

The release is near. You’re probably a bit anxious. After all, the site is visited by millions of users every day. What if something breaks? Will the users like the redesign?

It’s live.

You navigate to the site on your phone.

Huh.

Okay, it’s recommending the app. That’s fine though, you still have the option of using reddit in the browser.

You refresh the page.

I think it looks fine.

I can’t imagine what the web developer who had to implement this was feeling.

To add insult to injury, this is what happens when you try to navigate to a subreddit on the mobile site:

I think they're trying to tell me something.

There is a workaround to this: navigate to the subreddit from Google and you will be fine, as long as you don’t navigate elsewhere.

Why even bother with developing the website if all it does is advertise the app everywhere?

The little Wi-Fi AP that could

ihavesomethoughtsonyourblog@ounapuu.ee (Herman Õunapuu) — Thu, 23 Jul 2020 01:17:30 +0300

I have a bad habit of testing things whenever a “good” idea pops into my head. This is a short overview of one of them.

The Orange Pi Zero is a SBC (single board computer) that has a slow 32-bit ARM 4 core CPU, 512MB of RAM and no display output. It’s actually quite OK for many tasks, such as reverse proxy (assuming 100Mbit/s is enough for you), low-performance NAS (assuming you are fine with 10-12MB/s file transfer speeds) or a Syncthing relay. Oh, and you can also turn it into a Wi-Fi access point.

The Wi-Fi access point in its natural habitat.

Why?

Because Armbian made it really easy to test out and I was interested in seeing what kind of performance an old USB Wi-Fi dongle could offer. This cheap AP could also come into handy in situations where my main access point dies for some reason.

How?

Go to the Armbian device page for your SBC and download the latest image, then write it to your microSD card using the tool of your choice, start your SBC and finish the initial setup.

After all that is done, start armbian-config as root. This tool allows you to do many things over a terminal UI, including a no-hassle method of setting up an Wi-Fi AP.

The option you are looking for is under Network -> Hotspot.

This image will likely be way out of date in a couple of years.

Follow the on-screen instructions to set everything up. After the initial setup is done, make sure to navigate to the same menu again to change the SSID and the Wi-Fi password because the defaults are horribly insecure.

After all this you should be good to go!

Issues and troubleshooting tips

It would not be a proper project if something didn’t go wrong in the process.

Before starting this configuration, run ip a to see if your Wi-Fi device is present. In my case the Orange Pi Zero also has onboard Wi-Fi, but that particular one has poor support, so I use an USB Wi-Fi dongle instead.

If your Wi-Fi device is not showing up, make sure that the correct firmware is present. In my case I had to install firmware-atheros package so that my ath9k_htc based Wi-Fi adapter would start working. dmesg and lsusb -v can help determine what chip your Wi-Fi adapter is using and what driver it needs.

The Orange Pi Zero that I have could set everything up, but it would crash hard with the logs not really showing much. After testing the USB Wi-Fi adapter in all the different USB ports with no success, I opted to use armbian-config menu System -> CPU to limit the CPU clock speed to 480MHz and set the CPU governor to performance to rule out power delivery issues that might be caused by variations in CPU power usage. That seems to have done the trick, because after that change the Wi-Fi AP has been rock solid.

Testing and conclusions

After setting it up and tweaking it a bit I started testing this thing out. The AP had no major issues throughout the day and did not crash.

The performance, however, isn’t anything to write home about. Speed maxes out at 30Mbit/s and is generally lower than that, especially when more than one client is connected. You can do things like Steam Remote Play (stream your game over your home network), but the quality will suffer. It was still quite fun to mess around in GTA V regardless of the image quality and during ~1h of play time there were only 4-5 stutters that were caused by the network.

I am speed.

Would I use this as my main Wi-Fi access point?

Definitely not. Any half-decent router will out-perform this setup.

Can it still be useful?

Definitely. A crappy Wi-Fi AP is better than no AP.

Parts list

Orange Pi Zero SBC (512MB RAM)
Orange Pi Zero expansion board (adds two USB 2.0 ports)
Cheap case for the Orange Pi Zero
32GB Sandisk microSD card
TP-Link WN-722N USB Wi-Fi adapter
Official Raspberry Pi microUSB power adapter

Update (2020-08-22)

It’s still working and is happily serving as a temporary Wi-Fi access point.

During additional testing I could also hit 40 Mbps speeds when testing in an environment with far fewer competing Wi-Fi access points nearby.

Update (2020-08-30)

It’s still running well and it seems that it won’t be a simple temporary solution after all.

Update (2020-11-30, approximately)

This setup has now been retired. It had 0 issues throughout its lifespan as a Wi-Fi access point, but I replaced it with a more standard Wi-Fi access point so that this board can be used somewhere else.

The setup did surprisingly well and even survived nightly automated updates and restarts, which can be attributed to the good work that the Armbian project has done.