Being chronically late to meetings sucks.

Not only is it very rude, but you’re signalling that you don’t value your coworkers’ time.

However, I’ve picked up a technique that works unreasonably well within a team.¹

If you are late to the first meeting of the day three times within a quarter, then you will have to make pancakes for the whole team.

Let’s say that you have a daily stand-up taking place at 10:00.

Arriving at 10:00:59: completely OK.

Arriving at 10:01:00: You’re one step closer to making pancakes!

Keep in mind that you may hit some obstacles when implementing this rule, so feel free to adjust it. When proposing this idea in my current team, I learned that the office does not offer pancake-making facilities. The pancakes can be substituted for other types of cake or bringing in something else, as long as the team gives prior approval of that modification.

The pancake strikes can also be pooled together and spent with your teammates if they wish to do so.

If you’re struggling with your team being late to your daily meeting(s), then go ahead and add this rule to the working agreement. You do have a working agreement set up, right? Right?

And a free security tech tip to close out: if you see an unlocked work laptop at the office, open your internal chat application of choice on it and try posting to a public channel that you’ll be bringing cake/beers/candy to the office. Works wonders for enforcing the habit of locking your laptop up when leaving the desk!

I use Uptime Kuma to check the availability of a few services that I run, with the most important one being my blog. It’s really nice.

Today I wanted to set it up on a different machine to help troubleshoot and confirm some latency issues that I’ve observed, and for that purpose I picked the cheapest ARM-based Hetzner Cloud VM hosted in Helsinki, Finland.

Hetzner provides a public IPv6 address for free, but you have to pay extra for an IPv4 address. I didn’t want to do that out of principle, so I went ahead and copied my Docker Compose definition over to the new server.

For some reason, Uptime Kuma would start up on the new IPv6-only VM, but it was unsuccessful in making requests to my services, which support both IPv4 and IPv6. The requests would time out and show up as “Pending” in the UI, and the service logs complained about not being able to deliver e-mails about the failures.

I confirmed IPv6 connectivity within the container by running docker exec -it uptime-kuma bash and running a few curl and ping commands with IPv6 flags, had no issues with those.

When I added a public IPv4 address to the container, everything started working again.

I fixed the issue by explicitly disabling the IPv4 network in the Docker Compose service definition, and that did the trick, Uptime Kuma made successful requests towards my services. It seems that the service defaults to IPv4 due to the internal Docker network giving it an IPv4 network to work with, and that causes issues when your machine doesn’t have any IPv4 network or public IPv4 address associated with it.

Here’s an example Docker Compose file:

name: uptime-kuma
services:
  uptime-kuma:
    container_name: uptime-kuma
    networks:
      - uptime-kuma
    ports:
      - 3001:3001
    volumes:
      - /path/to/your/storage:/app/data
    image: docker.io/louislam/uptime-kuma
    restart: always
networks:
  uptime-kuma:
    enable_ipv6: true
    enable_ipv4: false

That’s it!

If you’re interested in different ways to set up IPv6 networking in Docker, check out this overview that I wrote a while ago.

When you’re dealing with a particularly large service with a slow deployment pipeline (15-30 minutes), and a rollback delay of up to 10 minutes, you’re going to need feature toggles (some also call them feature flags) to turn those half-an-hour nerve-wrecking major incidents into a small whoopsie-daisy that you can fix in a few seconds.

Make a change, gate it behind a feature toggle, release, enable the feature toggle and monitor the impact. If there is an issue, you can immediately roll it back with one HTTP request (or database query ¹). If everything looks good, you can remove the usage of the feature toggle from your code and move on with other work.

Need to roll out the new feature gradually? Implement the feature toggle as a percentage and increase it as you go.

It’s really that simple, and you don’t have to pay 500 USD a month to get similar functionality from a service provider and make critical paths in your application depend on them.² As my teammate once said, our service is perfectly capable of breaking down on its own.

All you really need is one database table containing the keys and values for the feature toggles, and two HTTP endpoints, one to GET the current value of the feature toggle, and one to POST a new value for an existing one. New feature toggles will be introduced using tools like Flyway or Liquibase, and the same method can be used for also deleting them later on. You can also add convenience columns containing timestamps, such as created and modified, to track when these were introduced and when the last change was.

However, there are a few considerations to take into account when setting up such a system.

Feature toggles implemented as database table rows can work fantastically, but you should also monitor how often these get used. If you implement a feature toggle on a hot path in your service, then you can easily generate thousands of queries per second. A properly set up feature toggles system can sustain it without any issues on any competent database engine, but you should still try to monitor the impact and remove unused feature toggles as soon as possible.

For hot code paths (1000+ requests/second) you might be better off implementing feature toggles as application properties. There’s no call to the database and reading a static property is darn fast, but you lose out on the ability to update it while the application is running.

Alternatively, you can rely on the same database-based feature toggles system and keep a cached copy in-memory, while also refreshing it from time to time. Toggling won’t be as responsive as it will depend on the cache expiry time, but the reduced load on the database is often worth it.

If your service receives contributions from multiple teams, or you have very anxious product managers that fill your backlog faster than you can say “story points”, then it’s a good idea to also introduce expiration dates for your feature toggles, with ample warning time to properly remove them. Using this method, you can make sure that old feature toggles get properly removed as there is no better prioritization reason than a looming major incident. You don’t want them to stick around for years on end, that’s just wasteful and clutters up your codebase.

If your feature toggling needs are a bit more complicated, then you may need to invest more time in your DIY solution, or you can use one of the SaaS options if you really want to, just account for the added expense and reliance on yet another third party service.

At work, I help manage a business-critical monolith that handles thousands of requests per second during peak hours, and the simple approach has served us very well. All it took was one motivated developer and about a day to implement, document and communicate the solution to our stakeholders.

Skip the latter two steps, and you can be done within two hours, tops.

In November 2024, my blog was down for over 24 hours.

Here’s what I learned from this absolute clusterfuck of an incident.

Lead-up to the incident

I was browsing through photos on my Nextcloud instance. Everything was fine, until Nextcloud started generating preview images for older photos. This process is quite resource intensive, but generally manageable. However, this time the images were high quality photos in the 10-20 MB size range.

Nextcloud crunched through those, but ended up spawning so many processes that it ended up using all the available memory on my home server.

And thus, the server was down.

This could have been solved by a forced reboot. Things were complicated by the simple fact that I was 120 kilometers away from my server, and I had no IPMI-like device set up.

So I waited.

50 minutes later, I successfully logged in to my server over SSH again! The load averages were in the three-digit realm, but the system was mostly operational.

I thought that it would be a good idea to restart the server, since who knows what might’ve gone wrong while the server was handling the out-of-memory situation.

I reboot.

The server doesn’t seem to come back up. Fuck.

The downtime

The worst part of the downtime was that I was simply unable to immediately fix it due to being 120 kilometers away from the server.

My VPN connection back home was also hosted right there on the server, using this Docker image.

I eventually got around to fixing this issue the next day when I could finally get hands-on with the server, my trusty ThinkPad T430. I open the lid and am greeted with the console login screen. This means that the machine did boot.

I log in to the server over SSH and quickly open htop. My htop configuration shows metrics like systemd state, and it was showing 20+ failed services. This is very unusual.

lsblk and mount show that the storage is there. What was the issue?

Well, apparently the Docker daemon was not starting. I was searching for the error messages and ended up on this GitHub issue. I tried the fix, which involved deleting the Docker folder with all the containers and configuration, and restarted the daemon and containers. Everything is operational once again.

I then rebooted the server.

Everything is down again, with the same issue.

And thus began a 8+ hours long troubleshooting session that ran late into the night. 04:00-ish late, on a Monday.

I tried everything that I could come up with:

• used the btrfs Docker storage driver instead of the default overlay one
  • Docker is still broken after a reboot
• replaced everything with podman
  • I could not get podman to play well with my containers and IPv6 networking
• considered switching careers
  • tractors are surprisingly expensive!

I’m unable to put into words how frustrating this troubleshooting session was. The sleep deprivation, the lack of helpful information, the failed attempts at finding solutions. I’m usually quite calm and very rarely feel anger, but during these hours I felt enraged.

The root cause

The root cause will make more sense after you understand the storage setup I had at the time.

The storage on my server consisted of four 4 TB SSD-s, two were mounted inside the laptop, and the remaining two were connected via USB-SATA adapters. The filesystem in use was btrfs, both on the OS drive and the 4x 4TB storage pool. To avoid hitting the OS boot drive with unnecessary writes, I moved the Docker data root to a separate btrfs subvolume on the main storage pool.

What was the issue?

Apparently the Docker daemon on Fedora Server is able to start up before every filesystem was mounted. In this case, Docker daemon started up before the subvolume containing all the Docker images, containers and networks was mounted.

I tested out this theory by moving the Docker storage back to /var/lib/docker, which lives on the root filesystem, and after a reboot everything remained functional.

In the past, I ran a similar setup, but with the Docker storage on the SATA SSD-s that are mounted inside the laptop over a native SATA connection. With the addition of two USB-connected SSD-s, the mounting process took longer for the whole pool, which resulted in a race condition between the Docker daemon startup and the storage being mounted.

Fixing the root cause

The fix for Docker starting up before all of your storage is mounted is actually quite elegant.

The Docker service definition is contained in /etc/systemd/system/docker.service. You can override this configuration by creating a new directory at /etc/systemd/system/docker.service.d and dropping a file with the name override.conf in there with the following contents:

[Unit]
RequiresMountsFor=/containerstorage

The rest of the service definition remains the same and your customized configuration won’t be overwritten with a Docker version update. The RequiresMountsFor setting prevents the Docker service from starting up before that particular mount exists.

You can specify multiple mount points on the same line, separated by spaces.

[Unit]
RequiresMountsFor=/containerstorage /otherstorage /some/other/mountpoint

You can also specify the mount points over multiple lines if you prefer.

[Unit]
RequiresMountsFor=/containerstorage 
RequiresMountsFor=/otherstorage 
RequiresMountsFor=/some/other/mountpoint

If you’re using systemd unit files for controlling containers, then you can use the same systemd setting to prevent your containers from starting up before the storage that the container depends on is mounted.

Avoiding the out of memory incident

Nextcloud taking down my home server for 50 minutes was not the root cause, it only highlighted an issue that had been there for days at that point. That doesn’t mean that this area can’t be improved.

After this incident, every Docker Compose file that I use includes resource limits on all containers.

When defining the limits, I started with very conservative limits based on the average resource usage as observed from docker stats output.

Over the past few months I’ve had to continuously tweak the limits, especially the memory ones, due to the containers themselves running out of memory when the limits were set too low. Apparently software is getting increasingly more resource hungry.

An example Docker Compose file with resource limits looks like this:

name: nextcloud
services:
  nextcloud:
    container_name: nextcloud
    volumes:
      - /path/to/nextcloud/stuff:/data
    deploy:
      resources:
        limits:
          cpus: "4"
          memory: 2gb
    image: docker.io/nextcloud:latest
    restart: always
  nextcloud-db:
    container_name: nextcloud-db
    volumes:
      - /path/to/database:/var/lib/postgresql/data
    deploy:
      resources:
        limits:
          cpus: "4"
          memory: 2gb
    image: docker.io/postgres:16
    restart: always

In this example, each container is able to use up to 4 CPU cores and a maximum of 2 GB of memory. And just like that, Nextcloud is unable to take down my server by eating up all the available memory.

Yes, I’m aware of the Preview Generator Nextcloud app. I have it, but over multiple years of running Nextcloud, I have not found it to be very effective against the resource-hungry preview image generation happening during user interactions.

Decoupling my VPN solution from Docker

With this incident, it was also clear that running your gateway to your home network inside a container was a really stupid idea.

I’ve mitigated this issue by taking the WireGuard configuration generated by the container and moving it to the host. I also used this as an opportunity to get a to-do list item done and used this guide to add IPv6 support inside the virtual WireGuard network. I can now access IPv6 networks everywhere I go!

I briefly considered setting WireGuard up on my openWRT-powered router, but I decided against it as I’d like to own one computer that I don’t screw up with my configuration changes.

Closing thoughts

I have not yet faced an incident this severe, even at work. The impact wasn’t that big, I guess a hundred people were not able to read my blog, but the stress levels were off the charts for me during the troubleshooting process.

I’ve long advocated for self-hosting and running basic and boring solutions, with the main benefits being ease of maintenance, troubleshooting and low cost. This incident is a good reminder that even the most basic setups can have complicated issues associated with them.

At least I got it fixed and learned about a new systemd unit setting, which is nice.

Still better than handling Kubernetes issues.

I was in a pinch.

I needed to make a full disk backup of a PC, but I had no external storage device with me to store it on. The local Wi-Fi network was also way too slow to transfer the disk over it.

All I had was my laptop with an Ethernet port, a Fedora Linux USB stick, and a short Ethernet cable.

I took the following steps:

• boot the target machine up with the Fedora Linux installer in a live environment
• modify the SSH configuration on the target machine to allow root user login with a password
  • it’s OK to do this on a temporary setup like this one, but don’t do it on an actual Linux server
• set a password for the root user on the target machine
  • only required because a live environment usually does not set one for root user
• connect both laptops with the Ethernet cable
• set static IPv4 addresses on both machines using network settings ¹
  • edit the “Wired” connection and open the IPv4 tab
  • example IP address on target: 192.168.100.5
  • example IP address on my laptop: 192.168.100.1
  • make sure to set the netmask to 255.255.255.0 on both!
• verify that the SSH connection works to the target machine
• back up the disk to your local machine using ssh and dd
  • example: ssh root@192.168.100.5 "dd if=/dev/sda" | dd of=disk-image-backup.iso status=progress
  • replace /dev/sda with the correct drive name!

And just like that, I backed up the 120 GB SSD at gigabit speeds to my laptop.

I’ve used a similar approach in the past when switching between laptops by running a live environment on both machines and copying the disk over with dd bit by bit.

You’ll also save time on not having to copy the data over twice, first to an external storage device, and then to the target device.

Good news, everyone! Doing IPv6 networking stuff on Docker is actually good now!

I’ve recently started reworking my home server setup to be more IPv6 compatible, and as part of that I learned that during the summer of 2024 Docker shipped an update that eliminated a lot of the configuration and tweaking previously necessary to support IPv6.

There is no need to change the daemon configuration any longer, it just works on Docker Engine v27 and later.

Examples

If your host has a working IPv6 setup and you want to listen to port 80 on both IPv4 and IPv6, then you don’t have to do anything special. However, the container will only have an IPv4 address internally. You can verify it by listing all the Docker networks via sudo docker network ls and running sudo docker network inspect network-name-here for the one associated with your container.

For services like nginx that log the source IP address, this is problematic, as every incoming IPv6 request will be logged with the Docker network gateway IP address, such as 10.88.0.1.

name: nginx
services:
  nginx:
    container_name: nginx
    ports:
      - 80:80
    image: docker.io/library/nginx
    restart: always

If you want the container to have an IPv4 and an IPv6 address within the Docker network, you can create a new network and enable IPv6 in it.

name: nginx
services:
  nginx:
    container_name: nginx
    networks:
      - nginx-network
    ports:
      - 80:80
    image: docker.io/library/nginx
    restart: always
networks:
  nginx-network:
    enable_ipv6: true

There are situations where it’s handy to have a static IP address for a container within the Docker network. If you need help coming up with an unique local IPv6 address range, you can use this tool.

name: nginx
services:
  nginx:
    container_name: nginx
    networks:
      nginx-network
        ipv4_address: 10.69.42.5
        ipv6_address: fdec:cc68:5178::abba
    ports:
      - 80:80
    image: docker.io/library/nginx
    restart: always
networks:
  nginx-network:
    enable_ipv6: true
    ipam:
      driver: default
      config:
        - subnet: "10.69.42.0/24"
        - subnet: "fdec:cc68:5178::/64"

If you choose the host network driver, your container will operate within the same networking space as your container host. If the host handles both IPv4 and IPv6 networking, then your container will happily operate with both. However, due to reduced network isolation, this has some security implications that you must take into account.

name: nginx
services:
  nginx:
    container_name: nginx
    network_mode: host
    # ports are not relevant with host network mode
    image: docker.io/library/nginx
    restart: always

If you want your container to only accept connections on select interfaces, such as a Wireguard connection, then you will need to specify the IP addresses in the ports section. Here’s one example with both IPv4 and IPv6.

name: nginx
services:
  nginx:
    container_name: nginx
    networks:
      - nginx-network
    ports:
      - 10.69.42.5:80:80
      - "[fdec:cc68:5178::beef]:80:80"
    image: docker.io/library/nginx
    restart: always
networks:
  nginx-network:
    enable_ipv6: true

What about Podman?

I’ve given up on Podman. Before doing things the IPv6 way, Podman was functional for the most part, requiring a few tweaks to get things working.

I have not managed to get Podman to play fair with IPv6. No matter what I did, I could not get it to listen to certain ports and access my services, the ports would always be filtered out.

Conclusion

I’m genuinely happy to see that the IPv6 support has gotten better with Docker, and I hope that this short introduction helps those out there looking to do things the IPv6 way with containers.

Imagine that you have an unencrypted drive containing your private data and one day it starts throwing a bunch of errors. You have backups of the data so you’ve got that part covered, but would you feel comfortable sending the drive in to be warrantied? You have no control over who has access to that drive, and due to the drive failing you can’t format it as well.

Do you take the financial hit and buy a new drive, or send it in regardless and risk someone looking through your files?

I’ve bought and sold a bunch of hard drives and SSD-s over the years¹, and warrantied a few of them. I encrypt the disk on my personal and work machines because of the obvious security benefits, but for a long time I avoided doing the same for other storage devices.

Then I realized that I can just encrypt them and use almost anything as the key. All you need is a reasonably sized file to pass to cryptsetup as a key-file, refer to that key-file in /etc/crypttab, and you’re good to go!

Be creative! The key file can be anything:

• a dank meme
• a very short low resolution cat video
• an adorable photo of your dog
• a precious family photo
• Twilight fan fiction that you wrote back in 2011
• your letter of resignation at your last job

Just mind the size, cryptsetup wasn’t very happy with a 17MB full resolution image. A JPEG that’s less than a megabyte in size worked well enough for me.

On Linux, you can encrypt a partition with a cat picture using a command like this:

cryptsetup luksFormat /dev/sdx1 --key-file /path/to/cat.jpg

You can make sure that the drive gets automatically decrypted at boot by defining it in /etc/crypttab. Find the UUID of the encrypted partition by running ls -lah /dev/disk/by-uuid and see which one matches with /dev/sdx1. If the UUID is ef277bc2-d953-44c4-88af-8320aca76969, then a line in /etc/crypttab would look like this:

encryptedcatdrive UUID=ef277bc2-d953-44c4-88af-8320aca76969 /path/to/cat.jpg 

Once unlocked, your decrypted partition will be available as /dev/mapper/encryptedcatdrive.

Just make sure that the key file isn’t placed on the encrypted drive itself, otherwise you’ll lock yourself out for good.

For more information on LUKS disk encryption and its capabilities, see this handy Arch Wiki page.

This approach comes with some great benefits and a few downsides.

On the bright side, you don’t have to worry about a stranger getting their hands on your data when selling a storage device or sending it in to be warrantied, assuming that you haven’t posted the key file on social media or anywhere else.² Just do a quick format of the drive, or format a very specific part to clear the LUKS headers, and the contents of the drive are gone!

If you can’t format the drive due to hardware issues, then you should still be safe, the attacker would have to first fix the drive (which isn’t a guaranteed success) and then figure out the key, which will require a lot of effort and time.

The one obvious downside is that if you lose the key file, you lose the data. You can mitigate this by either adding multiple key files as a backup option, or setting a strong password that you can use in case you lose your original key file.

If the drive contains something of value to your family members, then it may be a good idea to specify those details in a will, or by taping the backup password with instructions on the drive itself. If you end up selling the drive before your personal expiration date arrives, then you can simply remove the instructions. Just make sure that your family members or relatives know to plug in the drive to a Linux machine.³

I also like the thrill of hiding a key in plain sight. Did I use the cover photo of this post to encrypt my drives? You’ll never know, and that’s what makes it fun!

If you’re paranoid or targeted by state actors, then you probably shouldn’t follow this advice, except the part where I encourage encrypting stuff.

For a few months, I had issues with my Wi-Fi network. The 2.4 GHz network would be fine, but the 5 GHz one would suddenly stop working and completely disappear from the available Wi-Fi networks. OpenWRT upgrades also didn’t improve the situation. This was very annoying.

After some discussions with a friend, I learned about Dynamic Frequency Selection (DFS). Apparently some channels on the 5 GHz Wi-Fi spectrum are also used by weather and military radars, and those take priority. If such interference is detected, your Wi-Fi access point should switch to a different channel.

It turns out that some implementations are buggy and mine is one of them.

One quick but permanent fix is to manually select a Wi-Fi channel to operate in.

With OpenWRT, you can get a list of all the available Wi-Fi channels using this command:

iw list | grep dBm

Example output:

* 2412 MHz [1] (20.0 dBm)
* 2417 MHz [2] (20.0 dBm)
* 2422 MHz [3] (20.0 dBm)
* 2427 MHz [4] (20.0 dBm)
* 2432 MHz [5] (20.0 dBm)
* 2437 MHz [6] (20.0 dBm)
* 2442 MHz [7] (20.0 dBm)
* 2447 MHz [8] (20.0 dBm)
* 2452 MHz [9] (20.0 dBm)
* 2457 MHz [10] (20.0 dBm)
* 2462 MHz [11] (20.0 dBm)
* 2467 MHz [12] (20.0 dBm)
* 2472 MHz [13] (20.0 dBm)
* 5180 MHz [36] (23.0 dBm)
* 5200 MHz [40] (23.0 dBm)
* 5220 MHz [44] (23.0 dBm)
* 5240 MHz [48] (23.0 dBm)
* 5260 MHz [52] (20.0 dBm) (radar detection)
* 5280 MHz [56] (20.0 dBm) (radar detection)
* 5300 MHz [60] (20.0 dBm) (radar detection)
* 5320 MHz [64] (20.0 dBm) (radar detection)
* 5500 MHz [100] (26.0 dBm) (radar detection)
* 5520 MHz [104] (26.0 dBm) (radar detection)
* 5540 MHz [108] (26.0 dBm) (radar detection)
* 5560 MHz [112] (26.0 dBm) (radar detection)
* 5580 MHz [116] (26.0 dBm) (radar detection)
* 5600 MHz [120] (26.0 dBm) (radar detection)
* 5620 MHz [124] (26.0 dBm) (radar detection)
* 5640 MHz [128] (26.0 dBm) (radar detection)
* 5660 MHz [132] (26.0 dBm) (radar detection)
* 5680 MHz [136] (26.0 dBm) (radar detection)
* 5700 MHz [140] (26.0 dBm) (radar detection)
* 5720 MHz [144] (13.0 dBm) (radar detection)
* 5745 MHz [149] (13.0 dBm)
* 5765 MHz [153] (13.0 dBm)
* 5785 MHz [157] (13.0 dBm)
* 5805 MHz [161] (13.0 dBm)
* 5825 MHz [165] (13.0 dBm)
* 5845 MHz [169] (13.0 dBm)
* 5865 MHz [173] (13.0 dBm)

Notice the ones with (radar detection) at the end? Those are the potentially problematic channels. We’re going to avoid them from now on by picking a specific channel to use.

When it comes to the choice of the channels themselves, you’ll also have to consider the channel width. If you pick a channel next to one of the radar detection ones and with a large channel width, you might still run into issues.

Choosing a specific channel also comes with bandwidth and range trade-offs. If you don’t care much for those, go for the lowest one and with 40 MHz width. Picking the optimal Wi-Fi channel and channel width configuration for your specific needs is better explained by other resources.

I recommend getting a Wi-Fi spectrum analysis app that shows you the channels that are least populated by neighboring Wi-Fi access points.

It is also possible to define the list of channels that the Wi-Fi AP can automatically choose from using the channels option for the wireless interface. We can use this setting to avoid the radar detection channels completely. This setting doesn’t seem to be configurable via the graphical interface (LuCI), but you can change it in /etc/config/wireless using the command line and vi, over SSH.

More information about this option and others can be found in OpenWRT documentation.

Based on our example, a configuration that avoids radar detection frequencies can look something like this:

config wifi-device 'radio0'
        option type 'mac80211'
        option path 'pci0000:00/0000:00:00.0'
        option channel 'auto'
        option channels '36 40 44 48 149 153 157 161 165 169 173'
        option band '5g'     
        option htmode 'VHT40'
        option country 'EE'    
        option cell_density '0'

Using a manually specified channel has resulted in no Wi-Fi related issues for over half a year.

I consider this a permanent fix.

If you’re using a PC and don’t want to mess with Wi-Fi issues ever again, then just run some Ethernet cables. It’s worth it.

The ThinkPad T430 has a few options for running it with an external display:

• VGA port, which is pretty much obsolete at this point
• mini DisplayPort connector on the laptop itself
• DVI or DisplayPort on a dock

The mini DisplayPort port has annoyed me for as long as I’ve had this machine.

Most places where I’ve had to present something only offer an HDMI cable, which means that I always have to carry a dongle around, and I keep forgetting to bring one everywhere I happen to go.

Until now.

I have a few of these SATA HDD adapters that replace the optical drive on the ThinkPad T430, and I discovered that my mini DisplayPort to HDMI adapter can fit in one without a problem.

I don’t currently have a need for a second SSD in my T430, so this “mod” makes perfect sense to me.

I bet there is someone out there who is capable of routing an actual HDMI port in place of this adapter. The existence of FHD mods for this laptop suggests that this is possible. If you’re that person and created that mod, let me know.

My current ISP provides an internet connection over a copper wire. To use it, I have a crappy modem (Technicolor CGA2121, DOCSIS 3.0). It’s running in bridge mode, meaning that all it does is convert the signal running over the coax cable into plain old Ethernet.

My main networking device is a TP-Link Archer C7 v5. It runs OpenWRT. This router/Wi-Fi AP box connects to the modem and handles everything, including getting a public IPv4 address from the ISP.¹

After a power outage or my ISP doing maintenance, the public IP address has usually changed. This wouldn’t be a problem if I just stuck to the ISP-approved modem.²

With my setup, there was a problem. The OpenWRT box would try to operate with the IPv4 address that it was given since the DHCP lease had not yet expired. However, this meant that there was no internet connectivity. A reboot of the OpenWRT box would resolve the issue.

This manual workaround wasn’t good enough for me. It would be quite problematic if this issue happened while I was away from home because I’d still like to access my home server.

After traversing OpenWRT forums and consulting the Slack workspace of my local hackerspace, I found that bringing up the WAN interface again would result in the OpenWRT box getting a new public IPv4 address. Problem solved!

To automate this workaround, I created a single crontab entry in the OpenWRT box. This is also configurable in a graphical user interface as long as you have LuCI installed.

The crontab entry looks like this:

*/5 * * * * /bin/ash -c '/bin/ping -c 3 8.8.8.8 > /dev/null || /sbin/ifup wan'

Every 5 minutes, the router pings Google’s DNS server. If that command succeeds, then the internet connection works and that’s it. If the ping fails, then the other half of the shell command is executed, which brings up the wan interface on my router.

Feel free to use a different IP address to test with. Your WAN network interface might also have a different name.

The downside of this solution is that if the server you’re using to verify your internet connection is down or refuses pings, then you’ll be causing interruptions in your home network every 5 minutes.

Talking to the ISP about this issue was something I considered as well. Then I remembered that it took me 1.5 months of fighting chatbots and repeating the same information to different customer care agents to use my own modem that’s identical to the one the ISP uses. That’s a hell no from me.

ThinkPad keyboards were once well known for their great layouts, feel and functionality. This included the media playback control keys.

On the ThinkPad T430, the new chiclet keyboard layout moved the media keys to the function row. Still there, but less convenient to access.

The ThinkPad L390 Yoga doesn’t have any visible function keys for controlling media playback. However, I found that the play/pause and stop buttons are still functional on the up/down arrow keys. Left/right arrow keys act as Home and End keys.

evtest output for play/pause key combination:

Event: time 1714716211.222697, type 4 (EV_MSC), code 4 (MSC_SCAN), value e3
Event: time 1714716211.222697, type 1 (EV_KEY), code 143 (KEY_WAKEUP), value 1
Event: time 1714716211.222697, -------------- SYN_REPORT ------------
Event: time 1714716212.291293, type 4 (EV_MSC), code 4 (MSC_SCAN), value a2
Event: time 1714716212.291293, type 1 (EV_KEY), code 164 (KEY_PLAYPAUSE), value 1
Event: time 1714716212.291293, -------------- SYN_REPORT ------------
Event: time 1714716212.366568, type 4 (EV_MSC), code 4 (MSC_SCAN), value a2
Event: time 1714716212.366568, type 1 (EV_KEY), code 164 (KEY_PLAYPAUSE), value 0
Event: time 1714716212.366568, -------------- SYN_REPORT ------------
Event: time 1714716214.026847, type 4 (EV_MSC), code 4 (MSC_SCAN), value e3
Event: time 1714716214.026847, type 1 (EV_KEY), code 143 (KEY_WAKEUP), value 0
Event: time 1714716214.026847, -------------- SYN_REPORT ------------

evtest output for stop key combination:

Event: time 1714716254.780584, type 4 (EV_MSC), code 4 (MSC_SCAN), value e3
Event: time 1714716254.780584, type 1 (EV_KEY), code 143 (KEY_WAKEUP), value 1
Event: time 1714716254.780584, -------------- SYN_REPORT ------------
Event: time 1714716255.614775, type 4 (EV_MSC), code 4 (MSC_SCAN), value a4
Event: time 1714716255.614775, type 1 (EV_KEY), code 166 (KEY_STOPCD), value 1
Event: time 1714716255.614775, -------------- SYN_REPORT ------------
Event: time 1714716255.658388, type 4 (EV_MSC), code 4 (MSC_SCAN), value a4
Event: time 1714716255.658388, type 1 (EV_KEY), code 166 (KEY_STOPCD), value 0
Event: time 1714716255.658388, -------------- SYN_REPORT ------------
Event: time 1714716256.961601, type 4 (EV_MSC), code 4 (MSC_SCAN), value e3
Event: time 1714716256.961601, type 1 (EV_KEY), code 143 (KEY_WAKEUP), value 0
Event: time 1714716256.961601, -------------- SYN_REPORT ------------

On a very recent laptop, the ThinkPad P14s gen4, the media playback keys seem to be gone. Left/right arrow keys do still work as Home and End keys, though.

If you also miss your media playback control keys on your ThinkPad, then hit that Fn key and give it a go, you might get lucky!

Your service cannot process events fast enough during peak hours.

There is no obvious quick and dirty fix.

Refactoring would take ages.

People have been unhappy for a while now.

What the hell do you do?

Background

I had the pleasure of working with a legacy backend system recently. It had plenty of ongoing problems, but one of those was more acute compared to others: the service could not process certain very important events fast enough during peak hours and that was problematic for everyone that relied on those events.

The processing of events was very basic:

• load 1000 entities
• for each entity, do the following sequentially
  • do some processing
  • send a JMS message via ActiveMQ
  • wait for confirmation

The processing was triggered by the scheduled jobs system that the service had been relying on since its inception more than a decade ago. That system only allowed to run one instance of a scheduled job at any time.

This processing worked well for almost a decade at this point, but started causing more and more issues as the service was the backbone of a rapidly-growing business. At least it was a nice problem to have.

I was the lucky guy who got assigned to solving this issue, and in hindsight I’m really glad I got it because tackling this one was fun.

The slowness of the process was down to the part where events had to be sent via ActiveMQ. It’s possible to send events without waiting for any confirmation, but our service was going with the slower approach of waiting for a confirmation that the message was successfully sent. After consulting with my colleagues, going through the depths of git blame and doing some Jira archeology I soon learned that this slowness was there for a reason: some events could fail to be processed if the service suddenly died, which wasn’t that rare of an occurrence. Sure, the processing was quick, but you had the risk of losing events, and that was even worse than sending events with a big delay.

When looking at the behaviour of this code path I learned that sending each event took about 20 milliseconds. That is not a lot of time for a single message, but if you have queued up 1000 events, then that results in 20+ seconds required to send all of those messages. Take into account the fact that you are doing this processing sequentially and the fact that during peak hours you had to process thousands of events within one minute, and you can see where this becomes problematic.

To give some additional context: we’re dealing with a service that has at least 2 or more instances running at any time in a Kubernetes cluster. We had other high priority work ongoing as well and refactoring this part of the system was not feasible in a short time window. Spending weeks or even months on this issue was out of the question.

The solution

The first idea I tried was pretty basic: try to see if we could process each event in parallel using Java parallel streams. Hibernate and ActiveMQ put brakes on that idea pretty quick due to objects related to them not being thread-safe.

The second idea I tried was to find a way to bulk-send events since I suspected that the ActiveMQ connection setup and teardown was being done separately for each processed entity. That, however, was not the case.

The third idea was the one I went with. From my early programming days I learned about the modulo operation. I also had vague knowledge of Kafka and its way to split work via partitions. Didn’t take my mind long to connect the dots, and a TechTipsy certified quickfix™ was born.

I took the existing job, created multiple instances of it and gave each instance a number from 0 to 4, let’s call it a “worker ID”. Those modified scheduled jobs ran the same database query, but with a small adjustment: in addition to other criteria, each scheduled job only picked entities where the modulo 5 result of its numerical identifier matched the worker ID.

This means that the service could process 5 times more entries at once without having to commit to a big and risky rewrite.

The PostgreSQL modulo operator helped facilitate this process and was the key to avoiding loading all of the entities into memory and filtering that list down using the modulo operator in application code.

All-in-all, this took a few days of work to implement, roll out in staging and production, observing and refactoring to make the solution maintainable. For a service that’s part of the critical path in the tech stack and notorious for its slow deployment cycle, it was pretty fast.

Example

Here’s an example to help illustrate the process. The database holds 7 entities with the following ID-s: 1001, 1002, 1003, 1004, 1005, 1006, 1007.

Worker 0 starts up and selects all entities where the result of ID mod 5 = 0.

• 1001 mod 5 = 1
• 1002 mod 5 = 2
• 1003 mod 5 = 3
• 1004 mod 5 = 4
• 1005 mod 5 = 0 <– this one
• 1006 mod 5 = 1
• 1007 mod 5 = 2

Worker 0 selects entity 1005.

Worker 1 selects entities 1001, 1006.

Worker 2 selects entities 1002, 1007.

Worker 3 selects entity 1003.

Worker 4 selects entity 1004.

All the workers operate in parallel.

Caveats

This solution isn’t perfect and has some caveats related to certain implementation details.

Adding additional workers is relatively simple, but requires lots of small but well-documented steps. This will only be a problem if the system sees additional growth that it cannot handle, but that should be years from now. I like to think of it as job security for future generations.

Due to the way the scheduled jobs solution is built, it is possible for one instance of the service to run more than one worker at once, which could be a problem for compute-heavy or memory-hungry processing work. However, for this use case it’s not a problem.

All-in-all, I’m happy with the solution, the team was happy after I made the TechTipsy certified quickfix™ more maintainable, and everyone relying on this solution to work properly were also happy.

I hate buying things that are single-purpose, which is why I ended up with this setup.

Take a speaker, a battery, put them together, and what you now have is a portable speaker. Since I had access to both, I felt no need to buy a separate portable speaker for use in social events.

The fact that this JBL soundbar supports Bluetooth was what sealed the deal. Just make sure to keep the setup a fair distance away from any bigger bonfires, liquids and bugs, and you should be good to go.

This is a quick story about a fun discovery that I made a while ago.

For legal reasons, all of this is made up and no such service ever existed.

Once upon a time, we had this Java service that handled all the backend work that you’d expect to occur for a product with a web interface. The service wasn’t the newest thing on the block and had seen dozens of developers work on it over many years.

At one point, it was moved to a Kubernetes cluster. That meant configuring all the bits and pieces. In YAML, of course.

A couple of years ago I joined the project and started work on it. Things were constantly on fire, so the fact that the service took anywhere from 5 to 15 minutes to deploy to the staging environment wasn’t something I focused on. After pushing a commit, the team either took a coffee break, went to play some table tennis or started work on something else while the pipeline did its job.

It didn’t help that running the service locally was hand-waved away as something that was too impractical to do by people who had been in the project longer than me, so the team simply relied on local tests and checking the staging environment after they pushed a change.

A year or so later, I had enough. The project wasn’t also constantly on fire, only occasionally, so I decided to take the time to dig into our CI pipeline configuration to see what is causing the pipeline to be so slow. My investigation lead to Kubernetes configuration, specifically the part where resource limits were configured. For whatever reason, I found that the service was allowed an absurdly low CPU allocation, measured in millicores.

I increased the limit to something sensible, such as 1 full CPU core. The result? The startup time of each pod took 30-40 seconds now, resulting in deployments that took 2 to 3 minutes max. This is an insane improvement over the old deployments that took up to 15 minutes regularly.

To go even faster. I tweaked the maxSurge property of the rolling update deployment strategy to start up more new pods in parallel, further shortening the time that it took to deploy the service. The only thing I had to keep in mind was the number of database connections afforded to each pod and the maximum number of connections offered by the database. Start up too many pods in parallel, and you’ll find that your deployment fails due to the service exhausting the available database connections.

Some time later I learned the reason behind such a CPU resource limit configuration. Apparently it’s a good practice to set your resource limits based on the average load that your service exhibits after it has properly started up. It does make sense, especially if you don’t want to have your Kubernetes nodes sit idle due to inefficient resource usage.

This example case shows that it’s a trade-off that you’ll have to take into consideration, especially if your service starts up quite slowly and not in mere seconds.

At the time of writing this post, the tweaked configuration is still up and running in production.

Anyone that has bought themselves external WD drives from the Elements/My Book/Easystore series are probably familiar with the acoustic characteristics of the drives. The drives have a loud hum caused by WD running the drives at 7200rpm while claiming the drives to be “5400rpm-class” and the clacking of the read-write heads is audible as well. In a small space, such as an apartment, the hum is maddening, especially when you have more than one drive running at the same time.

After running such a setup for months, enough was enough. I bought some sound dampening foam and used that to try to limit the noise that my setup was making. However, that didn’t do much and the before/after noise comparisons didn’t have much of a difference. As a result of this testing, I did have various pieces of acoustic foam left, and after I noticed that my Lenovo M73 Tiny PC had a similar shape to the WD My Book 12 TB hard drives that I got recently, I had an idea.

I used four pieces of foam and placed them so that I could stack the hard drives on top. This resulted in the humming noise being completely eliminated. The sound of hard drive read-write heads is still there, but it is so much less audible now. After this change, I have started hearing the cooling fan more than the hard drives themselves.

This setup could probably be improved by switching around the setup so that the hard drives were at the bottom with the bottom drive having additional foam or rubber feet below it to support it better. For the time being, I’m perfectly happy with this arrangement, as long as it doesn’t fall over.

2021-04-23 update

I have now added a fan to the setup!

The fan itself is powered by one of the USB ports with the help of a spare sacrificial USB cable and a Noctua omni-join kit that I had left over.

The drive temperatures are now reading around 42C and 46C.