Networking has long been my Achilles heel. I know the very basics, but the more complex areas of networking have been a bit puzzling to me. By the time I figured out how IPv4 works, I found IPv6 and that my ISP supports it.

Back to square one.

That didn’t stop me from learning some bits, and after 8+ years of self-hosting as a hobby, I’ve settled on a setup that works for me and overcomes common residential internet connection nuances, such as dynamic IPv4 addresses and changing IPv6 prefixes. I’m sharing these tips and tricks with the goal of helping out other hobbyists out there that happen to share a similar stack.

Background

My ISP is polite enough to provide a public IPv4 address, and allowing incoming traffic is a toggle in their online self-service. Not perfect, but at least you can do it. However, they charge about 6 EUR a month for the static IP address service, which I am not willing to pay out of principle.

They also support IPv6, which is great, and they provide you a whole /56 slice of it to play with using IPv6 prefix delegation. Unfortunately they have configured the lease time for the prefix to be incredibly short: 26 minutes!

A router reboot or short power outage usually results in the IPv4 address and IPv6 prefix changing, which is really annoying as my services become unavailable for a short time.

Dynamic DNS

A common way to overcome the dynamic IP address limitation is to sign up with a provider to set up a DNS record that changes whenever your home IP address changes. My domain registrar does not have this as a feature, and I’m not interested in using a different provider, so I went in a different direction and built a home-grown script that does the same thing.

Initially, this script relied on a public service that tells you what your IP address is, and based on that I could check if things have changed and I need to update my DNS record. This approach has one glaring catastrophic failure mode though: that provider could lie to you one day and now you’ve pointed your DNS records at the attackers’ servers. :)

I ignored that failure mode for a while, but once I learned about the effectiveness of LLM-based tooling, I decided to give it a go and to build a better solution that takes into account my setup and requirements, while at the same time saving me from the frustration of troubleshooting and debugging this in a late evening. I’m still very limited on available free time, so optimizing for that is a priority for me.

My main networking gear runs OpenWRT, and it supports running shell scripts periodically in a crontab. The router has two WAN interfaces, one for IPv4 and one for IPv6. It already knows what IP address and prefix have been assigned to it, so I don’t have to rely on an external service provider for finding this out.

Handling IPv4 addresses is simple: check the IPv4 address of the WAN interface. Query your existing DNS records, diff it, and if it has changed, push an update in a separate API call. Super simple!

With IPv6, the approach is slightly different. Instead of the WAN interface, I have to get the IPv6 address of the target machine, and make sure that it’s routable over the public internet. When you’ve checked your network settings in an IPv6 network, you may have noticed a lot of different IP addresses there, with lots of letters thrown into the mix.

Here’s an example from the machine that is serving you this blog (likely out of date though!):

    inet6 fdb3:6dad:6dce::f41/128 scope global dynamic noprefixroute 
    inet6 fdb3:6dad:6dce:0:2e0:4cff:fe0c:9ddb/64 scope global noprefixroute 
    inet6 2001:7d0:856c:4000::f41/128 scope global dynamic noprefixroute 
    inet6 2001:7d0:856c:4000:2e0:4cff:fe0c:9ddb/64 scope global dynamic noprefixroute 
    inet6 fe80::2e0:4cff:fe0c:9ddb/64 scope link noprefixroute 

The two relevant ones are the ones that start with 2001:, others are link-local or accessible over the local network only. The shorter one consists of the IPv6 prefix part, and then the unique bit at the end is a predictable suffix that the host gets. The other one also works, but is as far as I understand randomly generated and more difficult to predict when we get around to next sections.

I know that there is probably a better way to do this, but I wanted to keep things simple enough so that I can troubleshoot them if needed. It may be possible to trigger this updater script on events that WAN and WAN6 interfaces send, but I have not validated this theory.

There are many different ways to find the IPv6 address of a particular host, so the script I have just tries multiple approaches to find the one that we’re looking for.

Here’s the script in case you’re interested in setting up something similar. It reads credentials from an .env file and is built around the Zone API. On OpenWRT, the only dependency that you need to install is curl, which to my surprise was not part of the default packages list, probably to save on space.

One lesson I learned from a previous iteration of the script: if you trigger DNS record updates every minute, then Zone will actually reach out to you via e-mail telling you to cut that shit out, politely. It was just one missing if statement, and yet it caused some frustration to engineers far away. Sorry!

Predictable IP addresses

It’s a good idea to set up static IP addresses for your hosts, both for IPv4 addresses and IPv6 prefix delegation via DUID-s.

The OpenWRT GUI LuCI makes it quite simple, just set the addresses as static on the landing page for the hosts that you are interested in forwarding traffic to, and you’re done!

My recommendation here is to also set a predictable IPv6 suffix, otherwise all your IPv6 traffic rules may break once again due to this nuance.

I like to make that host number the same for both IPv4 and IPv6, quick example:

• 192.168.1.2
• 2001:7d0:854f:8e00::2

Here’s a configuration snippet example from /etc/config/dhcp, look for the hostid option:

config host
        option name 'mycoolserver'
        option ip '192.168.1.69'
        list mac '12:34:56:78:90:AB'
        option duid 'yourduidgoeshere'
        option hostid '69'

Apply with service dnsmasq restart.

In LuCI, as of OpenWRT 25.12, look for “IPv6 token”.

Note that due to a bug, it doesn’t seem to be possible to set a numeric IPv6 token via GUI, which is why you will need to add it manually in CLI using the above approach.

Port forwards, traffic rules, potato, potahtoh

Whenever you want to make a local machine accessible on the internet for IPv4, the solution is simple: set up a port forward to that particular machine, and you’re done! It’s a common enough flow for people who’ve set up game servers and the like, and well understood by more novice users.

With IPv6, port forwards don’t help. You’ll have to check one tab over at “Traffic rules” in OpenWRT GUI.

It’s a common misconception that by using IPv6 you are exposing everything to the world as each device gets its own IPv6 address, but turns out that this is not the case in most common setups. By default, OpenWRT forwards only a few types of traffic to IPv6 hosts, such as ICMP packets that make ping work between devices over IPv6 across the public internet. If you are interested in allowing IPv6 clients to access services on your local server that has an IPv6 address, you will have to explicitly allow it by adding a new traffic rule.

There’s one issue with this approach that a lot of users seem to run into: if the IPv6 prefix changes, then all my traffic rules that are pointing to a particular host are automatically broken!

Luckily there is a clever workaround implemented on OpenWRT that bypasses this issue. Assuming that you followed the previous step and set yourself up with a predictable IPv6 suffix, when setting up a traffic rule, set the target device up as ::69/-64, just replace 69 with your actual suffix. The IPv6 prefix can now change, but the ports that you’ve made accessible on this specific host will remain working.

At this point, you should be all set with a reasonably well working setup where you’ve handled the issues with dynamic IPv4 and IPv6 prefix, and you can access your services over the public internet even when things happen.

Limitations

One issue that this setup has is the fact that DNS change propagation takes time. Usually clients will pick up the new records within 5 minutes, but in my professional career I’ve seen some clients take up to 24 hours or longer to finally start sending traffic to the new DNS record. Whenever your IP address changes, there will be a mini-outage. Not catastrophic if you’re just hosting hobby projects and personal services at home, but I wouldn’t host anything mission-critical in such a setup.

When your OpenWRT device is as underpowered as mine, then you may notice that the TLS encryption overhead when curl -ing around can be significant. I have set my dynamic DNS script to run every 5 minutes, and it shows up on the CPU usage graphs on my router.

Wireguard all the things!

I know that Tailscale is a popular method of connecting up your devices and making your personal services privately accessible over that, which significantly reduces your attack surface. Being behind a few updates or not being vigilant enough is less of an issue compared to exposing your services over the public internet.

You don’t necessarily need Tailscale for that though! If you just need a way to access your services over a private and secure network, then setting up a dedicated mini PC or single-board computer is a very good starting point. Let it be the server, allow traffic to move between the clients over the Wireguard interface, and you’re all set!

Alternatively, if you have an OpenWRT router, then you can do it right there, but I found the GUI management setup to be a bit clunky compared to deploying the plain configuration files to clients. When I did do that test, I discovered quickly that my router and its single ARM CPU core with no cryptography extensions is too slow for managing my Wireguard network, with speeds topping out at 20 Mbit/s. 20. The LattePanda IOTA can easily saturate its gigabit link, as does the ThinkPad T430, and even devices like the Orange Pi Zero can handle a theoretical maximum of about 240 Mbit/s over Wireguard measured using wg-bench.

My current Wireguard host is the LattePanda V1, the most unstable computer in my fleet. With a USB adapter, it can push almost half a gigabit of traffic over Wireguard.

If you’re like me, and you like hosting your services over Docker or Podman, then instead of listening on ports for all interfaces on your containers (default behaviour when setting up port forwards), I recommend listening only on the Wireguard interface. This makes the service only accessible over Wireguard, meaning that you only need to set up one port forward and traffic rule to connect to the Wireguard network, and then you have access to all of your services. The attack surface is significantly reduced, the whole Wireguard solution is stable and very small, and unless you leak your private key, you are reasonably secure!

Here’s a snippet from a compose file showcasing how to set this up for IPv4 and IPv6:

ports:
  - 10.69.69.12:2283:2283
  - "[fded:abba:acca::12]:2283:2283"

Want to make the service available over Wireguard and over the local network directly? Just add those to the list! Note that if your local address changes and you don’t update it in the compose file, your container will refuse to start up as it cannot listen to the interface any longer, but you can mitigate that with the static IP addresses step.

When you are going with this route, it is unlikely but still possible that by the time the container starts up, the Wireguard interface is not yet up. To resolve this, you can use systemd to set Wireguard up as a dependency that you will have to wait for before the container starts up.

I manage my Wireguard connection with wg-quick@interfacename service. You can set up a systemd override for Docker, or if you manage your Docker/Podman services via systemd, then you can set it up per-service using this pattern:

# /etc/systemd/system/myimmichserver.service.d/override.conf
[Unit]
Requires=wg-quick@interfacename.service
After=wg-quick@interfacename.service

By the way, systemd overrides are also really useful for ensuring that your storage that your containers rely on is properly mounted. If my service requires the path /immich to be available and mounted, add something like this:

BindsTo=immich.mount
After=immich.mount

If you unmount the mount point, it will also properly bring down the service. The service won’t start if the mount point is missing. I’ve had the issue with containers seeing blank mount points more times than I’d like to admit, and this has eliminated this issue for me.

systemd has received a lot of hate online, and I don’t think it’s fair. The ease with which you can set up dependencies on your system, set up resource limits, make services more restricted to improve the security posture is great and allows me to and avoid all sorts of failure modes. Production services that I’m responsible for make use of these systemd features, with great results.

For services that need to be public, such as Nextcloud and its public shareable links, this approach won’t work, obviously, but for things that only you and your family members use, this is a viable approach.

Conclusion

This setup works well enough for me to confidently host my blog and self-hosted services off of it. I’ve hit a lot of paper cuts and frustrations along the way, but after following this guide, you don’t have to do the same.

Yes, VLAN-s are intentionally missing from this guide. I’ll get to them eventually, maybe by Q4 2037 given my lack of free time.

And no, IPv6 isn’t complicated, it’s just different from what everyone is used to. If we started out with IPv6 right from the get-go, we wouldn’t be having dumb arguments online.

Good news, everyone! Doing IPv6 networking stuff on Docker is actually good now!

I’ve recently started reworking my home server setup to be more IPv6 compatible, and as part of that I learned that during the summer of 2024 Docker shipped an update that eliminated a lot of the configuration and tweaking previously necessary to support IPv6.

There is no need to change the daemon configuration any longer, it just works on Docker Engine v27 and later.

Examples

If your host has a working IPv6 setup and you want to listen to port 80 on both IPv4 and IPv6, then you don’t have to do anything special. However, the container will only have an IPv4 address internally. You can verify it by listing all the Docker networks via sudo docker network ls and running sudo docker network inspect network-name-here for the one associated with your container.

For services like nginx that log the source IP address, this is problematic, as every incoming IPv6 request will be logged with the Docker network gateway IP address, such as 10.88.0.1.

name: nginx
services:
  nginx:
    container_name: nginx
    ports:
      - 80:80
    image: docker.io/library/nginx
    restart: always

If you want the container to have an IPv4 and an IPv6 address within the Docker network, you can create a new network and enable IPv6 in it.

name: nginx
services:
  nginx:
    container_name: nginx
    networks:
      - nginx-network
    ports:
      - 80:80
    image: docker.io/library/nginx
    restart: always
networks:
  nginx-network:
    enable_ipv6: true

There are situations where it’s handy to have a static IP address for a container within the Docker network. If you need help coming up with an unique local IPv6 address range, you can use this tool.

name: nginx
services:
  nginx:
    container_name: nginx
    networks:
      nginx-network
        ipv4_address: 10.69.42.5
        ipv6_address: fdec:cc68:5178::abba
    ports:
      - 80:80
    image: docker.io/library/nginx
    restart: always
networks:
  nginx-network:
    enable_ipv6: true
    ipam:
      driver: default
      config:
        - subnet: "10.69.42.0/24"
        - subnet: "fdec:cc68:5178::/64"

If you choose the host network driver, your container will operate within the same networking space as your container host. If the host handles both IPv4 and IPv6 networking, then your container will happily operate with both. However, due to reduced network isolation, this has some security implications that you must take into account.

name: nginx
services:
  nginx:
    container_name: nginx
    network_mode: host
    # ports are not relevant with host network mode
    image: docker.io/library/nginx
    restart: always

If you want your container to only accept connections on select interfaces, such as a Wireguard connection, then you will need to specify the IP addresses in the ports section. Here’s one example with both IPv4 and IPv6.

name: nginx
services:
  nginx:
    container_name: nginx
    networks:
      - nginx-network
    ports:
      - 10.69.42.5:80:80
      - "[fdec:cc68:5178::beef]:80:80"
    image: docker.io/library/nginx
    restart: always
networks:
  nginx-network:
    enable_ipv6: true

What about Podman?

I’ve given up on Podman. Before doing things the IPv6 way, Podman was functional for the most part, requiring a few tweaks to get things working.

I have not managed to get Podman to play fair with IPv6. No matter what I did, I could not get it to listen to certain ports and access my services, the ports would always be filtered out.

Conclusion

I’m genuinely happy to see that the IPv6 support has gotten better with Docker, and I hope that this short introduction helps those out there looking to do things the IPv6 way with containers.

Just like most people out there, I have some files that are irreplaceable, such as cat pictures.

At one point I had a few single-board computers sitting idle, namely the Orange Pi Zero and the LattePanda V1, and a few 1TB SSD-s.

I hate idle hardware, so I did the most sensible thing and assembled a fleet of networked offsite backups for backing up the most important data.

My setup is based on various flavors of Linux, but the ideas will likely translate well onto other operating systems and solutions.

Networking

The most important part is the networking. The offsite backup endpoints connect together to my home server over a WireGuard network. The home server is, well, the server, and backup endpoints are clients.

I like this WireGuard Docker image a lot because it generates the server and client configurations automatically, but you can use plain WireGuard or a completely different networking solution to connect all the devices together. Some use Tailscale to make the setup process easier, but I like to keep things as self-hosted as possible.

I’m not a networking expert, but here’s how I’ve set up my network. For this example, the WireGuard network operates in the 10.13.69.0/24 range.¹

To only allow traffic between the devices and avoid tunneling everything through the home server, set the AllowedIPs setting to AllowedIPs = 10.13.69.0/24,10.13.69.1. We want to be able to access the backup endpoints, and nothing more.

All the devices have a static IP address in that network, such as 10.13.69.1 for the home server, 10.13.69.2 for a backup endpoint and so on.

The PersistentKeepalive = 25 option is present in the client configurations so that I don’t lose the ability to access the backup endpoints. With it, all the backup endpoints call back to the home server from time to time. The aforementioned Docker image automatically adds it to the generated configuration using the PERSISTENTKEEPALIVE_PEERS=all option. This setting is crucial. Without it, I sometimes ran into problems trying to connect from my home server to the backup endpoint, and that’s something you can’t easily alleviate without having physical access to the backup endpoints, which are offsite.

Remove the DNS configuration from generated WireGuard client configurations, as you don’t need it for this purpose.

Optionally, edit the /etc/hosts file for the home server and backup endpoints so that you can access your backup endpoints using simple hostnames, like orangepizero. Example row can look like this: 10.13.69.6 orangepizero.

if your WireGuard server operates in a network with a dynamic external IP address, as is common with many home internet connections, I recommend getting yourself a domain name that you can update whenever your IP address changes and using that in your WireGuard client configurations. Without this, an IP address change will result in your backup endpoints being inaccessible.

You’ll also likely need to set up port forwarding and/or traffic rules for your backup endpoints to be able to connect back to your WireGuard server.

Once you have the WireGuard connection set up and SSH running on the backup endpoints, you should be able to drop the backup endpoints into any network that you have permission for. Ask your friends and family, and sweeten the deal by offering free technical support or help in some other area in return. The cost of running a single-board computer 24/7 is minuscule with the typical power consumption being 1-3W, so that won’t be much of a concern.

Making backups

For making the actual backups themselves, you have all sorts of options.

I rely on rsync to copy the data over. It’s simple and it works, that’s all I expect from it. Example command: rsync -aAXvz /folder/to/back/up/ backupuser@backupendpoint:/backup/ --delete. The files will be compressed during transit with the -z option, and with --delete you’ll ensure that the target folder has all the files from the source, and nothing else.

The backup storage is specified in /etc/fstab with the nofail option present. This ensures that in case the disk dies, the backup endpoint will still boot properly, allowing me to access the machine to troubleshoot the issue and/or force a desperate reboot to try to fix things. A good alternative approach is to mount/unmount the remote disk manually as part of the backup script.

The backup storage uses the btrfs filesystem, and I use btrbk to take snapshots of the contents. If I accidentally delete all the files on the backup endpoint, then I can still recover from that situation because the data is still present in snapshots. 30 days is a good retention period: enough time to save the data in case of an accidental deletion, but short enough to avoid the backup disk getting full.

If you don’t want to use filesystem-level snapshots, then tools like restic are a good alternative. It can also operate over SSH and you can configure snapshot retention policies in your backup script. Just make sure to not lose the encryption password, and verify the backups once in a while.

Deployment

I manage my backup endpoints using some cobbled-together Ansible roles. I’ve perfected it to the point where the only manual steps are flashing the OS and setting up the storage, the rest is handled via Ansible.

I’d like to share my work here, but it will make Jeff Geerling cry. Maybe one day I’ll take the time to improve things…

Maintenance

All the backup endpoints update and reboot themselves regularly. It’s just the sensible thing to do.

Every 6-12 months I also do major OS version updates. It’s risky because of the whole offsite aspect of the solution, but so far I haven’t been burned yet.

Monitoring

Monitoring is an area where I have some room for improvement. So far, I’ve set up Prometheus node-exporter to all of the backup endpoints, and my home server keeps track of how the backup endpoints are doing.

This allows me to check once in a while if any of the backup endpoints has fallen off the network, or if the backup disk is getting full.

Issues I’ve faced

I’ve had this system running for a few years now, and it’s mostly stable! There have been some issues I’ve faced as well, though. Some are very specific to certain hardware, but I think there’s value in mentioning them.

I once blew up a backup server because of an Ansible configuration issue. That meant that I had to physically go pick up the server to re-image it.

The Orange Pi Zero was running quite hot, resulting in stability issues, so I put together a really janky cooling solution.

Hell, I did the same for the LattePanda as well.

It might look horrific but the extra cooling has fixed all the stability problems on both boards.

The lack of a real-time clock on the LattePanda has required me to make its backup script a bit special. I can’t rely on a systemd timer that automatically reboots the machine once in a while, so instead that part is present in the backup script. The issue is that the LattePanda boots up with the time being set in the past, and once it gets the actual time from the network, it will run all sorts of tasks because enough time has passed! This included the reboot timer as well.

At one point, the power supply on the LattePanda just died, and it was very visible on my graphs. That required a replacement.

Conclusion

That’s how I back up the most important data. I hope that this has given you inspiration to take your own backup approach to the next level!

I like Fedora Linux.

It’s the Linux distro that stopped my habit of distro-hopping. Big deal? Ooh, big deal!

It’s stable in day-to-day use, ships with up-to-date software, and the software selection is adequate out of the box.

It also ships with a fresh Linux kernel version¹, resulting in a good experience even on modern hardware.

I’ve been happily using it for about 4 years at this point, both on personal and work computers.

However, every time I recommend it to my friends, family and colleagues, I have to mention a few things that a new Fedora Linux user might want to do first so that the experience is as smooth as possible.

If it weren’t for those aspects, Fedora would be pretty much perfect.

Set up RPM Fusion

There are some packages that are not available on Fedora by default, likely as a result of copyright laws, lawyers and Red Hat (IBM) not having the appetite to fight it out in the courts.

This is why RPM Fusion exists:

RPM Fusion provides software that the Fedora Project or Red Hat doesn’t want to ship.

The installation instructions are here, and I personally prefer this one-liner setup:

sudo dnf install https://mirrors.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm https://mirrors.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm

This step is a prerequisite to the next one.

Enable hardware accelerated video playback

This is the part where I blame lawyers.

To have a good chance at smoothly playing back videos in a browser or using VLC, you have to do a few extra steps, because someone fucked up.

Check this guide for up-to-date information.

Switch to the full version of ffmpeg:

• sudo dnf swap ffmpeg-free ffmpeg --allowerasing

AMD GPU users:

• sudo dnf swap mesa-va-drivers mesa-va-drivers-freeworld
• sudo dnf swap mesa-vdpau-drivers mesa-vdpau-drivers-freeworld

Intel GPU users (Broadwell and newer, 2014+):

• sudo dnf install intel-media-driver

Old Intel GPU users (Ivy Bridge and older, <=2013):

• sudo dnf install libva-intel-driver

NVIDIA GPU users: good luck.²

You can verify the enabled profiles using vainfo.

Here’s the output on my machine before completing the steps (AMD GPU).

Note the lack of H.264 and HEVC profiles.

me@mymachine:~$ vainfo
Trying display: wayland
libva info: VA-API version 1.21.0
libva info: Trying to open /usr/lib64/dri-nonfree/radeonsi_drv_video.so
libva info: Trying to open /usr/lib64/dri-freeworld/radeonsi_drv_video.so
libva info: Trying to open /usr/lib64/dri/radeonsi_drv_video.so
libva info: Found init function __vaDriverInit_1_21
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.21 (libva 2.21.0)
vainfo: Driver version: Mesa Gallium driver 24.1.7 for AMD Radeon 780M (radeonsi, gfx1103_r1, LLVM 18.1.6, DRM 3.57, 6.10.8-200.fc40.x86_64)
vainfo: Supported profile and entrypoints
      VAProfileJPEGBaseline           :	VAEntrypointVLD
      VAProfileVP9Profile0            :	VAEntrypointVLD
      VAProfileVP9Profile2            :	VAEntrypointVLD
      VAProfileAV1Profile0            :	VAEntrypointVLD
      VAProfileAV1Profile0            :	VAEntrypointEncSlice
      VAProfileNone                   :	VAEntrypointVideoProc

Here’s what it looks like after the tweaks are applied.

me@mymachine:~$ vainfo
Trying display: wayland
libva info: VA-API version 1.21.0
libva info: Trying to open /usr/lib64/dri-nonfree/radeonsi_drv_video.so
libva info: Trying to open /usr/lib64/dri-freeworld/radeonsi_drv_video.so
libva info: Trying to open /usr/lib64/dri/radeonsi_drv_video.so
libva info: Found init function __vaDriverInit_1_21
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.21 (libva 2.21.0)
vainfo: Driver version: Mesa Gallium driver 24.1.7 for AMD Radeon 780M (radeonsi, gfx1103_r1, LLVM 18.1.6, DRM 3.57, 6.10.8-200.fc40.x86_64)
vainfo: Supported profile and entrypoints
      VAProfileH264ConstrainedBaseline:	VAEntrypointVLD
      VAProfileH264ConstrainedBaseline:	VAEntrypointEncSlice
      VAProfileH264Main               :	VAEntrypointVLD
      VAProfileH264Main               :	VAEntrypointEncSlice
      VAProfileH264High               :	VAEntrypointVLD
      VAProfileH264High               :	VAEntrypointEncSlice
      VAProfileHEVCMain               :	VAEntrypointVLD
      VAProfileHEVCMain               :	VAEntrypointEncSlice
      VAProfileHEVCMain10             :	VAEntrypointVLD
      VAProfileHEVCMain10             :	VAEntrypointEncSlice
      VAProfileJPEGBaseline           :	VAEntrypointVLD
      VAProfileVP9Profile0            :	VAEntrypointVLD
      VAProfileVP9Profile2            :	VAEntrypointVLD
      VAProfileAV1Profile0            :	VAEntrypointVLD
      VAProfileAV1Profile0            :	VAEntrypointEncSlice
      VAProfileNone                   :	VAEntrypointVideoProc

Enable Flathub

This step is only necessary if you did not enable third-party repositories during the inital setup of your Fedora Linux installation.

Setting up Flathub is just one command:

flatpak remote-add --if-not-exists flathub https://dl.flathub.org/repo/flathub.flatpakrepo

Check the Flathub setup instructions for Fedora for up-to-date info.

🇪🇪 Specific to Estonia: ID card support

Estonia has an ID Card system (Eesti ID kaart), and the desktop application is also officially supported on Linux. Ubuntu Linux.

To use the ID card in Fedora, you’ll need to install the necessary packages:

sudo dnf install -y open-eid

Firefox users will need to make sure that the Web eID extension is installed. If it wasn’t installed with the above command, you can also get it here.

The desktop application used for opening and digitally signing documents, DigiDoc4, has a bad habit of breaking sometimes due to changes. Luckily there exists an unofficial version of DigiDoc4 as a Flatpak and it’s available on Flathub.

It’s mentioned in the official open-eid/linux-installer repo as well, which lends some credibility (plus I know the person that built it).

As of 2024-10-14, ID card works on Firefox and the DigiDoc4 client from the Fedora official repository does not crash and burn.

GNOME: fix the “Alt+Tab” behaviour

GNOME is a great desktop environment, but it, too, has its quirks.³

If you’re used to “Alt+Tab” keyboard shortcut simply switching windows, then you’ll have to tweak it in settings.

• Open “Settings”
• Navigate to “Keyboard”
• Click on “View and Customize shortcuts”
• Use the search bar to:
  • disable “Switch applications” shortcut
  • set the “Switch windows” shortcut to “Alt+Tab”

Here’s what it looks like on GNOME 46.

GNOME: minimize, maximize buttons on windows

By default, you’ll only see the “Close” button on windows, but it is possible to also show minimize and maximize buttons.

Install GNOME tweaks with sudo dnf install -y gnome-tweaks.

Open it, navigate to Windows, and toggle Maximize and Minimize settings.

GNOME: app status indicators

Some apps, such as Solaar, like to run in the background and show you a status indicator on your taskbar. This functionality is not present on GNOME by default, but you can easily fix it with the AppIndicator and KStatusNotifierItem Support extension.

GNOME: corner tiling

As of GNOME 46, you cannot do corner tiling out of the box. The best you can do is split windows horizontally, which is fine for single ultra-wide monitor setups. There are situations where corner tiling comes in really handy.

The best solution I have found so far seems to be the Tiling Assistant extension. It’s configurable and will probably suit your workflow well.

At the time of writing, it does have a few small but annoying enough bugs that are a show-stopper for me, but compared to other solutions this one at least works.

Optional: disable SELinux

There are two types of people: those that swear by SELinux, and those that swear due to SELinux.

If you’re running into weird issues and you don’t know why, then see if SELinux is to blame.

I run into issues with SELinux every time I try to move files around with rsync, resulting in a simple rsync -aAXv source/ destination/ command becoming rsync -aAXv --filter='-x security.selinux' source/ destination/. It’s frustrating.

To fix this, I simply disable SELinux. Open up /etc/selinux/config, find the line SELINUX=enforcing and change it to SELINUX=disabled.

You’ll need to restart your machine for it to take effect.

Don’t use ZFS

ZFS is a great filesystem, but not on distros that ship new kernel versions.

This is due to OpenZFS needing to be built against the Linux kernel, and new kernel releases often break compatibility. Those breakages may lead to your ZFS pools not being imported at boot as a result.

You can use ZFS if you use LTS kernels built by someone else, or if you simply don’t update your system, but I don’t recommend either approach.

Use a different distribution, such as Debian, if you really need to use ZFS daily.

If you don’t care for ZFS but like data integrity checks, snapshots and don’t use a lot of disks in an array, then btrfs is totally fine.

Don’t use NVIDIA GPU-s

I’ve never had a good experience with the open source driver nouveau, and the proprietary drivers are a headache and break things.

This might be about to change, but I’ll believe it when I see it.

Conclusion

I hope that this has made your Fedora Linux experience just a little bit better.

Here’s to hoping that IBM does not fuck up Fedora Linux. My expectations are low.

If you have any other tips to share then please contact me! You can find the contact details below.

I helped a family member upgrade to a newer iPhone and make some room so that the internal storage does not run out.

They had Nextcloud installed on the current phone, but due to limitations of the Nextcloud iOS app, the backups only take place if the app itself is open, meaning that we had hundreds of photos and videos that were not yet backed up.¹

The network was slow and I was in a time crunch, so I opted to copy the media off of the phone by connecting the iPhone to my laptop over a Lightning cable. After entering the PIN on the phone again and approving the connection, I could see the iPhone in my file manager, but opening it would show nothing at all. This was very unusual as other iPhones I used in the past just worked on my laptop.

Rebooting the phone and connecting it again did nothing. Different known good Lightning cables did not improve things, either. Even a separate Windows 10 machine could not see any images, so it wasn’t a Linux thing.

Various Apple-related posts online were devoid of useful advice.²

I was not ready to give up just yet and I finally stumbled upon this Ask Ubuntu answer. This had potential, so I gave it a go, and I’m happy to report that it worked on my Fedora Linux machine!

These are the steps that I took. The commands need to be run in a terminal window.

Tested with Fedora Linux 40.

• install the necessary packages
  • on Fedora, run sudo dnf install -y ifuse libimobiledevice libimobiledevice-utils
• connect your iPhone to your PC
  • if your phone is not yet paired, then run idevicepair pair
  • if it’s already paired, verify the connection by running idevicepair validate
• mount your phone to an empty folder of your choice
  • if an empty folder does not exist, run mkdir ~/my-iphone (use any folder name that you want)
  • to mount the phone, run ifuse ~/my-iphone
• you can now browse your media via your file manager or terminal by navigating to ~/my-iphone
  • photos and videos taken by the camera will be in the DCIM folder
  • other folders, such as Photos and Downloads may also interest you
  • to make a full copy of your phone to a folder named iphone-backup, run cp -r ~/my-iphone ~/iphone-backup
  • alternative way to create a full backup: rsync -aAXv ~/my-iphone ~/iphone-backup
• once you’re done, unmount the iPhone by running umount ~/my-iphone
• it is now safe to disconnect your iPhone from the PC

If you have copied everything over and want to make more space on your phone using this method, then you need to delete files under the DCIM folder and the cached thumbnails under the PhotoData folder. iOS seems to generate thumbnails for the photos you take and even if you delete all the actual photos off of the device, then the photo thumbnails will still show up in the Photos app and elsewhere. This confused the hell out of me after I deleted the backed up photos off the phone the first time.

Alternatively, you can just delete files from the Photos app on the phone itself to avoid this nuance.

Linux has its flaws, but at least it provides you with the power and tools needed to get you out of tricky situations.

A family member has a Canon PIXMA MP250 printer, originally released in 2009. It has been a very reliable piece of hardware, especially for a printer.

Then came Windows 10. The printer would not work out of the box with it and the official drivers got stuck during installation. Fiddling with the printer in device manager, trying to install drivers via Windows Update and stars aligning got the printer to work again.

Then came Windows 11. Nothing I did could get it working now, and the printer is not even officially supported by Canon on this version of Windows.

This printer works out of the box on any mainstream Linux distribution, and the family member had been using the printer with Ubuntu for years, but a certain piece of important software in Estonia was not working properly a lot of the time, which triggered a switch to Windows.¹

Quick searches online suggested that I could turn this printer into a network printer, as long as I had some patience and a spare computer.

The idea is simple: take a spare computer, hook it up to the printer, install CUPS on it, configure it and you’re good to go!

I initially tested this setup with a Zimaboard running Fedora Server, but the final solution used an old Raspberry Pi B+ running the latest version of Raspberry Pi OS.

How to set it up

The steps will be similar with other pieces of hardware and Linux distributions, you may just need to adjust a few commands and package names.

The steps are roughly based on these two guides:

• Setting up Network Printer with Raspberry Pi and CUPS
• Raspberry Pi Print Server: Setup a Network Printer

If you’re following along with a Raspberry Pi, set up your OS first. A minimal installation with no graphical interface will be just fine.

Next, install CUPS and optionally the right printer drivers for your printer. The Canon PIXMA MP250 requires the package printer-driver-gutenprint.

sudo apt install -y cups printer-driver-gutenprint

Make sure that CUPS is running.

sudo systemctl enable --now cups.service

Next, allow your user to manage printing. In this example, the username is pi.

sudo usermod -a -G lpadmin pi

Next, make sure that CUPS allows connections from other machines in your network, and then restart it.

sudo cupsctl --remote-any
sudo systemctl restart cups

Find out the IP address of the Pi and navigate to it in your browser. It will look something like https://192.168.0.49:631. The browser will likely warn you about a self-signed certificate. It’s safe to proceed.

If you haven’t already connected your printer to the Raspberry Pi via USB, then do so now.

Use the graphical interface to add a new printer under the Administration section. You’ll need to enter the username and password for your user here. It’s the same user we used earlier, so in this example it’s pi.

If you can’t find your printer model listed in the Add printer view, then you are probably missing the proper driver package, or your printer simply isn’t supported.

After adding your printer, you can use the CUPS web interface to print a test page.

If that worked, then you should now be able to use any other device on the network to print!

On Linux and Windows, open the settings section that handles printing, and add a new printer. This networked printer should appear automagically, no additional configuration required.

I also did testing using a Fairphone 5 running Android 13, and an iPhone X running iOS 16. In both cases the printing just worked.

I’ve used the scanning functionality of this particular printer on Linux and it has worked, however I forgot to test this out using this setup. I’ll try to test that out once I get the chance to try the printer out again.

Hardware tech tips

I first ran this setup with a Zimaboard to test the idea out. It worked very well, but as a simple print server it is a bit overkill.

I then replaced it with a Raspberry Pi B+, and while the printing works, I discovered that a single ARM CPU core is apparently a limiting factor during printing. A simple test page print pegged the CPU at 100%, and that’s with the 700->1000 MHz overclock applied.

If you don’t mind a slower printing speed, then any Raspberry Pi will be absolutely OK. More performant versions of Raspberry Pi will likely fare even better.

If you already have a small home server running, then you may want to hook the printer up to that and set up the printer there. It’s already running 24/7, so it would be a perfect match.

One useful aspect of this setup is that you can queue up print jobs even when the printer itself is powered off. Once you turn it on, the print jobs will start, assuming that you have enough paper and ink.

Free open source software prolongs the useful life of hardware once again. See y’all in October 2025!

2024-06-15 update

This post got featured on:

• XDA Developers
• hackster.io
• Hackaday
• Hacker News

And even the leader of OpenPrinting, Till Kamppeter, chimed in! They also held a talk about OpenPrinting at FOSDEM 2024.

Regarding scanning, seems like there’s a separate project for that called SANE - Scanner Access Now Easy.

Find out more about OpenPrinting here.

2024-11-14 update

A reader shared with me the p910nd project that can be used for a similar purpose. I have not yet tried this myself.

There’s even an OpenWRT page that describes its use on a router!

This post illustrates something that is not common on modern laptops: changing the CPU.

Background

One of the modifications that you can do to a ThinkPad T430 laptop is changing the CPU to a more powerful quad core model. That’s exactly what I did back in 2017, but due to the poor availability of compatible CPU-s in my region I went for one with a 45W TDP, which is a bit much for a laptop designed for CPU-s with a 35W TDP.

You could work around this issue with software, and that’s what I also did for a long time. Since I now own two T430-s and the availability of 35W parts is better now, I decided to go for one to see how well these two options compare (result: it doesn’t overheat as often, but it managed to do it once anyway).

Unlike my T430, most modern laptops come with a soldered CPU. There do exist laptops that come with a socketed CPU, but those are not that common and are designed for very specific use cases.

Soldering the CPU right to the board has its benefits, such as the machine being thinner, and CPU-s don’t really die all that often. However, what we have lost out on is the option to upgrade older machines with more powerful components, extending the useful lifetime of the device.

Yes, you can replace soldered CPU-s, but that requires a skillset and equipment that few people have.

A couple of years ago, I worked on a ThinkPad T60. That ThinkPad had been in use by a family member before that, and I started its cleanup by disassembling the whole machine and making sure that it was pristine. However, once I put it all back together, I saw that it was password protected. I wasn’t even able to access the BIOS. After repeatedly asking for hints for the password and trying some simple cases myself, I never cracked it.

Not too long ago I got curious and looked for a way to overcome this issue. That’s when I stumbled upon this SuperUser post about resetting the supervisor password on a ThinkPad.

Once my to-do list was down to a reasonable size, I decided to finally give this a go, since it didn’t seem that difficult. Here’s an illustrated guide on how I approached this task.

Sources that are relevant to this topic:

• How to remove the Supervisor BIOS password for an IBM ThinkPad

• www.ja.axxs.net

• “How to unlock BIOS Supervisor Password from Lenovo Thinkpad Laptop (no damage to laptop)” by Arpan Dubey

Setup

What you’ll need to pull this off:

• your ThinkPad T60 laptop that has a supervisor password enabled
• a screwdriver
• anything that conducts electricity and can be used to short two pins
• a steady hand
• a magnifier, if your eyesight isn’t that great

To begin the process, you need to remove the palm rest of the ThinkPad. Follow the service manual if needed. Removing the keyboard screws is optional, but it might help you access the correct pins better.

The process

The steps for performing the supervisor password clearing:

• turn on the laptop
• when you see the ThinkPad logo, short two specific pins
• keep shorting those pins!
• press F1 to go to the BIOS
• navigate to the section in BIOS where you can set the Supervisor Password
• enter the supervisor password setting prompt and enter a blank password in the first field
• stop shorting the pins, and then enter an empty password to the second field
• save settings and restart
• after rebooting, you should be able to boot without having to enter a password

What follows is an illustrated guide of the process.

This is the step where I messed up initially. The contact between the pins was not good and I got prompted with the supervisor password prompt. Power the machine off and try again if you face the same issue.

Once the pins are shorted, you should be able to access the BIOS with a little bit of F1 key spamming.

After saving the settings and rebooting, you should be greeted with a successful boot.

Once you’ve confirmed that everything’s OK, assemble the machine again and enjoy the laptop that you just saved from becoming e-waste.

Caveats

This guide might not be terribly relevant in 2022. I still remember starting my software development career in 2016 with a ThinkPad T60 and it was painful even back then. These laptops are classics, but they are really starting to show their age. I’ve even encountered issues like the WiFi chip causing lots of trouble, with the connection being very spotty and borderline unusable.

But hey, at least it’s not completely unusable any more!

I’ve always been a fan of tinkering with cooling setups on my computers. I’ve even went as far as writing crappy solutions to make up for deficiencies on the hardware level. After years of dumb experiments I’ve seen how little you can get away with in cooling and how to run your machines as quietly as possible without giving up too much performance.

I also appreciate simplicity, which is why I’m currently running the simplest damn solution you can imagine to control how my ThinkPad T430 and ASRock DeskMini X300 run: a shell script.

It also comes with some added benefits:

• self-documenting
• highly customizable
• low resource usage

These scripts are not “battle-hardened” and using them incorrectly may or may not result in hardware failure. Use at your own risk. The only guarantee I can give is that it works on my machine™.

If you’re someone who doesn’t like tinkering with their computers and wants their machines to “just work”, then this article likely isn’t for you.

ThinkPad T430

With the ThinkPad T430, the script does two things:

• control the fan speed
• preemptively throttle the CPU and GPU

This type of behaviour is optimized for mostly quiet operation. The fan is running at its lowest speed when under a small load, and the CPU/GPU throttling has an added benefit of reducing the overall power usage, too. The difference between non-turbo and turbo speeds on the Intel i7-3820QM can be as much as 13-15 watts.

To enable fan control on ThinkPads, you need to have a file at /usr/lib/modprobe.d/thinkpad_acpi.conf with the following contents:

options thinkpad_acpi fan_control=1

Reboot, and you can now have full control over the fan speed in your ThinkPad!

The script itself looks like this:

#!/bin/bash

set -e

while true; do

  temp=$(cat /sys/class/thermal/thermal_zone1/temp)

  if ((temp > 90000)); then
    echo level 7 >/proc/acpi/ibm/fan
  elif ((temp > 75000)); then
    echo level 5 >/proc/acpi/ibm/fan
  elif ((temp > 60000)); then
    echo level 3 >/proc/acpi/ibm/fan
  elif ((temp > 30000)); then
    echo level 1 >/proc/acpi/ibm/fan
  else
    echo level 0 >/proc/acpi/ibm/fan
  fi

  if ((temp > 97000)); then
    echo 350 >/sys/class/drm/card*/gt_max_freq_mhz
    echo 350 >/sys/class/drm/card*/gt_boost_freq_mhz
  elif ((temp > 90000)); then
    echo 650 >/sys/class/drm/card*/gt_max_freq_mhz
    echo 650 >/sys/class/drm/card*/gt_boost_freq_mhz
  else
    echo 1250 >/sys/class/drm/card*/gt_max_freq_mhz
    echo 1250 >/sys/class/drm/card*/gt_boost_freq_mhz
  fi

  if ((temp > 70000)); then
    echo 1 >/sys/devices/system/cpu/intel_pstate/no_turbo
  else
    echo 0 >/sys/devices/system/cpu/intel_pstate/no_turbo
  fi

  sleep 0.5
done

What we’re doing is reading the temperature of the CPU package and then setting the fan speed and throttling based on that. Note that the temperatures are represented without any decimal places. A reading of 78000 means that the chip is running at 78 °C.

On my ThinkPad T430, the fan speed levels can be roughly described as such:

• level 1: lowest speed, barely audible
• level 3: audible, but tolerable
• level 5 and above: it’s probably loud enough to bother you

The second section controls the integrated GPU. What I’ve found in my testing is that a weak integrated GPU can still chug up to 20 watts, which is a considerable amount, especially on a laptop. The card* wildcard is there because the integrated GPU may change between card0 and card1, and I’m too lazy to make the script any smarter.

The values written there correspond to the maximum GPU clock speed. 1250 happens to be the max clock speed (in MHz) and at that speed the GPU is using a lot of power. 650 is the speed at which the integrated GPU is running most efficiently. 350 is the lowest speed, and you will notice it due to the performance being extra crappy.

With the CPU, I chose to just turn the turbo boost on or off. With turbo boost off, the CPU will top out at 2.7 GHz and around 22 watts of power usage. With turbo boost, the CPU can run anywhere between 3.4-3.7 GHz, depending on the number of cores under load. You’ll likely notice much higher CPU temperatures and power usage climbing to around 35 watts. Fast at short and bursty workloads, and yet efficient and relatively cool at sustained loads.

And at the very end, you have a simple sleep statement. I’ve set it to half a second, but feel free to set it as you see fit. Higher intervals might not be that good of an idea because the script may not be able to respond quickly enough to changing thermals, just keep that in mind.

ASRock DeskMini X300

Although I’ve opted for a more experimental setup with my home server and the ASRock DeskMini X300 is no longer running as one, I think it’s still worthwhile to share how I configured the fan control logic on this machine.

With the ASRock DeskMini X300, my goal was simple: let it run as quietly as reasonably possible without losing too much performance.

This machine was running 24/7 as a server and has a powerful 8-core AMD Ryzen 7 5700G CPU in it. One thing this script is heavily relying on is the fact that Ryzen CPU-s have well-engineered boosting logic in them, meaning that they’re already designed to run at the highest speed possible and only limit their speed if they hit power or thermal limits.

One thing you may need to do first is installing lm-sensors and running sudo sensors-detect. By default the CPU fan speed controls were not properly exposed, but after running sensors-detect and saying yes at every prompt, they were picked up.

The script looks like this:

#!/bin/bash

set -e

# /sys/devices/platform/nct6775.656/hwmon/hwmon3

echo 1 >/sys/devices/platform/nct6775.656/hwmon/hwmon3/pwm2_enable

while true; do

  temp=$(cat /sys/class/hwmon/hwmon4/temp1_input)

  if ((temp > 94000)); then
    echo 170 >/sys/devices/platform/nct6775.656/hwmon/hwmon3/pwm2
  elif ((temp > 90000)); then
    echo 140 >/sys/devices/platform/nct6775.656/hwmon/hwmon3/pwm2
  elif ((temp > 70000)); then
    echo 120 >/sys/devices/platform/nct6775.656/hwmon/hwmon3/pwm2
  elif ((temp > 40000)); then
    echo 30 >/sys/devices/platform/nct6775.656/hwmon/hwmon3/pwm2
  else
    echo 0 >/sys/devices/platform/nct6775.656/hwmon/hwmon3/pwm2
  fi

  sleep 5
done

The hwmon values and which fan you’re controlling is likely different on your own PC. Take a look at the paths found in the script and poke around while also physically looking at your PC. Try enabling PWM control by writing to pwm*_enable handles and then setting the fan speed. As a result of your poking, you should see the CPU fan change its speed. If it doesn’t, try another value.

The values you can write are in the range 0-255, where the higher value corresponds to a higher speed.

I’ve limited the TDP of my CPU to 35 watts using a setting found in UEFI settings. This results in the CPU running cool and quiet even under a full load.

systemd service

To run your fan control script as a systemd service, drop the script to your preferred location (I placed it in /root/.local/bin/fancontrol), then create a file /etc/systemd/system/fancontrol.service with the following contents:

[Unit]
Description=Quick fan control software

[Service]
ExecStart=/root/.local/bin/fancontrol

[Install]
WantedBy=multi-user.target

And once that is done, enable and start your fan control service with sudo systemctl enable --now fancontrol.service.

If you encounter issues or something isn’t working right, check the status of the service with systemctl status fancontrol.service or full logs with journalctl -u fancontrol.service.

Conclusion

If you want something that works out of the box and does the smart stuff for you, then give this Arch Linux wiki page a look. If you’re not into that and want to easily tune the behaviour of your machine, then feel free to use these scripts as a baseline for your own experimentation.

As some of you might own, I’m still rocking a ThinkPad T430, a laptop model originally released in 2012. It’s not the fastest laptop out there, but it is plenty fast for a number of tasks, even most software development work.

I also try to keep my machines up to date and properly backed up. On my laptop, this means having around a couple of scheduled tasks, such as restic backups. They don’t really take up a lot of time, but if an intensive scheduled task starts while you’re having a video call over Google Meet while also sharing the screen, things can get a bit slow, ruining the experience.

After a couple of these instances, I started thinking about finding a solution.

Scheduled tasks

Here’s a quick rundown of regular tasks that I like to run on my laptop.

Backups using restic: hourly backups to my home server. The file scanning process of the backup is quite intense, eating up a lot of my CPU time.

System updates using dnf and flatpak: usually run daily, because I’m too lazy to install the updates using a GUI, and at the same time I cannot afford to run out of date software.

Backups of critical files from my server to my laptop: there are some files I just cannot afford to lose. For this reason, I make a daily backup of the files to my laptop.

Filesystem maintenance: I don’t trust my storage devices (and neither should you), which is why I run a weekly scrub on my SSD that’s running the btrfs filesystem. If there are errors detected, I will know as soon as possible.

These tasks can be quite intensive, especially on hardware that’s a bit older.

With some tasks, such as system updates, you may also run into random issues. Updating Mozilla Firefox while it’s actively running results in the browser requiring a restart immediately after opening a new tab. This is very annoying and inconvenient when you’re in the middle of your work.

The “time-to-sleep” script

To avoid intensive scheduled tasks running while I’m doing actual work, I tried rescheduling these tasks to a more quiet time. All I needed to do was to edit the systemd timers so that the jobs start after midnight and leave the laptop running.

This solution has quite an obvious downside: my laptop will be powered on pretty much 24/7, and since electricity is super expensive and laptop maintenance a bit of a pain, I’d be wasting resources, time and eventually money.

I could make the last scheduled task make the machine shut down, but what if I have a long-running file copy operation running? It would be interrupted, unless I specifically disabled the scheduled task.

Then it clicked for me.

I know when I’m planning on going to sleep. Sleeping also has a regular cycle as it happens daily. Why not just put all the scheduled daily tasks into one script that I can run whenever I intend to shut down my laptop?

And thus, the time-to-sleep script was born. It’s a simple Bash script that lives in /root/.local/bin/time-to-sleep. To make sure I can also reasonably observe the behaviour of the script, I’ve implemented it as a simple systemd service that lives in /etc/systemd/system/time-to-sleep.service. It’s disabled by default, but can be started manually.

[Unit]
Description=Run maintenance tasks

[Service]
ExecStart=/root/.local/bin/time-to-sleep

I have an alias in ~/.bashrc that I use to start the service.

alias time-to-sleep="sudo systemctl start time-to-sleep.service"

Whenever the time comes to go to sleep, or when I’m just going to do something else and can leave the laptop running at my desk, I just open a terminal window, type time-to-sleep and I’m good to go. If I’m somewhere with a very poor internet connection or need to conserve battery power, I can opt to not run that script.

Going with this approach also saves me a lot of trouble on the technical side as well. I don’t need to think of a magical way to automatically detect if I’m working at the moment, or if I’m running on battery, or if I’m behind a limited internet connection. Sometimes your job becomes so much easier if you make some assumptions about when and where the script runs. You’re in control of choosing when to run it, after all.

In case something goes wrong, I can rely on the set -e line in my script to fail hard. When I come back to my machine, it will be quite noticeable that something has gone wrong, as the laptop will still be running. Who needs a fancy monitoring setup when you have visual evidence of failure?

As for the hourly backups, I solved that problem by only making backups of more important files. It takes much fewer resources compared to the full /home folder backup that runs as part of the time-to-sleep script.

Conclusion

After all this work, I don’t have to worry about any scheduled tasks hogging up all the resources while working. At the same time, I still get all the benefits that the scheduled tasks bring.

Example

In case you need it: here’s an example of the script that I’m running, with all the personal details scrubbed out. Feel free to use it as inspiration for your own time-to-sleep script!

#!/bin/bash

set -e

echo "Restic full home folder backup."
export RESTIC_REPOSITORY='sftp:backupuser@backupserver.lan:/path/to/repo/'
export RESTIC_PASSWORD='someverysecurepasswordgoeshere'

restic unlock

restic backup --verbose --exclude-caches --cleanup-cache \
  --iexclude=/home/*/downloads \
  --exclude=/home/*/.cache \
  --exclude=/home/*/.gradle \
  --exclude=/home/*/.local/share/Trash \
  /home

restic forget --prune \
  --keep-hourly 0 \
  --keep-daily 7 \
  --keep-monthly 0

echo "Restic full home folder backup done."

echo "Backing up important data from the server."

rsync -az --delete-before \
  user@myserver.lan:/path/to/nextcloud/ \
  /storage/backups/nextcloud/

chown -R localuser:localuser /storage/backups/nextcloud/

echo "Backing up important data from the server done."

echo "Updating system."

dnf update -y

flatpak update -y && flatpak uninstall --unused -y

echo "Updating system done."

echo "Scrubbing disk."
btrfs scrub start -B /
echo "Scrubbing disk done."

echo "All done! Going to sleep."

shutdown now

You’ve disabled the boneheaded Lenovo WiFi adapter whitelist using a tool like 1vyrain, installed a better adapter like the Intel Dual Band Wireless-AC 7260, and found that Bluetooth is not working as intended on your ThinkPad T430 running Linux?

Here’s a guide on what you can try to resolve some issues you may encounter.

The integrated Bluetooth adapter

The ThinkPad T430 comes with Bluetooth out of the box. Mine is this one:

Bus 001 Device 004: ID 0a5c:21e6 Broadcom Corp. BCM20702 Bluetooth 4.0 [ThinkPad]

I run Fedora Linux and frankly the experience with the stock adapter is horrible. Getting stuck while pairing a device or the GNOME Bluetooth settings GUI hanging isn’t a rare sight. Sometimes I’ve had to run a script to reset all USB devices to get things sorted out, or restart the machine altogether. This is not fun when a work call started 3 minutes ago and your Bluetooth headset is just not working.

After installing the new Intel WiFi adapter, I thought that I’d just disable the integrated Bluetooth controller via the UEFI settings. That should do the trick, right?

Nope. Apparently this disabled ALL bluetooth devices, including the one provided by the Intel adapter.

Well, no problem, I’ll just physically remove the Broadcom device and that should force the ThinkPad to use the Intel adapter, right?

Nope. Now the machine just shows no Bluetooth devices.

udev to the rescue

If you install the Intel adapter and don’t touch any of the settings, you’ll have two Bluetooth adapters. On my laptop, the OS used the Broadcom adapter by default, which was evident by all GUI interactions interfacing with the Broadcom one and throwing out errors in dmesg -w as a result.

Since disabling the Broadcom bluetooth adapter via other means did not work, I had to look for some solutions. One that I stumbled upon relied on udev rules.

Essentially, what you want to do is drop a file in /etc/udev/rules.d/, name it something like 81-bluetooth-hci.rules and make the contents something like this:

SUBSYSTEM=="usb", ATTRS{idVendor}=="0a5c", ATTRS{idProduct}=="21e6", ATTR{authorized}="0"

Make sure to replace the idVendor and idProduct with the correct values for your Bluetooth adapter. Output of lsusb will likely contain those values.

In my case, the output of lsusb is

Bus 001 Device 004: ID 0a5c:21e6 Broadcom Corp. BCM20702 Bluetooth 4.0 [ThinkPad]

The idVendor and idProduct info is in this string: ID 0a5c:21e6.

After making this change and restarting, you should see thatbluetoothctl list outputs only one Bluetooth device.

[host@T430 ~]$ bluetoothctl list
Controller AC:FD:CE:30:EB:BA T430 [default]

Because Bluetooth is still a hot mess in 2022, I still have to occasionally beat it with a hammer. However, the experience with the Intel adapter is much better and a simple “turn the adapter off and on again in the GUI” solved 99% of the issues I’ve encountered since then.

“Have you tried turning it off and on again?”

If for whatever reason this solution does not apply to you, and you feel like you want to hit your machine with a big hammer without having to restart it, then here’s a script that resets all the USB devices on your machine. I’ve observed that most Bluetooth adapters run over USB, which is why this script works with those as well.

It has helped me countless times when desperate for a quick Bluetooth fix on various desktops and laptops that I’ve used.

Run it at your own risk.

#!/bin/bash

for i in /sys/bus/pci/drivers/[uoex]hci_hcd/*:*; do
  [ -e "$i" ] || continue
  echo "${i##*/}" > "${i%/*}/unbind"
  echo "${i##*/}" > "${i%/*}/bind"
done

Update: I’ve switched to btrbk for managing btrfs snapshots due to certain deficiencies that I had with snapper. Those may have been fixed since then, but btrbk has served me very well, and I’m sticking with it in the foreseeable future.

Friday afternoon. You’re trying out a script that you wrote to mass-rename and move some files around. You finish the script and test it out.

Oops.

All the files now have all the wrong names, and some have been randomly moved 10 folders deep. It’s a mess.

And you didn’t make a backup of your files right before this step because you thought that you wouldn’t need it. It’s just a simple script, after all…

If you’ve found yourself in a similar situation and want to know how to avoid problems in the future, then this article is for you

btrfs: overview

btrfs (butter-fs, bee-tee-arr-eff-ess) is a modern filesystem present in the Linux kernel that has a lot of great features:

• different RAID levels: RAID0, RAID1, RAID10…
• data integrity guarantees
• snapshots
• and many more!

btrfs does have some downsides and has had its fair share of issues and controversy in the past, but at least during the last couple of years it’s been fine in my simple RAID1 + snapshots use cases.

In this post, we’ll focus on the snapshots feature that btrfs offers.

If you’re not sure what snapshots are, then you can think of them as a still image of the state of your filesystem at a certain point in time. You can create a snapshot of the filesystem, delete all your files, and still be able to get your files back just like you left them.

If you’re a developer who has used git and doesn’t mind oversimplified (and probably incorrect) explanations: you can think of snapshots like git commits. You can take snapshots (commit), roll back to a previous snapshot (reset/revert) or view the contents of a previous snapshot (checkout commit).

One thing to keep in mind with snapshots is that they will take up space, especially if you make a lot of them and move files around a lot.

There are many tools available to manage btrfs snapshots. The one I’ve opted for is snapper.

Snapper quick-start

Here’s a quick guide on how to get started with snapshotting your btrfs filesystem. The example is based on a system with a btrfs filesystem mounted on /home

• Install snapper
  • Ubuntu/Debian: sudo apt install snapper
  • Fedora: sudo dnf install snapper
• Setup the configuration: sudo snapper -c home create-config /home
• /etc/snapper/configs now has a text file named home that you can configure to your liking. My personal recommendation is to use the TIMELINE snapshots configuration to configure the maximum number of hourly and daily snapshots.
• snapper ships with two systemd timers that automatically take new snapshots and remove old ones. Make sure that they are enabled:
  • sudo systemctl enable --now snapper-timeline.timer snapper-cleanup.timer
• After a while, you should be able to see snapshots appear in the output of sudo snapper -c home list.
• If you want to browse the files for the old snapshots, go to /home/.snapshots.

Example output of sudo snapper -c home list on my machine:

> sudo snapper -c home list

    # | Type   | Pre # | Date                     | User | Cleanup  | Description | Userdata
------+--------+-------+--------------------------+------+----------+-------------+---------
   0  | single |       |                          | root |          | current     |         
   1  | single |       | K 16 märts 2022 08:12:01 | root | timeline |             |         
 242  | single |       | K 16 märts 2022 13:00:16 | root | timeline |             |         
 302  | single |       | K 16 märts 2022 14:00:05 | root | timeline |             |         
 362  | single |       | K 16 märts 2022 15:00:10 | root | timeline |             |         
 422  | single |       | K 16 märts 2022 16:00:19 | root | timeline |             |         
 461  | single |       | K 16 märts 2022 17:20:29 | root | timeline |             |         
 501  | single |       | K 16 märts 2022 18:00:02 | root | timeline |             |
...         

That’s it!

Snapper tech tips

If you’re new to snapshots, then I recommend keeping an eye on your disk usage and tuning the snapper timeline related configurations to your liking. For my /home folder, my configuration looks like this:

# create hourly snapshots
TIMELINE_CREATE="yes"

# cleanup hourly snapshots after some time
TIMELINE_CLEANUP="yes"

# limits for timeline cleanup
TIMELINE_MIN_AGE="300"
TIMELINE_LIMIT_HOURLY="48"
TIMELINE_LIMIT_DAILY="7"
TIMELINE_LIMIT_WEEKLY="0"
TIMELINE_LIMIT_MONTHLY="0"
TIMELINE_LIMIT_YEARLY="0"

With this configuration, I can undo my screw-ups for up to 7 days, with more granular control for changes done within the last 48 hours.

If you have data that changes more often than other data on the same filesystem, then I recommend utilizing btrfs subvolumes and creating separate snapper configurations for each.

For example: a folder containing gigabytes of cat pictures is unlikely to change all that often, so a longer snapshot retention policy of 24 hourly/30 daily snapshots is a sensible one.

However, the contents of my downloads folder changes more frequently and may contain bigger files that I need temporarily, such as the ISO file for the latest Fedora/Debian release. In a situation like that, having a 6 hourly/7 daily snapshots policy will make more sense, especially if I don’t want to run out of space due to excessive number of snapshots taken for that filesystem.

Further reading: ZFS snapshots

ZFS is a fantastic filesystem that offers many of the features boasted by btrfs, and some that btrfs doesn’t have. ZFS also supports snapshots. To set up automatic snapshot creation and pruning on ZFS, I can recommend sanoid.

Do you have a need to host Wikipedia on your computer?

Or a StackExchange site, like Super User?

It’s easier than you think!

Step 1: download stuff!

The Kiwix project provides ZIM files for many popular websites, including Wikipedia. These files can be downloaded over at Kiwix wiki.

Take a look at the content packages available and download those that you are interested in.

Step 2: install Kiwix

Kiwix provides all sorts of clients for different operating systems. If you don’t want to go crazy with setting up a server and you just want to browser your offline copy of Wikipedia or any other resource, then go ahead and use that.

However, if you want to make the resources available for other people as well, then you need to install kiwix-serve. On Debian and Ubuntu systems, this is just a sudo apt install kiwix-tools in your terminal.

Once you have it installed, all you have to do is to run kiwix-serve -p 8080 your-file-here.zim and your content package is served to anyone who can connect to your machine with a browser over port 8080.

If you have multiple ZIM files in the same folder, then you can serve all of them at once with something like kiwix-serve -p 8080 *.zim.

Step 3: enjoy!

You should now be up and running with your very own hosted Wikipedia instance! Just keep in mind that your copy might not be 100% up to date. The Kiwix project does repackage these sites quite regularly so keep an eye out for updates.

Tips and tricks

I have set up my own Kiwix instance and the way it works is quite simple.

I use Deluge to download and seed the content packs that I have downloaded.

Then I have also set up a systemd service that automatically starts Kiwix on boot. It is located at /etc/systemd/system/kiwix-serve.service and looks like this:

[Unit]
Description=Serve all the ZIM files loaded on this server

[Service]
Restart=always
RestartSec=15
User=kiwix
ExecStart=/usr/bin/bash -c "/usr/bin/kiwix-serve -p 8080 /path/to/your/files/*.zim"

[Install]
WantedBy=network-online.target

Note that I have set up a separate non-interactive user for this purpose. The path to the ZIM files simply refers to the download folder of the Deluge torrent client.

Making this available to the world is simply a matter of either exposing the selected port to the world, or using nginx as a reverse proxy. There are plenty of guides available on how to achieve the latter so I won’t go into detail about that here.

That’s it!

2024-09-05 update

It’s been a fun experiment, but LLM-bots are overwhelming my Kiwix instance and I can’t be bothered with setting up any protections right now, so the kiwix.ounapuu.ee instance is shut down for now.

Introduction

Before we jump into all the nitty-gritty details, I’d like to go over what we are dealing with here since these topics may be unfamiliar to you. VFIO is quite a niche topic and not everyone knows about it.

GPU passthrough: the process of allowing a VM (virtual machine) to use a dedicated GPU. This allows you to run GPU-heavy workloads inside the VM, such as gaming or anything that benefits from GPU compute power.

VFIO: the framework that allows us to perform this operation.

IOMMU: hardware feature that supports all of this.

virt-manager: GUI application that I use to manage VM-s.

Reasons why you may want to go through all the hassle and use this approach:

• you’re on Linux but want to play games occasionally without rebooting into Windows
• you want to do all your work, gaming and server workloads on one machine
• you simply don’t want to build a dedicated PC for playing games and would like to utilize the existing resources on your main machine
• you want more control over the Windows installation because you simply don’t trust Microsoft
• you want to have more control over the Windows installation and the ability to revert the installation to an earlier point in time using a filesystem that supports snapshots (BTRFS, ZFS)
• you have multiple separate VM-s for different purposes, but you would like to use one single GPU for all of them

One thing to keep in mind is that the hardware you use matters a lot here as that provides the foundation for getting this solution working. Here are some of the requirements you should know about:

• The GPU you want to pass through has to support UEFI. GPU-s from the past 5 years should be good on that, but if you have a really old GPU that you want to use for testing, then that might not work out that well.
• You need to have a good-enough CPU and plenty of RAM. After all, you’re essentially running a full PC within another PC.
• Your motherboard has to support IOMMU and have IOMMU groups that allow you to isolate the GPU and only pass the GPU to the VM. Level1Techs does quite a few motherboard reviews and Wendell goes over the IOMMU groups and suitability for VFIO in them. It also helps if you look around the internet for enthusiasts that have already bought the same motherboard, they may have posted information about the groups. Here’s a handy script that allows you to check your IOMMU groups.
• Some games may ban you if they detect that you are running the game inside a VM. It’s great that they’re trying to do something against cheaters, but unfortunately VFIO users get unfairly treated here. Please do some research beforehand if you play competitive multiplayer games a lot!

I hope that this introduction helped you understand the basics. Now, let’s just jump right into it.

Setup

We’re going to test this setup on hardware that I have covered previously. What makes this setup a bit special is that we’re using an AMD Ryzen 7 5700G APU on an mITX motherboard and a single dedicated GPU, allowing us to do big things in a small package. Yes, there are setups where you can do VFIO with a single GPU and pass it between the host OS and the VM, but that setup might be a bit tricky to use.

The OS is Fedora 34. To get this working, I’ve used various resources:

• beginners guide on Level1Techs forums
• Fedora-specific guide by Wendell from Level1Techs
• this brilliant Arch Wiki guide that covers most of what you need to know about setting this up. It’s not only useful for Arch, a lot of what is written here applies for any distro.

Testing

I decided to do the initial testing with the Nvidia GT 710. It’s slow, buggy on Linux on Nouveau open source drivers, and I had it available. I also recently heard that Nvidia stopped being hostile towards their customers in one aspect by allowing GeForce GPU-s to be used in a Windows 10 VM without having to resort to workarounds.

The testing itself was relatively straightforward. The only issues I had were PEBCAK issues, possibly related to me performing this testing after work. The main issues I ran were either small typos in dracut configuration or the wrong device ID-s being added to kernel boot parameters. Once I discovered and fixed those, it was all smooth sailing.

I’m not going to rewrite a full guide here, please refer to the previously linked resources if you need more details than that. Those resources are also much more likely to receive updates about new finds and features.

The steps I took were roughly these:

• Make sure that IOMMU was enabled in UEFI settings. It’s probably set to auto by default, make sure to set it to enabled.
• Install the GPU that I want to pass through and connect it to a monitor.
• Install virtualization packages on Fedora: sudo dnf install @virtualization
• Enable IOMMU and preload the VFIO kernel module by adding amd_iommu=on rd.driver.pre=vfio-pci to Linux kernel parameters.
  • I use GRUB2, these parameters live in /etc/sysconfig/grub.
  • To apply these changes, you have to regenerate the GRUB configuration. Under Fedora, this is done using grub2-mkconfig -o /etc/grub2-efi.cfg
• Get the device ID-s for the GPU and the related audio device that you’re planning on passing through
  • To see the devices and ID-s, run lspci -nnk. The ID-s look something like 1002:aaf0.
• Bind the GPU to vfio-pci to avoid the GPU driver from taking control of the GPU, otherwise you cannot pass it to the VM.
  • I opted to go for the simple approach and added the device ID-s to kernel parameters: vfio-pci.ids=1002:67df,1002:aaf0
  • Since I went with this approach, then I have to regenerate GRUB configuration again.
• Make sure that the initramfs loads the necessary vfio drivers early in the boot.
  • With Fedora 34, this means creating a file /etc/dracut.conf.d/10-vfio.conf with contents add_drivers+=" vfio_pci vfio vfio_iommu_type1 vfio_virqfd ".
  • Make sure that you don’t have any typos in that.
  • Regenerate the initramfs: dracut -f
• Reboot!
• Using virt-manager, select the VM you want to pass the GPU through and add two PCIe devices: the GPU and its associated audio device.
  • If you don’t have the VM set up yet, then proceed with the normal installation without the GPU passed through yet. Make sure to create an UEFI VM, otherwise you might run into issues. This can be configured in virt-manager _ Overview_ section by selecting the Q35 chipset and setting the firmware to OVMF_CODE.df.
  • If you have passed through the GPU, then make sure to remove the Display Spice device from the VM.
  • If you want to also control the VM, you need to also pass through your USB devices, such as a Logitech wireless receiver.
• Start the VM and cheer once you notice TianoCore appearing on the monitor that’s connected to the passed through GPU!

Results: Nvidia GT 710

Initial testing with Nvidia GT 710 was successful. By successful, I mean that the GPU displayed an image and did not install GPU drivers automatically.

To overcome that last issue, I downloaded the latest official Nvidia GPU drivers and was good to go. The news were true, error code 43 was no longer an issue!

Things got a bit less exciting when I got reminded that this GPU is weak. Very weak. Regardless, I decided to demonstrate its computing prowess by downloading the art of rally demo. Side note: it’s a great game, go try it out!

The demo ran! Not well, but it still ran!

Satisfied with the results, I decided to go for gold and replace this GPU with the AMD RX 570 that I “borrowed” from this PC I built recently.

Results: AMD RX 570

After switching the GPU, I replaced the device ID-s with the new values and continued with the VFIO adventure. This time, the GPU drivers were automatically installed by Windows and everything just worked. That’s not supposed to happen, at least in my experience something always goes wrong. Always.

I installed the latest drivers from AMD-s website and continued with testing.

Furmark? Runs as expected.

GTA IV? A stuttery mess, so yeah, runs as expected.

At this point the VM was assigned 4 CPU cores and I had performed no CPU pinning or optimizations of any kind, so the results were pretty good.

Storage

Because I built this setup on my current workstation/server machine, the storage situation is a bit tricky. The other VM that runs all the services has full access to the two 12TB hard drives and I wasn’t interested in setting up a networked storage setup. The only free spots I had were as follows:

• 120GB partition from the NVMe SSD. Good enough for storing the Windows 10 system files.
• 2x 250GB partitions on the Samsung SATA 1TB SSD-s. Just enough to hold my recently played games, but not much more. Running in striped configuration under Windows.

I later opted to expand the game storage partitions to 375GB, which meant that I had to get rid of the extra overprovisioning space that I had left aside. This setup is fine, but I’m losing out on some of the benefits that come with a virtualized Windows setup.

With regards to passing this storage to the VM, I had two options:

• keep using the SATA virtual disk: works out of the box, but might not perform as well.
• use virtio: you need to manually load and install the drivers for Windows to recognize these disks, but allegedly this has better performance than the SATA implementation.

I started off with SATA and did a comparison against virtio using CrystalDiskMark.

virtio did come ahead in these comparisons, but I would probably have been fine with the SATA performance as well.

Conclusion

One of the main reasons I do these kinds of setups is the technical challenge. It sounds weird, but sometimes I am happier after having completed something technical and novel compared to actually using the setup.

Previously I’ve had mixed results with GPU passthrough, with ThinkPad T430 and eGPU somehow managing to run GTA V in a VM , while in another testing session I ran into issues which turned out to be related to the CPU being faulty (lots of PCIe errors).

I do have more practical future plans with this. The plan is to use this VM as a gaming VM that I can stream games from using Parsec to any device that I wish. One idea is to use the Nvidia Shield TV as a low-power box that is capable of performing streaming. Alternatively, I could also get a super tiny form factor Dell/Lenovo/HP PC that has plenty of power to drive a 4K display and uses a reasonable amount of power. More on that in a future post.

Regarding storage I foresee an upgrade coming soon, either to the drives themselves or the whole setup since ATX motherboards support more SATA ports and expansion cards. If you also pair that up with a case like Masterbox Q500L with a good power supply placement, then you’d still have a relatively small setup.

Background

I have a Lenovo ThinkPad X230. It’s a small and good laptop that I’ve used as my main laptop and a server as well.

A couple of years ago I messed around with flashing alternative BIOS implementations on the X230, such as the skulls project, which made installing prebuilt coreboot images very simple. Of course, before doing any testing, I made backups of the stock BIOS implementation (ooh, foreshadowing!).

Skulls was nice, it booted fast, but changing the boot order likely required modifications and rebuilding the coreboot image (I wanted to set the mSATA SSD as the first bootable device). It was also not UEFI-compatible, which could have probably been solved with something like TianoCore, but I didn’t feel like messing with that too much. Eventually I flashed the original UEFI implementation back and went on with my life.

Then I stumbled upon the 1vyrain project project, which allows you to “jailbreak” your BIOS. This modification also enables a lot more bells and whistles in the BIOS under the Advanced menu option, plus it also gets rid of the WiFi adaptor whitelist that Lenovo has put in place, allowing you to now use any adaptor that’s compatible with the form factor.

The mod was very easy to install and had no issues, until I got myself a DDR3 SODIMM RAM kit. One of the sticks was not working properly, showing up with graphical glitches and failing a memtest run. To troubleshoot it, I decided to change the speed at which the memory runs from DDR3-1600 to DDR3-1333. It booted, and still had the issues.

Then I changed it to DDR3-1066.

Black screen. Uh-oh.

Initial troubleshooting

Thinking that this is just a simple issue that I could get fixed by clearing the BIOS settings and starting fresh, I looked up some basic instructions on doing so.

Things I tried:

• remove the battery (even if it is completely dead, like mine), and hold the power button for 60 seconds, then attach the battery and turn it on again.
• remove both the battery and disconnect the internal CMOS battery that sits under the palmrest and hold the power button for 60 seconds again.
• disconnect all the batteries and leave the laptop sitting on a shelf, then try again.
• try “the secret ThinkPad power button combo”:

… unplug the AC adapter and take out the battery. Then, you push the power button 10 times in a row at one second intervals. Next, you push and hold the power button for 30 seconds. Then you put the battery back in and push the power button… and she lives. The computer came back, good as ever.

After all of that, the laptop was still as good as dead. You could see the light turn on, the fan starts spinning, but nothing shows up on the screen. At that point I knew that the BIOS reflash was probably my only choice.

Flashing the BIOS

To fix the issue, I had to flash the BIOS externally. Luckily I still had the original BIOS backup images available.

With the help of a good friend, we set everything up and did some tests to ensure that the connection is good and that read/write operations on the chips return consistent results.

Once we had specified the correct chip type by looking at what was written on the chip package itself, we flashed both the top and bottom chips with the original images that were once made as a backup. After reading the chips back and comparing checksums using md5sum, it all looked good.

We disconnected the SOIC clip, attached the power cable and tried turning the laptop on. Power button light comes up for 1 second and then turns off. Not a good sign.

We flashed both chips again and made sure that everything is in order. No dice, the symptoms are the same.

A closer look

It didn’t take too long for me to start thinking about what I could use all these extra laptop parts for. After all, the screen was good, it had 16GB DDR3 RAM, two SSD-s, a screen assembly in good condition etc. I was stopped in my tracks when my friend noticed something.

It turns out that during this flashing process, we (probably me) had managed to get rid of a resistor that sat next to the top flash chip. Whoops. Turns out that it was an important one, because over at Reddit, someone else also ran into this issue and had similar symptoms .

To resolve this, my friend had the idea of putting a 10K resistor in there to see if it fixes things. The issue was that the only one available right from the get-go was a very big one, so he had to be clever with it. He ended up positioning it upright with a piece of wire handling the other side.

This, however, didn’t improve the situation much.

Schematics to the rescue

To understand the role of the missing resistor better, we started looking for schematics online. I managed to find an OK looking one from a random forum and started browsing through it. The keyword of the day was “SPI” and when searching for it, we found some promising results. My friend found the resistor and found that we could just short it, since the resistor used to be between two wires. And that he did.

We attempted to power on the machine again and this time the error sequence was different as the laptop went into some sort of a party mode with lights flashing. The usual troubleshooting steps didn’t work that well here, either, so it’s possible that the stock BIOS image did not like something here.

What we did have was the option of flashing the known-good skulls image on. I handled that part quite OK, since all the commands were in my bash history anyway. The excitement, however, got the best of me and I managed to repeat the original mistake and I bumped the soldered piece of wire off of the board, putting us back to the original issue.

After another session of carefully connecting the two wires, we repeated the steps and after the flash, we finally saw the Skulls splash screen.

It works!

After booting into a Fedora 34 liveusb environment with iomem=relaxed, I grabbed the latest release of skulls ( released on 4/20 🔥) and used the internal flashing method to update.

Reboot, and we saw party mode again. Having used skulls in the past, I knew that this was likely down to skulls (or coreboot) being very picky about RAM. The solution here is to reseat the RAM modules and turn it on again. Why does that work? Not sure, but it did.

To make this solution more resistant to my unintentional attempts of breaking things, I added a glob of hot glue next to the top chip to make sure that I don’t knock anything off of the board again.

Testing

Once the laptop was assembled again, I did some testing to see if everything still works as intended.

Fedora 34 liveusb boot resulted in a black screen once, but on the second try, it worked again.

Windows 10 installed fine on the machine and while some functions don’t work as well (TrackPoint and touchpad being the prime examples here), it was still usable. Unfortunately I did not manage to write down the original Windows 10 key that is embedded to the stock BIOS image, so I could not activate it. But hey, at least it works.

Lessons learned

Even though this adventure started with a self-inflicted wound, I still consider it a success, as I learned a couple of things.

• It’s very easy to get confused when reviewing how the wires should be set up on the SOIC test clip and the device that’s actually performing the flashing.
• If a piece of software exposes a bunch of knobs and levers and warns you about catastrophic failures that might occur as a result, then trust it and don’t mess with things that you have no clue about. With this example case, though, I didn’t expect a memory speed change to result in such a catastrophic failure in the first place…
• Developers that know hardware are worth their weight in gold.
• Access to schematics is very valuable when fixing hardware.

And on the final note, I would like to give a shout-out to Arti Zirk, the guy who helped fix the mess I created. If you need someone who is well-versed in anything related to software development, Linux and embedded systems, then he is your guy.

flashrom references

This section has some commands that might be useful to you. Unless you’re using the exact same flasher, you will probably need to specify a different programmer with the -p flag.

The names and extensions for the input-output files are arbitrary. A file is a file.

Reading/writing the top chip.

Note that I had to specify the chip type, otherwise flashrom would complain.

The top chip is 4MB and the bottom chip is 8MB. This should also be reflected in any output that you get.

--verify option is also useful for checking if the results are OK. I also recommend reading the image from the chip after a write and comparing its checksum using md5sum, just to be extra safe.

# Read the existing image on the chip
flashrom -p ft2232_spi:type=2232H,port=B,divisor=4 -r top.bin -c MX25L3206E/MX25L3208E 

# Write the specified image to the chip
flashrom -p ft2232_spi:type=2232H,port=B,divisor=4 -w top_backup.bin -c MX25L3206E/MX25L3208E 

Reading/writing the bottom chip

# Read
flashrom -p ft2232_spi:type=2232H,port=B,divisor=4 -r bottom.bin -c MX25L6406E/MX25L6408E

# Write
flashrom -p ft2232_spi:type=2232H,port=B,divisor=4 -w bottom_backup.bin -c MX25L6406E/MX25L6408E

Reddit is a great starting point for getting new ideas for your homelab: racks full of machines in /r/homelab, storage measured in terabytes (or even petabytes) over at /r/datahoarder, all the different services that people host over at /r/selfhosted. This can be a bit overwhelming for someone just starting out in this area, which is why I decided to write a small guide on how to start off on your self-hosting adventure.

Why self-host in the first place?

Hobbyists and professionals have a lot of different reasons for self-hosting services. These include (but are not limited to):

• it’s an excuse to play around with computer hardware and rationalize the purchase of thousands of dollars of hardware
• reduce reliance on “big tech”
• the services solve an actual problem for the user (home automation, secure file storage, backups etc.)
• learning new tech and solutions that otherwise have not come up during their day-to-day work

Regardless of the specifics, the main part of the self-hosting experience is to just have fun and work on challenges that you find interesting.

Step 0: find a problem to solve

One of the best motivators for setting up your homelab/self-hosting setup is to solve a real problem that you have.

Do you want to automate your home with HomeAssistant?

Or do you want your family to be able to back up their valuable photos and videos to your server using Nextcloud?

Or do you want to share your media collection with your family and friends using Jellyfin?

It doesn’t really matter what solution you end up using or how you do it, as long as it is a secure solution and that solves your original problem without creating 10 additional ones.

Step 1: find a place for running your software

Once you have a problem that you are going to solve, you will need to also find a home for your project to live in. For this, you have a lot of different options:

• using a virtual machine at a cloud service provider (DigitalOcean, Linode etc.). Performance is not the best and beefier configurations can be quite expensive, but this is a reliable option with a fast network.
• renting a server from a service provider (example: Hetzner). More performant than a virtual machine at most cloud service providers, but probably more expensive as well. Usually comes with a fast network.
• an old laptop that you have collecting dust on your shelf. Not the fastest or the quietest option, but it does not use a lot of power.
• an old desktop PC. Probably plenty fast for any task that you can throw at it, but its power usage is higher compared to laptops.
• a tiny desktop PC. Low power usage, but can pack quite a punch. For inspiration, check out Project TinyMiniMicro by ServeTheHome.
• a single-board computer, such as a Raspberry Pi or alternatives. Plenty of performance for most workloads, uses very little power (5-15W generally) and silent as well.
• a second-hand rack-mounted server. Probably quite performant, but uses a lot of power and is very noisy.
• a NAS box from one of the more popular providers (Synology, QNAP, TerraMaster etc.). Useful for when you need a lot of storage on your machine or something that just works.

Your choice will likely be affected by other factors as well:

• your home internet connection. If your ISP does not allow opening ports or if your upload/download speeds are not that great, then using a cloud service provider or renting a server might be a better option for you.
• living in a small apartment will likely rule out noisy solutions, such as full-blown rack-mounted servers or powerful desktop PC-s.
• if all you want to do is local testing, then a virtual machine on your already existing desktop/laptop PC might be a better fit, as it does not require you to buy any new/used hardware.
• reliability and price of electricity. In some countries, running a server in your home might result in a huge power bill or might not be possible at all due to the power grid being unreliable.

The workloads that you plan on running will also have an effect on your chosen approach. You will need to take into account the resources that your workload requires. It does not make much sense to get a rack-mount server for hosting your Wordpress blog if a Raspberry Pi can do the same job just as well. On the other hand, running a CPU-heavy service like Jellyfin on a Raspberry Pi will be a painful experience when compared to running it on a desktop PC or an enterprise-grade server. If your service will be used by a high number of users, then you might want to opt for a more powerful machine.

Some solutions may also benefit from faster, SSD-based storage. Loading a lot of smaller files or running a big database off of hard drives will be noticeably slower than running them off of fast SSD-s.

There is no one size fits all solution. It’s better to do a little bit of initial research and use the right tool for the job. If in doubt, ask for help from someone more experienced.

Step 2: find the software solution that fits your use case the best

Just like with hardware, there is also a lot of choice when it comes to software.

Some solutions are pretty low-level and require setting up the OS and the services manually. This is not very user-friendly, but it offers the most potential for learning about your system and how to manage it. This approach will likely require you to learn new things and pay more attention to automating common tasks (updates, monitoring, backups) and making sure that everything is secure (no weak passwords, services properly isolated, regular and frequent updates etc.). If something goes wrong, then you will need to fix it yourself, but at least you have all the tools and knowledge to do so.

Example: Debian with ZFS for your main storage and services running using Docker or Podman (my personal preference).

There are also solutions that cater to people who just want to get work done. These solutions usually provide the user with a web-based user interface that can manage the system and any services that run on it. Do you want to set up your very own Wordpress blog? Just click this button and you will soon be set up with one without having to even know what the hell a “Docker” is. The downside of this approach is that when things go wrong, you will still need to familiarize yourself with the inner workings of the system. Troubleshooting issues with no preparation and learning new things under stressful circumstances is not very fun.

Cloud service providers have also started offering one-click solutions to setting up the more popular services. your very own Minecraft server is just a click away!

Examples:

• a NAS box from Synology that is aimed towards consumers
• VM provided and set up by Linode using a one-click solution in their web UI.

And then there are solutions that fit in somewhere in-between. Solutions like TrueNAS (formerly FreeNAS) and OpenMediaVault are made for users that can probably set things up themselves as well, but are just too lazy to do it. Using these solutions is not in any way worse than the alternatives. If it solves a problem and fits your use case, then go ahead and use it! Just keep in mind that you might need to peek under the hood when there are issues where the GUI cannot help you out.

As always, if in doubt, ask for assistance. It is perfectly fine for you to pick one of the user-friendlier options first and dig deeper once you feel more comfortable. Self-hosting is not a race, you can do everything at your own pace.

Step 3: managing your setup

Once you have everything up and running, there are some topics that you really should pay attention to in order to avoid big problems down the line.

Backups

If the only copy of your files lives on your NAS and something should happen to them (accidental delete, hard drives suddenly die, you get hit by ransomware etc.), then you are in a world of hurt. Having backups is a mandatory part of your self-hosting adventure. A mix of automated and manual backups is better than having no backups at all.

Some ideas for backups:

• external hard drive attached to your server that receives automated backups that run on an schedule. Just make sure that any failures during the backup will be communicated to you (over e-mail, for example). It’s also a good idea to occasionally check if the last backup was successful or not. Downside of this approach is that since this drive is still connected to the server, it is subject to other events that can shred your data (ransomware, power surges etc.).

• offline backups that you will manually run on a regular schedule. Example: copying all files from the server to an external hard drive on the 1st day of each month. If you accidentally delete all files on your server or a piece of malware does this for you, or if your server is toast due to a power surge, you will still have a copy of the data that cannot be affected by those kinds of issues.

• snapshots. While technically not a backup, snapshots do help against accidental deletes. Depending on your setup, you might be able to retrieve files deleted a week ago if you keep snapshots of your data for the last 30 days.

• a hard drive at your friends house. Helps protect against the worst case scenario where your house burns down.

• (encrypted) backups to cloud storage providers, such as Backblaze, OneDrive or Google Drive. It will require a monthly subscription, but at least you will likely have a copy of the file somewhere. Just keep in mind that if you should get locked out of the service or fail to pay for it, then you might not be able to access the data hosted at that service.

Security

If you are exposing a service to the wild wild west that is the internet, then you will be a target. Automated scanners will poke your services and if they find a vulnerability, they will use it. To prevent the worst, consider the following:

• keep everything up to date. Don’t forget that for some updates to apply (example: kernel updates), you need to restart the machine. For other services (example: ssh), they can be restarted without restarting the whole machine. Regular updates will help prevent most security issues.

• only expose services that you absolutely need to expose. With something like a web server, you really don’t have a lot of choice here, but with services aimed towards a known set of users, you can limit the exposed surface area by using a VPN or enforcing other restrictions (whitelisted IP addresses, fail2ban to block most intrusion attempts, etc.).

• make sure that your configuration is correct. A lot of intrusions are caused by configuration issues that usually end up with a service being exposed to the web without any passwords or other protections set up. It’s a good idea to check if you are accidentally exposing too much by checking what others are seeing when they try to connect to your service.

• try to set up your services so that you are prepared for an intrusion. If the attacker is able to take control of service X, but it is completely isolated from other services and devices on the network, then the damage they can do is likely limited.

Monitoring

Your setup will likely have issues. Hard drives might die, or the CPU might be overheating due to years of dust accumulation, or one service might be using up all the available resources, causing others to slow down as well. A better understanding of what your system is doing will help out a lot here.

Cloud hosting service providers usually provide at least a basic level overview of your resource usage: CPU, RAM, disk, network etc. If you are running your own machine, then you will need to figure this out on your own.

To understand what your system is doing, a tool like Netdata is a good starting point. If your server slows down, then you can most likely pinpoint the reason by looking at the resource indicators.

It is not realistic for you to keep an eye on all the indicators all the time, so having some kind of alerting set up is also a great idea. Events, such as a hard drive starting to throw errors, the CPU usage being at 100% for more than an hour, or a service not being reachable, are all something that you might be interested in hearing about.

You don’t have to monitor everything and have alerts set up to catch everything, but it is a great starting point for responding to issues proactively. Ending up with all your data lost because you did not hear about hard drive failures is not a great situation to be in.

What next?

If you have reached this point, then you are probably aware of most of the topics that will come up during your self-hosting adventure. You will encounter issues that will be specific to the approach that you have chosen at the hardware and software steps, but that’s perfectly normal. Most of these issues have solutions that are one search away, and if you have trouble figuring something out, then feel free to ask for help!

If you need new ideas, then check out the subreddits mentioned at the start of this post, you might find some gems in there! Do keep in mind that you will likely end up in an endless loop of building something, being happy with it for a week and finding something else to try out. It’s really fun, though.

2025 June update: the contents of my archive are now easily accessible over at nfs-cars.com, a read-only static website that hosts content that was previously on nfscars.net. I’m not affiliated with that website, but I’m very happy that it exists!

Background

My mother once had a laptop: the Compaq Armada 1592DT. It came with Windows ME which later got “upgraded” to Windows 98 SE after I managed to completely screw up the OS, a whopping 96MB of RAM and a hard drive that probably wasn’t much larger than 1-3GB. It wasn’t powerful or anything, but some of my earliest memories of playing video games on that thing were related to Sports Car GT and Need for Speed III Hot Pursuit. Yes, both were demos, and the performance was probably in the 10-20FPS range, but it was still great for a younger me. That laptop eventually got replaced by a Windows 98 based desktop which wasn’t a lot better with its 64MB of RAM, but it could play RuneScape (barely) and also Need for Speed III.

Need for Speed III Hot Pursuit holds a very special place in my heart. It looked great at the time, the police chases were fun and challenging, the soundtrack was fantastic and the track featured in the demo (Rocky Pass) is still something I could probably draw up from memory, assuming that I could draw. At some point I discovered that it is possible to mod the game to include new cars and tracks, which resulted in ridiculous speeds, cars that could absolutely destroy the police cars, and lots and lots of game crashes. One of the sources for these mods was nfscars.net.

nfscars.net

Depending on when you will be reading this, nfscars.net is/was a website that hosts/hosted thousands cars, tracks and other mods for various Need for Speed titles. In 2020, I decided to check this site out again to see if it is still operational.

The last post referred to an “upcoming” NFS title that was already released a year ago.

The link to the forums resulted in an error page being shown.

Should the site suddenly disappear without any warning, all the content would be lost. Since I have a small datahoarding bug, I decided to take matters into my own hands and do my best to archive the contents of the site. That, and nostalgia for the old Need for Speed titles was what pushed me to start this archiving adventure.

Initial efforts

Before trying any fancy technical solutions to crawl the site and get all of the assets that way, I decided to start from the source and tried contacting the site owner to arrange the download of the assets. That did not work out at all, since the contact e-mails were either not active or the inboxes were full.

I signed up for the site and tried to message some user accounts which hopefully were related to someone who was in charge. No dice.

There is a small chatroom on the site named “Shoutbox”, so I tried asking about the site owners there. It seems that a couple of users are still occasionally chatting there, mainly about random topics and spammy users. Unfortunately none of the users there - even the moderators - could not assist with setting up a backup of the site.

With the domain (potentially) expiring on 2021-05-02, I felt like I had to act now, because the site was probably running on borrowed time.

Technical solutions

I checked a selection of solutions for performing a crawl on the site and decided on Heritrix, mainly due to this being backed by the Internet Archive. If they cannot get archiving right, then who can?

Setting up Heritrix turned out to be quite a hassle. There does not seem to be a quick guide that you can follow to get started in 5 minutes. After stumbling around, learning the config file format and understanding that the best way to configure crawls was to do it externally with a text editor, I was set up to launch my first crawl. Even then, I hit some roadblocks:

• The crawl was slow. I understand that Heritrix is just trying to be polite here, but if you have millions of URL-s to crawl and the site might go down at any moment, then time really isn’t on your side.
• Checkpointing is supported, but for some reason the default configuration does not enable it. If the machine that you are running this on should restart, then your crawl will be stopped and you cannot resume at that point, you can only start over.
• Heritrix is based on Java, which means that theoretically it should run on anything that runs Java, such as an ARM CPU based SBC. However, I ran into weird java.nio related issues with symlinking on those platforms and I just could not get them resolved.
• If you run too many crawls at once, then Heritrix is perfectly capable of killing your LAN by using up all the available connections. It managed to rack up 16000+ active connections, which hit the 16384 connection limit on my consumer-grade router. That was not fun to debug.
• Automatically continuing crawls from a checkpoint on system startup requires you to use the Heritrix API, which isn’t too complicated, but I just wish that this was another feature that it supported out of the box.

After messing up a couple of times and trying again, I finally managed to complete a crawl of the site. This only took 6 months, where the crawl itself ran for 3 months. MONTHS!

What the hell is a WARC?

Heritrix collects the results of the crawl into a special format called WARC, which it then compresses, resulting in lots of little .warc.gz files. If you extract one of these, you will end up with a .warc file.

To extract files from a WARC file, you need one of these tools listed here. After trying a couple, I decided to go with warcat since it seemed to mostly work and it supported combining smaller WARC files into bigger ones and extracting the contents.

Depending on the size of the site, the extracting process might take a while. Crawls can contain thousands or even millions of small little files, and if you are using hard drives for storing these, then it will be slow. I have not done testing on an SSD, but it will probably be much faster due to its superior random I/O performance.

Once I had extracted the contents, I did some checking to see if the assets that I care about are present in the crawl. Did a file search for a modded car in Need for Speed III.

No results.

Okay, that’s odd. Tried it with another file name.

No results.

Well, what do we have? I checked the extracted contents using ncdu and found that while Heritrix managed to grab a lot of the webpages and the images from the galleries, it failed to download files, such as .zip and .rar files that contain the modifications themselves.

Getting the final pieces of the puzzle

I started investigating the download functionality of the site. On every page load, the download link would have a new identifier attached to it. Upon clicking on it, the download would start. If you wait too long, then the download link would expire.

Firefox has a handy feature where you can copy the request into a curl command. I did that, added some parameters to it so that it would use the filename that comes from the response header and made it save the file to disk. It worked!

Now I had to figure out how to collect all of these download links. The site has sections for each game, which is also reflected in the URL. However, I found that when you navigate to the page of a mod and change the numerical ID at the end, you could cycle through all the cars, tracks and other tools for all games, even if the URL was technically referring to one game in the series.

After bashing around for a while, I came up with this masterpiece:

#!/bin/bash
cd /your/download/destination || exit 1

for index in {1..18050}; do
  echo "Index at $index, starting."
  filePath=($(curl --silent https://www.nfscars.net/need-for-speed-hot-pursuit/1/files/view/$index/ -H 'Cookie: csrftoken=XXX; sessionid=YYY' | grep -oE "files/download/send/[0-9]+/[0-9a-zA-Z]+"))
  echo "File path: $filePath"
  echo "Making directory $index"
  mkdir -p "$index"
  echo "Going into directory $index"
  cd $index

  for path in "${filePath[@]}"; do
    echo "Starting download on path https://www.nfscars.net/$path"
    curl --silent "https://www.nfscars.net/$path" -H 'User-Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:85.0) Gecko/20100101 Firefox/85.0' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' -H 'Accept-Language: en-US,en;q=0.5' --compressed -H 'Connection: keep-alive' -H 'Referer: https://www.nfscars.net/need-for-speed-hot-pursuit/1/files/view/1/' -H 'Cookie: csrftoken=XXX; sessionid=YYY' -H 'Upgrade-Insecure-Requests: 1' -OJ
    echo "Download $path done."
  done

  cd /your/download/destination 
  echo "Back in home dir. Index $index end."
done

Not the prettiest script around, but it did the trick. In a day or so the assets were downloaded and you could use the index to connect these to a page that is contained in the crawl.

Results

All this effort would be for nothing if I didn’t share my results, so I decided to pack these up and serve them for anyone that wants to keep a backup of at least one part of Need for Speed history.

The archived content can be found on the Internet Archive. Here’s a newer crawl from 2021.

The torrent files are also mirrored here:

• 2020 crawl

• 2021 crawl

Pieces of history get lost all the time with sites going offline. I’m hoping that at least with this effort I have managed to do my part in preserving a tiny piece of it.

2021-03-15 update

The site seems to be down at the moment and has been for a couple of days. Oh no.

2021-03-20 update

Somehow the site is alive again.

2024-08-20 update

It’s dead.

However, I’m delighted to find people using this archive to get their favourite mods!

This post includes a guide on how to match up your favourite mod with the correct filename in my archive. I do wish that I had more time to properly organize that archive while creating it…

2025-06-10 update

I learned that there exists a browsable mirror of the nfscars.net archive! You can find it on nfs-cars.com. I’m very happy that it exists.

Issue description

You are running a Linux-based machine with an install of ZFS on Linux. Everything seems to work correctly, but after restarting your machine, the ZFS pool is not visible. You can still import your pool manually using zpool import poolname or zpool import -a.

In my situation, this issue does not occur with Debian 10 on an x86-based machine, but it does occur on a Raspberry Pi 4 running Ubuntu Server 20.04.1 LTS.

Setup:

• Raspberry Pi 4 (4GB)
• Ubuntu Server 20.04.1
• 32GB microSD card for root filesystem
• 2x 8TB WD Elements external USB 3.0 drives (ZFS mirror)

Potential causes and solutions

When looking this issue up, there were many potential causes and solutions provided:

• ensure that all ZFS related services are enabled
• add a startup delay in ZFS related configuration files
• just import the pools (duh)
• recreate the ZFS cache file

In my specific scenario, none of these seemed to do the trick.

Investigation

The ZFS pools should be automatically imported on boot by a service named zfs-import-cache.service. The cache file should contain information about the imported pools.

Once you have started your system and don’t see any ZFS pools available, try running systemctl status zfs-import-cache.service or journalctl -u zfs-import-cache.service. This should show you if the automatic import succeeded or not.

The pool tanker is a ZFS mirror that is supported by two 8TB WD Elements external hard drives. They are big, loud and require some time to spin up the platters before you can actually read data off of them. At this point in the investigation, a good friend recommended that I check what disks were recognized by the system at the time when this import service was starting up.

For that, you need to modify the systemd service by running systemctl edit zfs-import-cache.service and put these contents in there:

[Service]
ExecStartPre=/usr/bin/lsblk

This addition will run the lsblk command before the command specified in the systemd service is actually executed. The output of this command will be visible in the service logs. After making this addition, reboot your system and run journalctl -u zfs-import-cache.service again. You should be able to see all the connected drives on the system.

In my case, the system recognized the microSD card and an SSD that was connected over USB, but not the hard drives. This was a very clear indication that the system was not waiting until the drives had spun up before attempting to import the ZFS pool.

Hacky solution

To resolve this issue quickly and move on with more interesting things, you can use the same trick that we used for troubleshooting this issue, but instead of listing the connected hard drives, we can make the systemd service delay its execution by putting a sleep statement in there.

Edit the systemd service again with systemctl edit zfs-import-cache.service and replace the contents with this:

[Service]
ExecStartPre=/usr/bin/sleep 15

This will make the service sleep for 15 seconds before importing the ZFS pool(s). It could be 10, or 30, or whatever value works for you, 15 seconds was chosen here because it is more than enough time for the disks to spin up.

Proper solution (probably)

There are probably better solutions, such as messing around with udev rules or enforcing the delay somewhere else in the ZFS pool import chain of operations, but I have not investigated them at this point. Feel free to contact me with a better solution, and I will update the post accordingly.

Introduction

This is a short overview of one way to take your old video tapes and make digital copies of their contents.

To get started, you will need the following:

• old video tapes that you want to make copies of (VHS and its variants)
• a device that can read analog AV signals and convert them to digital signals
• a device that can play back the tapes (VHS player, camcorder etc.)
  • in this example, we will be using a camcorder that has video playback capabilities.
• a computer with enough free space to store the digital copies
  • for VHS and H.264+MP3, this means roughly 1.5GB of space for 1 hour of footage
• VLC media player
• time and patience

In this example, we will be using the Nedis VGRRU100BK. Not because it is any good, it was just something we had access to.

The process

Step 1: setting up the physical devices

Set up your video playback device so that it can play the cassettes that you are planning to record.

Connect the AV output cables (composite or S-video) to the input of the USB device. For this specific device, it means connecting the appropriate composite AV cables together and connecting them to the USB stick using a proprietary connector.

The end result might look something like this:

Step 2: testing the initial display output

In this example, we will be using VLC to play and record the videos. We are using Linux here, but the process should be very similar on Windows.

Open up VLC.

In the top-left corner, click on Media and select Open Capture device....

You should see a screen similar to this one:

Your computer probably has multiple options for selecting audio and video devices, since microphones and webcams will likely show up here. If you are unsure which one is the correct one, go through each one-by-one until you find the options that work.

In this example, our video capture device is named /dev/video2 and the audio feed for the capture device is hw:4,0.

In the bottom-left corner, you have the option to play back the video signal directly in VLC. You can use this to check which video and audio device is the correct one.

If all goes well, you should be able to see the output in VLC:

Step 3: recording

Once you have found the correct configuration, you can click on the arrow next to the Play button and select the Convert option.

In this view, you can configure deinterlacing, select the video profile, and set where the resulting video file will be saved.

For our purposes, we have chosen to deinterlace the video, since it provided a higher quality output, and we have stuck with MP4 as the output format because it is an acceptable default option for us.

Displaying the output while converting the video did not work for us and that might be related to how VLC accesses the video feed in Linux (only one program can read the stream at a time, it seems).

You can now click Start in VLC to start recording the video. You will also need to hit Play on your video playback device.

Wait until the VHS tape has finished playing and then hit the stop button in VLC.

Step 4: verify results

You can now open the resulting video file and check the results.

If you have additional VHS tapes to record, then repeat step 3.

Troubleshooting notes

There were some challenges to get this setup working.

Not able to read the output from the USB capture device

Being able to read the video signal from the USB capture device turned out to be a headache. The software that comes with the USB capture device (Arcsoft ShowBiz) had issues displaying the video signal and was generally unreliable and crashing.

Other software, like OpenShot and Kdenlive video editors, could not recognize the video capture device.

Since we found that this USB capture device emulates an USB webcam, we tried viewing its signal using a browser or a camera app in the OS (Cheese in Linux, Camera in Windows), but that did not work out either.

We eventually found success with VLC and then chose it for performing our video recording effort. It has multiple advantages over the software shipped with the capture device itself:

• it just works
• free and open source, meaning that you can do your recording on different platforms
• no need for a license key
• up to date (Arcsoft ShowBiz installation files were unchanged since 2012)

VLC truly is a Swiss army knife for media playback and recording.

VHS playback troubles

We initially tried the recording process with a proper VHS player, but while recording the footage and reviewing it later, we noticed that at random points the VHS player had fast-forwarded over portions of the video.

The camcorder did not exhibit this behaviour, so we went with that. However, with the camcorder we could only output mono sound, which means that the resulting video file will only have one sound channel present. This will be especially noticeable when listening to the recorded footage using headphones. To fix this, we used HandBrake to copy the single audio channel to both channels, which resolved that issue for us.

I have a bad habit of testing things whenever a “good” idea pops into my head. This is a short overview of one of them.

The Orange Pi Zero is a SBC (single board computer) that has a slow 32-bit ARM 4 core CPU, 512MB of RAM and no display output. It’s actually quite OK for many tasks, such as reverse proxy (assuming 100Mbit/s is enough for you), low-performance NAS (assuming you are fine with 10-12MB/s file transfer speeds) or a Syncthing relay. Oh, and you can also turn it into a Wi-Fi access point.

Why?

Because Armbian made it really easy to test out and I was interested in seeing what kind of performance an old USB Wi-Fi dongle could offer. This cheap AP could also come into handy in situations where my main access point dies for some reason.

How?

Go to the Armbian device page for your SBC and download the latest image, then write it to your microSD card using the tool of your choice, start your SBC and finish the initial setup.

After all that is done, start armbian-config as root. This tool allows you to do many things over a terminal UI, including a no-hassle method of setting up an Wi-Fi AP.

The option you are looking for is under Network -> Hotspot.

Follow the on-screen instructions to set everything up. After the initial setup is done, make sure to navigate to the same menu again to change the SSID and the Wi-Fi password because the defaults are horribly insecure.

After all this you should be good to go!

Issues and troubleshooting tips

It would not be a proper project if something didn’t go wrong in the process.

Before starting this configuration, run ip a to see if your Wi-Fi device is present. In my case the Orange Pi Zero also has onboard Wi-Fi, but that particular one has poor support, so I use an USB Wi-Fi dongle instead.

If your Wi-Fi device is not showing up, make sure that the correct firmware is present. In my case I had to install firmware-atheros package so that my ath9k_htc based Wi-Fi adapter would start working. dmesg and lsusb -v can help determine what chip your Wi-Fi adapter is using and what driver it needs.

The Orange Pi Zero that I have could set everything up, but it would crash hard with the logs not really showing much. After testing the USB Wi-Fi adapter in all the different USB ports with no success, I opted to use armbian-config menu System -> CPU to limit the CPU clock speed to 480MHz and set the CPU governor to performance to rule out power delivery issues that might be caused by variations in CPU power usage. That seems to have done the trick, because after that change the Wi-Fi AP has been rock solid.

Testing and conclusions

After setting it up and tweaking it a bit I started testing this thing out. The AP had no major issues throughout the day and did not crash.

The performance, however, isn’t anything to write home about. Speed maxes out at 30Mbit/s and is generally lower than that, especially when more than one client is connected. You can do things like Steam Remote Play (stream your game over your home network), but the quality will suffer. It was still quite fun to mess around in GTA V regardless of the image quality and during ~1h of play time there were only 4-5 stutters that were caused by the network.

Would I use this as my main Wi-Fi access point?

Definitely not. Any half-decent router will out-perform this setup.

Can it still be useful?

Definitely. A crappy Wi-Fi AP is better than no AP.

Parts list

• Orange Pi Zero SBC (512MB RAM)
• Orange Pi Zero expansion board (adds two USB 2.0 ports)
• Cheap case for the Orange Pi Zero
• 32GB Sandisk microSD card
• TP-Link WN-722N USB Wi-Fi adapter
• Official Raspberry Pi microUSB power adapter

Update (2020-08-22)

It’s still working and is happily serving as a temporary Wi-Fi access point.

During additional testing I could also hit 40 Mbps speeds when testing in an environment with far fewer competing Wi-Fi access points nearby.

Update (2020-08-30)

It’s still running well and it seems that it won’t be a simple temporary solution after all.

Update (2020-11-30, approximately)

This setup has now been retired. It had 0 issues throughout its lifespan as a Wi-Fi access point, but I replaced it with a more standard Wi-Fi access point so that this board can be used somewhere else.

The setup did surprisingly well and even survived nightly automated updates and restarts, which can be attributed to the good work that the Armbian project has done.