Do you have a need for speed really fast disk performance?

Are you unwilling or unable to buy/rent a machine with fast, NVMe-based storage?

Are you OK with potential data corruption?

Then cache=unsafe is the solution to all your problems!

We had an interesting conundrum at work recently. Our platform does not use a lot of resources, but there are bursts of activity that require a lot of CPU and performant disk IO from our infrastructure. This was previously handled by manually starting some expensive AWS EC2 instances to cope with the load, but this manual process was error-prone due to the human factor (which did end up causing an actual production outage once), and AWS is stupidly expensive for what you get in return.

Around this time I also learned about a Proxmox server that we were underutilizing. My goal was to investigate the resources that we had available and to ensure that we didn’t have to think about taking any manual actions while at the same time not relying on AWS and its expensive resources.

I set up a few VM-s on the Proxmox machine, and did some testing. CPU, memory, that was all fine, but the IO-bound workloads that we had to run during those bursty periods would still be relatively slow. Not much slower than the main infrastructure provider that we were using, but slow enough for a beefy machine to not be able to handle more than a few parallel IO-heavy workloads running at the same time.

We exhausted a few other wild-ass ideas during the investigation:

  • Docker on a RAM-backed storage drive
    • online resources did not inspire confidence in this working well, so we didn’t try this
  • optimizing the workload to not be IO-heavy
    • unsuccessful after spending a few hours on it, the high IO was a consequence of making an intentional trade-off to reduce CPU load, and the IO requirement was much more manageable
  • putting certain folders in the container itself on RAM-backed storage
    • highly container specific, and did not yield the desired results

Then one day I was browsing around Proxmox and noticed an interesting option on the virtual storage drives: setting the cache mode to unsafe. With this one trick, your VM will see really fast disk speeds up to a certain point, and it’s invisible from the perspective of your workloads, no customization needed. In a way, this is like one of the RAM-backed storage options, but for the whole VM.

The major trade-off is that an unexpected shutdown of the VM or the VM host will likely result in data corruption. This is because you’re writing everything to memory first, and eventually the writes will end up on persistent storage, whenever the disks catch up with you. Something happens while changes are in memory, and they are lost.

In our case, the data corruption risk is completely OK, as the workloads are ephemeral, the results of the work are sent to another machine immediately after completion, and the configuration of the machine is largely automated with Ansible.

One instance of our workload would usually result in writing 50 MB to disk, and we observed about 300-500 IOPS of performance from HDD-backed storage. The disks were not able to handle more than one at a time if we cared about execution time.

With the unsafe trick, and on some relatively old hardware (assume DDR3 memory), we saw numbers as high as 15K IOPS and disk throughput of 500+ MB/s. This was more than enough to handle peak loads, and the resources were always on and available on a rented server with a stable price that compared extremely well to AWS.

Cloud service providers have their benefits, sure, but when all you need is raw speed and configurability to make it happen, then owning a physical Linux server (or a few of them for redundancy) is a no-brainer, slam-dunk decision, as long as you have someone in your team that knows how to manage one. Since you’re working with Linux VM-s already in the cloud, then you already have that person in your team, don’t you? :)