How Burnout Almost Ended The World

The real danger of unrelenting expectation.

April 09, 2024

For those who don’t know, an incredibly dangerous exploit was recently injected into one of the most prolific software projects in history. This could have allowed some mysterious hacker backdoor access to a vast number of computer systems around the world.

“almost ended the world” is only a minor exaggeration. Banks, large corporations, government entities; these all could have been seriously affected. The theoretical cost is impossible to estimate.

This article doesn’t get into the weeds about the technology. Rather, we’ll chiefly discuss what happened on a human level. At the end of the day, humans failed. This is a story about what we expect from each other and from ourselves.

Setting The Stage

While this article isn’t about the technology, I think it’s useful to explain the gist. Basically, there’s an operating system called Linux. It’s free, open source, and runs on servers and peoples computers when they have too much free time on their hands (looking at you Arch users ;). Essentially, Linux is the internet. All the computers that run your favorite websites are running Linux.

Linux relies on many open source libraries, modules, and other dependencies. You can think of Linux as a tool belt, and there are numerous tools within Linux for doing a variety of things. One of those tools is “XZ utils”, a compression tool that makes data smaller using, what I can only expect, is black magic and a lot of elbow grease.

XZ utils had a backdoor injected into it which effected OpenSSH. The details aren’t important in this article, but what is important is the impact. Any server running Linux, if this exploit actually got out, could have been arbitrarily accessed, meaning the hacker could have done pretty much anything.

Luckily, the exploit was randomly found by a microsoft engineer who noticed something fishy. If you’re really curious about the technical details a lot of information is surfacing. This video is a good place to start.

I don’t really care about the technical details past this point. I’m sure it’s fascinating, but if you look at my articles I think you’ll find my plate is full with data science, thank you very much.

What I can relate to is how this exploit happened in the first place. The passion, the expectation, and the pain.

How The Internet Really Works

Some people think of the internet is made up of big companies making a lot of money. Those companies employ a bunch of well paid engineers, and those engineers are wizards in ivory towers which rangle pixies flying through cables under our feet.

That conceptualization is true, but only for a small piece of the internet. There’s another group of wizards which reside deep in the shadows, working on secret incantations for little more reward than the pleasure of the craft. These are open source maintainers.

I mentioned that Linux, XZ Utils, and other tools are “open source”. That is to say, they are public projects which are licensed to and maintained by the public. Anyone can look at the linux source code, and anyone can work on it and suggest feedback. This is the Linux kernel, the core of Linux:

In this link you can see the 15,000+ people who have worked to maintain the Linux kernel. Some might be working in a company, or are paid by some foundation or grant, but many are likely volunteers, working on the bedrock of the internet for fun.

Not all projects enjoy this level of support, however. This is the github repo for XZ Utils, the library which was compromised with the back door attack:

This library, which is used to handle the compression for a lions share of the internet, only has 30 contributors. There’s a famous meme that describes this reality in a nutshell.

Amazon, Netflix, Tesla, Google, I wonder how much of their infrastructure runs on open source code. In fact, I bet XZ Utils played a part in you being able to read this article right now.

The Culture of Open Source

The ideal of open source is that anyone can contribute to a project and make it better. The reality is a bunch of people using your code and asking you to fix it when it doesn’t work. The creator of Linux has a famous reputation for pretty salty one liners for that exact reason.

Those that can, do. Those that can’t, complain.

- Linus Torvald

This has been an known and serious problem in the open source community. Imagine having a fun side project that grows into a cornerstone of the internet. As these projects grow organically into the fundamental building blocks of the internet, many maintainers feel a sense of responsibility. Instead of dropping the ball, saying “screw this”, and letting ungrateful users sort their own messes out, they work tirelessly and stressfully to maintain and improve their tool for the benefit of all who use it.

That fact has been a known problem for a long time. With the XZ Utils exploit, it became a target for attack.

Social Engineering Behind The XZ Utils Attack.

Basically, some hacker (or group of hackers, perhaps working together) Intentionally caused the core maintainer of XZ Utils to burn out. They compromised his mental and emotional health, manipulated him, and used that as an inroad for gaining the trust necessary to inject the backdoor into XZ Utils.

First, they bombarded him with feature requests, and asked the core maintainer why no progress was being made?

User: “Is XZ for Java still maintained? I asked a question here a week ago and have not heard back” — Source

Which, naturally, made the core maintainer feel apologetically behind.

Maintainer: “Yes, by some definition at least, like if someone reports a bug it will get fixed. Development of new features definitely isn’t very active. :-(” — Source

Another user chimes in, telling the core maintainer that another maintainer “Jia Tan” has been incredibly helpful, to which the maintainer replies

Maintainer: “Jia Tan has helped me … and he might have a bigger role in the future … It’s clear that my resources are too limited … so something has to change in the long term.” — Source

And other users chime in, voicing their opinion.

User: “Progress will not happen until there is new maintainer. … The current maintainer lost interest or doesn’t care to maintain anymore. It is sad to see for a repo like this.” — Source

Ok, if you’re a software developer you might not be surprised by this, but take a step back and look at what’s happening here. Imagine you have a pet project that grows into a popular tool, and massive companies rely on your work to get things done on a daily basis. Any other industry would be expecting royalties, sponsorships, maybe a brand deal. In open source software, it’s “tisk tisk tisk, you’re not working for free hard enough”.

This is the maintainers response to all this criticism.

Maintainer: “I haven’t lost interest but my ability to care has been fairly limited

mostly due to longterm mental health issues but also due to some other

things. Recently I’ve worked off-list a bit with Jia Tan on XZ Utils and

perhaps he will have a bigger role in the future, we’ll see.

It’s also good to keep in mind that this is an unpaid hobby project.

Anyway, I assure you that I know far too well about the problem that

not much progress has been made. The thought of finding new maintainers

has existed for a long time too as the current situation is obviously

bad and sad for the project.” — Source

I’ll spare you the pain, but the users don’t let up on this poor guy, they double down. They say things like “I am sorry about your mental health issues… but the community desires more”, etc. etc. They blast the poor guy. They berate him, make him feel inadequate, and pressure him.

Why, you might ask? Jia Tan

Jia Tan was another maintainer, helping the core maintainer of XZ Utils with bugs, features, and all sorts of little improvements. Jia was Probably a godsend to the core maintainer. The problem was, Jia Tan was the hacker. As this mounting pressure by expectant users grew, the core maintainer leaned on Jia Tan to help maintain the library.

The thing is, The hackers did this for two years. They wore down this poor guy for two… years… A bunch of random users berated him for not working fast enough, and this “Jia Tan” guy helped him pick up the slack. They were all working together the whole time just to get Jia Tan more control over the library. Once Jia Tan got sufficient control, he slipped a back door under the rug, which was then almost used as a dependency within Linux.

I’ll let you reflect on how diligent you would have been under the circumstances; being bullied by people you’re working for for free for two years. How quickly would your favorite side hobby lose its luster? I’ll be honest, I’d flip the internet the bird and drop that like a hot rock. You guys have fun.

The Implications of The XZ Utils Attack

I’m sure security researchers and open source project maintainers are thinking hard about this problem. Some of the things I’m thinking about are this:

  • The way software is made, with libraries on libraries on libraries, is looking a lot less cute

  • Being rude to each other online has transcended from an annoyance to a security concern

  • If we don’t do something about this, and find a way to support more open source maintainers, this will happen again (If it hasn’t already).

  • It’s important to only work for free if you really, really want to. Otherwise, you might find yourself forced into an uncomfortable responsibility. People love asking for more; hackers and users alike.

The hackers injected the backdoor in a really clever way that made it very hard to detect. I mention this because I don’t want the core maintainer of XZ Utils to catch any more flack. He did his best, I think he’s had enough.

Follow For More!

I describe papers and concepts in the ML space, with an emphasis on practical and intuitive explanations.

Sources: