The XZ Backdoor Attack on the Open‑Source Ecosystem

 4 min read

YouTube video ID: aoag03mSuXQ

Source: YouTube video by VeritasiumWatch original video

PDF
  • In the late 1960s AT&T’s Bell Labs created Unix, freely shared among universities and research labs.
  • By the 1980s AT&T began suing Unix clone developers, culminating in a lawsuit against the University of California at Berkeley.
  • This shift pushed companies to require non‑disclosure agreements (NDAs), limiting code sharing.

Richard Stallman’s Response

  • While working at MIT, Stallman faced an NDA when trying to obtain source code for a Xerox 9700 laser printer that kept jamming.
  • The experience convinced him that NDAs “have victims” and inspired him to found the Free Software Foundation (FSF) in 1985.
  • Stallman drafted the GNU General Public License (GPL) and launched the GNU Project to recreate Unix‑like functionality (utilities, shell, kernel) from scratch.

Birth of Linux

  • In 1991 Linus Torvalds, a student at the University of Helsinki, released a free kernel (initially called “Freax,” renamed “Linux” by a friend).
  • Combined with GNU utilities, Linux became a complete operating system.
  • The open‑source model allowed anyone to inspect, modify, and redistribute code, spawning two competing ideologies: proprietary closed‑source and open‑source software.

Linux’s Ubiquity

  • Linux runs on an estimated 30 million devices, from smartphones (Android) to supercomputers, Pentagon systems, nuclear submarines, banks, hospitals, and most internet servers.
  • Its security model relies on the assumption that many eyes make bugs shallow (“Linus’s Law”).

The Fragility of the Open‑Source Ecosystem

  • Open‑source projects depend on thousands of small libraries, each often maintained by a single volunteer.
  • When a widely used library is compromised, every downstream project that links to it can be affected.

XZ: A Critical Compression Library

  • Lasse Collin began maintaining the XZ compression tool (based on LZMA) in 2005, releasing it publicly in 2009.
  • XZ became the default lossless compressor for many Linux distributions and a dependency of OpenSSH.

The Attacker: Jia Tan

  • Jia Tan approached the XZ community in early 2024, offering new features and quickly gaining trust.
  • He positioned himself as a “good coder” and a responsive contributor, convincing maintainers that his changes were harmless.

Three‑Stage Backdoor Plan

  1. Trojan Horse
  2. Modified XZ’s source on GitHub to change the bug‑report contact email and add subtle binary blobs (test data) that would later unpack a hidden payload.
  3. The payload remained invisible because test blobs are rarely inspected.

  4. Goldilocks (Precise Timing)

  5. Targeted the RSA authentication step in OpenSSH, which resides in a shared crypto library, not in OpenSSH itself.
  6. Used an IFUNC resolver to run early during program startup, setting a dynamic audit hook that fires when the linker writes the real address of RSA_Decrypt into the Global Offset Table (GOT).
  7. The exploit had to rewrite the GOT entry after it was filled but before the table became read‑only, a narrow “Goldilocks zone.”

  8. Cat Burglar (Stealthy Access)

  9. The backdoor checks for a secret master key (a miniature cryptographic exchange) before executing.
  10. If the key is present, it grants root access; otherwise, normal authentication proceeds.
  11. The code also erases its own logs and includes numerous safety checks to avoid crashes and detection.

Deployment Attempts

  • Jia Tan pushed the compromised XZ into Fedora pre‑release, Debian testing, and Ubuntu pre‑release builds, aiming for inclusion in the upcoming RHEL 10.
  • He used the backdoor to open a netcat listener on a compromised server, copy files, and obtain root privileges.

Discovery and Fallout

  • Andrés Freund, a Microsoft employee working on PostgreSQL, noticed a 400‑500 ms slowdown on a Debian unstable system in March 2024.
  • Tracing the delay led him to a recent XZ update; Valgrind flagged invalid writes and a memory leak in the malicious code.
  • Andrés reported the issue directly to the Debian security team and a public security mailing list.

  • Red Hat responded by rolling back Fedora and alerting users to revert.

  • The open‑source community examined the XZ source, confirming the backdoor’s mechanisms.

Attribution Speculation

  • The attack involved months of social engineering, pressure on Lasse Collin, and use of free email accounts with minimal footprints.
  • Timestamps on the malicious commits were mostly in UTC+8 (Beijing time), with occasional commits in UTC+2 (Israel/Russia).
  • Some analysts suggest a nation‑state actor (possible Chinese or Russian APT groups) due to the long‑term, low‑return nature of the operation.

Broader Implications

  • The incident shows that open‑source projects can be single points of failure when a critical library is compromised.
  • It also demonstrates that open scrutiny can eventually uncover such backdoors, whereas closed‑source code might hide them indefinitely.
  • The community’s rapid response highlighted both the strengths and weaknesses of the open‑source model: many contributors can detect issues, but the reliance on unpaid volunteers creates opportunities for targeted attacks.

Lessons Learned

  • Dependency hygiene: projects must audit transitive dependencies, especially those that become part of security‑critical software.
  • Supply‑chain monitoring: automated tools (e.g., Valgrind, reproducible builds) are essential for catching subtle malicious changes.
  • Maintainer support: sustainable funding for critical maintainers like Lasse Collin can reduce the risk of burnout and neglect that attackers exploit.

The XZ backdoor episode serves as a cautionary tale about the complexity of modern software supply chains and the need for vigilant, well‑resourced stewardship of the open‑source ecosystem.

  Takeaways

  • The transition from freely shared Unix code to restrictive NDAs in the 1980s motivated the creation of the Free Software Foundation and the GNU GPL to preserve software freedom.
  • Linux, built on GNU utilities and a free kernel, became a ubiquitous operating system powering millions of devices and illustrating the power of open‑source collaboration.
  • The compromise of the widely used XZ compression library demonstrates how a single open‑source component can become a supply‑chain vulnerability affecting many downstream projects.
  • The multi‑stage backdoor attack used precise timing, IFUNC resolution, and stealth techniques to gain root access, but was uncovered through performance monitoring and debugging tools.
  • The incident underscores the importance of dependency hygiene, supply‑chain monitoring, and sustainable maintainer support to protect the open‑source ecosystem.

Frequently Asked Questions

Who is Veritasium on YouTube?

Veritasium is a YouTube channel that publishes videos on a range of topics. Browse more summaries from this channel below.

Does this page include the full transcript of the video?

Yes, the full transcript for this video is available on this page. Click 'Show transcript' in the sidebar to read it.

Helpful resources related to this video

If you want to practice or explore the concepts discussed in the video, these commonly used tools may help.

Links may be affiliate links. We only include resources that are genuinely relevant to the topic.

PDF