On March 29, 2024, Andres Freund, a Microsoft engineer working on PostgreSQL performance, noticed something strange. SSH connections to his Debian testing machine were taking an unusual 500 milliseconds. His investigation led to one of the most sophisticated supply chain attacks ever discovered: a backdoor deliberately inserted into xz-utils, a compression library so fundamental that it is installed on virtually every Linux system in existence.
The xz incident was a watershed moment for open source security. It exposed vulnerabilities not in code, but in the human and organizational systems that sustain critical infrastructure. More than a year later, the question is: what actually changed?
The Incident: What Actually Happened
The attack was a masterclass in patience and social engineering. An individual using the identity “Jia Tan” began contributing to the xz-utils project in 2022. Over nearly two years, they built trust through legitimate contributions — fixing bugs, improving documentation, responding to issues. They gradually became a co-maintainer of the project.
The original maintainer, Lasse Collin, was a solo developer maintaining xz-utils as a side project. He was experiencing burnout, and Jia Tan offered to help shoulder the burden. This is a story familiar to anyone in the open source world — overworked maintainers desperate for help accepting contributions from willing volunteers.
Once trusted, Jia Tan introduced a series of carefully obfuscated changes. The backdoor was not in the xz source code itself but in the test fixtures — binary files that appeared to be compressed test data but actually contained the payload. The build system was modified to extract and inject this payload during compilation, but only under specific conditions that targeted Debian and RPM-based distributions. The result was a backdoor in the SSH authentication path that could potentially allow unauthorized remote access.
The sophistication was remarkable. The malicious code was split across multiple commits over months. The test files that contained the payload passed casual review. The build system modifications were disguised as maintenance improvements. Automated scanners and CI systems did not flag anything suspicious.
What It Revealed About OSS Trust
The xz incident did not reveal a new vulnerability. It made an existing one impossible to ignore: the open source ecosystem runs on trust, and that trust has almost no formal verification mechanisms.
The Maintainer Problem
Critical infrastructure depends on projects maintained by one or two people, often unpaid, often burned out. The xz-utils maintainer was a single individual who had been working on the project for over a decade. When someone offered to help, saying no was not a realistic option. This pattern repeats across thousands of critical open source projects.
The uncomfortable truth is that many of the libraries your production systems depend on are maintained by people who have every rational reason to walk away. The xz attack exploited this reality directly.
The Identity Problem
Jia Tan was a pseudonym. The identity behind the attack has not been publicly identified. Open source contribution platforms — GitHub, GitLab, mailing lists — have minimal identity verification. You can create an account, build a contribution history, and gain commit access to critical projects without ever proving who you are.
This is, in many ways, a feature of open source. Pseudonymous contribution has a long and valuable history. But the xz incident demonstrated the security implications of this model when applied to critical infrastructure.
The Review Problem
The malicious changes passed review because they were designed to. Binary test fixtures are not human-readable. Build system changes are tedious to review in detail. The attack surface was in the areas of a project that receive the least scrutiny — not the core algorithm, but the surrounding infrastructure.
SLSA and Sigstore: Progress on Supply Chain Verification
The industry response to supply chain threats has coalesced around two major initiatives: SLSA (Supply-chain Levels for Software Artifacts) and Sigstore.
SLSA Framework
SLSA defines a graduated set of security levels for software supply chains. At SLSA Level 1, the build process is documented. At Level 2, the build is automated and produces signed provenance. At Level 3, the build environment is hardened and the provenance is verified. At Level 4, the build is hermetic and all dependencies are explicitly declared.
# Example SLSA provenance attestation
{
"_type": "https://in-toto.io/Statement/v1",
"subject": [{
"name": "myapp",
"digest": {"sha256": "abc123..."}
}],
"predicateType": "https://slsa.dev/provenance/v1",
"predicate": {
"buildDefinition": {
"buildType": "https://github.com/slsa-framework/slsa-github-generator",
"externalParameters": {
"source": {
"uri": "git+https://github.com/org/repo@refs/tags/v1.2.3",
"digest": {"sha1": "def456..."}
}
}
},
"runDetails": {
"builder": {
"id": "https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@refs/tags/v2.0.0"
}
}
}
}Adoption of SLSA has increased significantly since the xz incident. GitHub Actions now supports SLSA Level 3 provenance generation natively. Google, npm, and PyPI have all integrated SLSA verification into their ecosystems. But adoption remains uneven — many critical projects still operate at Level 0.
Sigstore
Sigstore provides keyless signing for software artifacts. Instead of managing long-lived signing keys (which themselves become attack targets), Sigstore uses short-lived certificates tied to identity providers like GitHub or Google. This makes it practical for individual developers to sign their releases without the overhead of traditional PKI.
npm now signs all packages with Sigstore by default. PyPI has integrated Sigstore attestations. Container image signing with Sigstore (via cosign) is becoming standard practice in cloud-native deployments. These are meaningful steps, but they primarily verify that an artifact came from a claimed source — they do not verify that the source itself is trustworthy.
SBOMs in Practice
Software Bills of Materials (SBOMs) have moved from theoretical concept to regulatory requirement. The US Executive Order on Cybersecurity mandated SBOMs for software sold to the federal government. The EU Cyber Resilience Act has similar requirements. In practice, most organizations are generating SBOMs but few are using them effectively.
An SBOM tells you what components are in your software. By itself, that is a list, not a security control. The value comes from what you do with the information: monitoring components for newly discovered vulnerabilities, verifying that components have not been tampered with, and ensuring that license obligations are met.
# Generate SBOM with syft
syft packages ./myapp -o spdx-json > sbom.spdx.json
# Scan SBOM for known vulnerabilities
grype sbom:sbom.spdx.json
# Verify component integrity
cosign verify-attestation \
--type spdx \
--certificate-identity builder@github.com \
--certificate-oidc-issuer https://accounts.google.com \
myregistry/myapp:latestThe tooling for SBOM generation has matured considerably. Syft, CycloneDX, and language-specific tools can generate accurate SBOMs for most technology stacks. The gap is in consumption — integrating SBOM data into security workflows, procurement decisions, and incident response processes.
Dependency Review Tools
The xz incident accelerated adoption of dependency review tools that analyze changes to project dependencies before they are merged.
GitHub Dependency Review automatically flags dependency changes in pull requests, highlighting newly introduced vulnerabilities, license changes, and risky version bumps. Socket.dev goes further, analyzing package behavior to detect supply chain attacks — looking for indicators like network access, file system writes, and obfuscated code that suggest malicious intent rather than known CVEs.
Deps.dev from Google provides a comprehensive view of the open source dependency graph, including security advisories, SLSA provenance, and OpenSSF Scorecard ratings. It is becoming the go-to resource for evaluating the trustworthiness of open source packages.
These tools are valuable but have a fundamental limitation: they work best for known attack patterns. The xz attack was specifically designed to evade automated detection. Novel, sophisticated attacks will continue to bypass automated tooling.
Reproducible Builds
Reproducible builds — the ability to independently verify that a binary was built from a specific source — would have made the xz attack detectable. If anyone could have built xz-utils from source and compared the result to the distributed binary, the injected backdoor would have been evident.
The Reproducible Builds project has made significant progress. Debian now tracks reproducibility across its archive — over 95 percent of packages are reproducible. Arch Linux, NixOS, and Guix have similar initiatives. But reproducibility remains challenging for projects with complex build systems, non-deterministic toolchains, or embedded timestamps.
In practice, few organizations verify that the binaries they deploy match the source code they review. This is the gap that reproducible builds aim to close, and adoption is growing but still far from universal.
Social Engineering in Open Source
Perhaps the most unsettling lesson from the xz incident is that the attack vector was not technical — it was social. The attacker gained trust through legitimate, helpful contributions. Pressure was applied to the maintainer through sock puppet accounts complaining about slow releases and suggesting that Jia Tan be given more authority.
This is extraordinarily difficult to defend against. You cannot simultaneously have an open, welcoming community and rigorous identity verification for every contributor. The OpenSSF has published guidelines for maintainer security practices, including recommendations for multi-maintainer approval for critical changes, separation of duties between code and release processes, and vigilance about social pressure campaigns.
But these guidelines add friction and overhead to already-strained maintainers. The fundamental tension remains: the open source model depends on trust and low barriers to contribution, while security demands verification and gatekeeping.
What Organizations Should Actually Do
Given all of the above, what should organizations running on open source software actually do? Here is a prioritized, practical list.
1. Know your critical dependencies. Generate SBOMs and identify which open source components are on critical paths. Not all dependencies carry equal risk — focus on the ones that handle authentication, cryptography, network access, and data processing.
2. Monitor dependency health. Use OpenSSF Scorecard, Deps.dev, and Socket.dev to assess the health and security posture of critical dependencies. Single-maintainer projects with no CI, no signed releases, and no review process are high-risk — not because the maintainer is malicious, but because they are vulnerable.
3. Pin and verify dependencies. Use lock files, hash verification, and SLSA provenance checks. Do not pull latest in production builds. Every dependency update should be a deliberate, reviewed decision.
4. Implement dependency review in CI. Every pull request that modifies dependencies should trigger automated analysis. At minimum, check for known vulnerabilities and license changes. Ideally, use behavioral analysis tools like Socket.dev.
5. Fund your dependencies. If your business depends on an open source project, fund it. This is not altruism — it is risk management. Funded maintainers are less burned out, more able to maintain security practices, and less vulnerable to social engineering attacks from hostile contributors offering to help.
6. Build organizational capability. Someone in your organization should understand your dependency tree, the security posture of critical components, and the process for responding to supply chain incidents. This capability does not exist by default — it must be deliberately built.
What Changed and What Did Not
The xz incident was a catalyst. It accelerated trends that were already underway — SLSA adoption, Sigstore integration, SBOM requirements, dependency review tooling. The security tooling ecosystem is measurably better than it was in March 2024.
What did not change is the fundamental structure of the open source ecosystem. Critical infrastructure still depends on underfunded, overstretched maintainers. Identity verification for contributors remains minimal. Social engineering remains a viable attack vector. And the vast majority of organizations consuming open source software still do not have adequate processes for evaluating and managing supply chain risk.
The xz incident was caught by one engineer noticing a 500-millisecond delay. We were lucky. The question is whether the industry will do enough to ensure that next time, luck is not required.