# Open Source Is a Dumpster Fire Right Now

Source: https://www.testifysec.com/blog/litellm-1-107-0-backdoor-replay/

Author: TestifySec

Published: 2026-10-21

litellm 1.107.0 was built and published by its own release pipeline, so every signature check passed. A live replay with CI/lock, and the producer and consumer policies that refuse it.

*Written with an AI from a real CI/lock run, and edited and narrated by Cole Kennedy and the TestifySec team.* 

Open source is a dumpster fire right now. Every conference talk this year is about software factories, where agent fleets open a hundred PRs a day and humans review whatever's left. Meanwhile, on 21 October 2026, a backdoored release of one of the most installed AI libraries on PyPI shipped through what looks like its own release pipeline. The checks that would have caught it are three fields long.

 

A software factory runs on trust. You have to believe that what it builds is what it says it is. Open source doesn't have that yet, and somebody has to build it before the factory. Skip that step and the factory just ships the backdoor faster.

 

At TestifySec, this is the problem we've worked on since 2021. We built Witness and Archivista, the attestation tools we donated to the CNCF's in-toto project, where they became subprojects in January 2024. Cole Kennedy, our CEO, contributed to the CNCF's Software Supply Chain Best Practices paper and its Secure Software Factory reference architecture. Frederick Kautz, our co-founder, co-authored NIST SP 800-204D, NIST's guidance on building supply chain security into CI/CD pipelines. We helped write the reference architecture for a software factory, and we're telling you where it starts.

 

## What happened to litellm 1.107.0?

 

This is a build pipeline compromise. SLSA's threat model calls it building from modified source: the artifact came out of the project's real release pipeline, but from source that doesn't match anything in the public repository. Its closest relatives are SolarWinds and xz. In 2020, attackers inside SolarWinds' build environment injected SUNBURST into Orion during compilation, and SolarWinds signed the result with its own certificate. We [wrote up SUNBURST and the SUNSPOT implant](https://testifysec.com/blog/software-supply-chain-history-of-failure/) in 2021, and [SolarSploit](https://github.com/testifysec/solarsploit) is our red-team tool that reproduces that CI compromise. In 2024, the xz-utils release tarballs carried a build script that wasn't in the git repository, and it planted a backdoor in liblzma. litellm 1.107.0 has a piece of each. The real pipeline did the building, as at SolarWinds, and the shipped artifact holds files the repository never saw, as with xz.

 

`litellm==1.107.0` went up on PyPI around 7:40 a.m. Eastern: an sdist and seven wheels covering Linux, macOS, and Windows. PyPI pulled it by 9:40. [SafeDep's write-up](https://safedep.io/litellm-pypi-compromise-credential-stealer-backdoor/) has the detail. It's the second litellm compromise this year. The [March incident](https://www.testifysec.com/blog/cilock-litellm-supply-chain-attack) also ran code at Python startup. The payload has three parts.

 **Startup hook.** The wheels drop `cook.pth`, `cook.py` and `sitecustomize.py` into `site-packages`. A `.pth` file runs at interpreter startup, so *every* Python process on the box runs the payload. That includes pip, your IDE's language server, and your CI jobs. Nothing has to `import litellm`. **Stealer.** It takes SSH keys, cloud credentials, kubeconfig, browser passwords, shell history, CI secrets, Vault tokens, the configs for basically every AI coding tool, and the LiteLLM proxy's own table of provider API keys. It encrypts the lot and POSTs it to `www.litellm[.]dev`. **Backdoor.** `sitecustomize.py` forks a daemon that pulls a C2 domain from a GitHub repo every hour and `exec()`s whatever it serves. 

## Why didn't a signature catch it?

 

It looked legitimate. SafeDep's assessment (moderate to high confidence; the root cause isn't confirmed) is that BerriAI's private release pipeline built these wheels itself from tampered source. It used the same maturin version as the real releases, every file had a valid `RECORD` entry, and the release bot published the GitHub release on its usual ten-minute delay.

 

So "is it signed by the real pipeline?" is the wrong question. If the real pipeline built it, the signature is genuine. A valid signature on a backdoor is still a backdoor. (PyPI had no attestations for this release anyway, or for the two before it.)

 

The useful questions are about *what the pipeline built*. Each one had an answer for 1.107.0:

 

| Question | Answer for 1.107.0 |
|-----------------------------------------------|-------------------------|
| Was the build tree clean? | no |
| Was the built commit on `main`, reviewed? | no public commit has it |
| Does the tag match the version it ships? | tag says 1.108.0 |

 

The sdist matches `main` at `a8113b0`, plus an edited `pyproject.toml` and an extra `cook/` directory. The edit set the version back to 1.107.0, pinned the companion packages to rc2 values so a dependency diff looks normal, and added one line: `data = "cook"`. That line tells maturin to copy `cook/` into the wheel. Meanwhile the tag `v1.107.0` points at `66c8f5b`, whose `pyproject.toml` says `1.108.0`.

 

Any one of those checks refuses this release, and none of them needs machine learning or a threat feed. The build machine already knows every answer.

 

## Would attestation have stopped it?

 

To find out, we rebuilt the attack in a toy repository and ran both policies below against real [cilock](https://cilock.dev/) evidence. There's a tag `v1.107.0` on a commit whose `pyproject.toml` says 1.108.0, a later untagged `main`, and the same edit: the version set back, `data = "cook"`, and an untracked `cook/` with a `.pth` in it. The pipeline builds the wheel under `cilock run -a git` and signs it with its real key. Every command in the video is a live run.

 [![Video: It passed every signature check. The litellm 1.107.0 backdoor, replayed with CI/lock](https://www.testifysec.com/media/blog/litellm-1107-replay-thumbnail.jpg)](https://www.testifysec.com/media/blog/litellm-1107-replay.mp4) *Three minutes, narrated by Cole Kennedy. Toy repository, not BerriAI's. Signing uses local ed25519 keys, one for the pipeline and one for the maintainer. [Play the video](https://www.testifysec.com/media/blog/litellm-1107-replay.mp4).* 

Here's what came back:

 

| Build | Producer gate | Consumer gate |
|----|----|----|
| 1.106.2: clean tagged tree, approved by the maintainer | pass | pass |
| 1.107.0 as it happened: built from `main` with the edit uncommitted | refused: `cook/` untracked, `pyproject.toml` modified, no release tag | refused, same reasons |
| 1.107.0, smarter attacker: commits the edit and moves the tag | refused: no maintainer approval of these bytes | refused, same reason |
| 1.107.0 as PyPI served it, with no evidence | n/a | refused: nothing to verify |
| 1.106.2 rebuilt with the approved wheel plus a second wheel nobody approved | refused: more than one artifact | refused: the approval doesn't name the second wheel |

 

A pass only means something if the rules actually run, so we checked that too. We swapped the approval rule for one that always denies, re-signed the policy, and re-ran 1.106.2. It failed with the swapped rule's message, so the approval rule really runs.

 

## What does the producer policy check?

 

The producer policy is the release gate. It runs in the publisher's pipeline after the build and before `twine upload`, and it wants two keys. The pipeline signs `release-build`. A maintainer signs `release-approval`, whose product is the wheel itself, so the approval covers the exact bytes.

 

{
 "expires": "2027-10-21T00:00:00Z",
 "publickeys": { "": {…}, "": {…} },
 "steps": {
 "release-build": {
 "functionaries": \[{ "type": "publickey", "publickeyid": "" }\],
 "attestations": \[
 { "type": "https://aflock.ai/attestations/git/v0.1",
 "regopolicies": \["build*clean", "build*tagged"\] },
 { "type": "https://aflock.ai/attestations/product/v0.3",
 "regopolicies": \["one\_artifact"\] }
 \]
 },
 "release-approval": {
 "functionaries": \[{ "type": "publickey", "publickeyid": "" }\],
 "attestationsFrom": \["release-build"\],
 "attestations": \[
 { "type": "https://aflock.ai/attestations/git/v0.1",
 "regopolicies": \["approval\_binds"\] },
 { "type": "https://aflock.ai/attestations/product/v0.3",
 "regopolicies": \["approval\_bytes"\] }
 \]
 }
 }
 }

 

The real file carries each rego module as base64. We show them by name here and leave out the keys.

 

The tree checks read the `git` attestor, which records per-file worktree status and the refs on the built commit:

 

# build\_clean: nothing modified, nothing untracked.
 # A clean tree has no status key at all, so read it with a default.
 deny\[msg\] {
 status := object.get(input, "status", {})
 paths := \[p | status\[p\]\]
 p := paths\[\_\]
 msg := sprintf("tree not clean at build time: %s is %s", \[p, status\[p\].worktree\])
 }

 

# build\_tagged: a release is built from a tagged commit, not whatever main is.
 release*tag { r := input.refs\[*\]; startswith(r, "refs/tags/v") }

 

deny\[msg\] {
 not release\_tag
 msg := sprintf("built commit %s carries no release tag", \[substring(input.commithash, 0, 12)\])
 }

 

The approval step reaches back into the build's evidence. It checks that the maintainer approved the same commit and the same bytes, and that the wheel's version matches the tag:

 

# approval\_binds (on the approval's git attestation)
 build\_git := input.steps\["release-build"\]\["https://aflock.ai/attestations/git/v0.1"\]

 

deny\[msg\] {
 input.attestation.commithash != build\_git.commithash
 msg := sprintf("approval is for %s, build is %s", \[input.attestation.commithash, build\_git.commithash\])
 }

 

deny\[msg\] {
 v := artifact*versions\[*\] # parsed from the build's product filenames
 not tag\_versions\[v\] # parsed from refs/tags/v\* on the built commit
 msg := sprintf("artifact version %s does not match the built commit's tag", \[v\])
 }

 

# approval\_bytes (on the approval's product attestation)
 # The two digest sets must be equal: nothing approved that wasn't built,
 # and nothing built that wasn't approved.
 build\_leaves := input.steps\["release-build"\]\["https://aflock.ai/attestations/product/v0.3"\].leaves
 build*digests\[d\] { d := build*leaves\[\_\].fileDigest }
 approved*digests\[d\] { d := object.get(input.attestation, "leaves", \[\])\[*\].fileDigest }

 

deny\[msg\] {
 leaf := object.get(input.attestation, "leaves", \[\])\[\_\]
 not build\_digests\[leaf.fileDigest\]
 msg := sprintf("approved %s (%s) is not what release-build produced", \[leaf.path, leaf.fileDigest\])
 }

 

deny\[msg\] {
 leaf := build*leaves\[*\]
 not approved\_digests\[leaf.fileDigest\]
 msg := sprintf("release-build produced %s (%s), which the approval does not name", \[leaf.path, leaf.fileDigest\])
 }

 

## What does the consumer policy check?

 

This one runs on your side, before `pip install`. You pin the publisher's two keys out of band. It re-asserts the tree checks instead of trusting that the producer's gate ran, because a bypassed producer gate is exactly the attack. It drops `one_artifact`, which is the publisher's hygiene and not yours, and it expires in three months so the key pins get looked at again.

 

{
 "expires": "2027-01-21T00:00:00Z",
 "publickeys": { /* the publisher's pipeline and maintainer keys, pinned */ },
 "steps": {
 "release-build": { /* pipeline key; git: build*clean, build*tagged */ },
 "release-approval": { /\* maintainer key; attestationsFrom release-build;
 git: approval*binds; product: approval*bytes \*/ }
 }
 }

 

Wire it in front of the install. Download wheels only, since an sdist can run build-backend code just to compute its metadata. Verify by digest, then install from the verified directory with no index:

 

pip download litellm==1.107.0 --no-deps --only-binary=:all: -d ./wheels
 cilock verify -p consumer.policy.signed.json -k consumer.pub -f ./wheels/litellm-1.107.0-\*.whl \\
 && pip install --no-index --find-links ./wheels litellm==1.107.0

 

# 1.107.0 shipped with no attestation at all, so verify exits 1 and pip never runs.

 

Two parts of this are still glue. pip has no hook for it, so you'd be wrapping every dependency yourself, not just the one in the news. And the publisher has to ship the evidence somewhere you can fetch it. PyPI supports attestations, but 1.107.0, 1.106.2, and 1.107.0rc2 carried none.

 

## What doesn't this prove?

 

If the attacker owns the build host, the build host's attestations say whatever the attacker wants. An attacker on a compromised runner can make the `git` attestor report a clean tree. That's why the approval step matters most: a second key that isn't the build's, approving the exact bytes. One compromised machine can't forge both signatures. That only holds if the maintainer's key lives somewhere the pipeline can't reach, and if the maintainer actually looks before signing.

 

And producer-side evidence protects nobody who doesn't check it. The registry doesn't enforce it and the installer doesn't ask for it. Until one of them does, verification is on you.

 

## What does a software factory need first?

 

A software factory takes in code nobody read and ships code nobody reads. That only works on trust. TestifySec runs one, and it holds up because every step leaves evidence a policy checks: what went in, what came out, and who approved it.

 

Without that evidence, more automation means a bigger blast radius. The more code ships without a human reading it, the more the release gate has to be mechanical, and the less anyone can rely on a person noticing that a tag says 1.108.0 on a 1.107.0 release. Nobody noticed this one. It shipped on a weekday morning to one of the most installed AI packages there is.

 

You build that trust out of boring checks. Is the tree clean? Is the built commit the tagged one? Did a second key approve these bytes?

 

Build the trust first. Then build the factory.

 

## What if you installed 1.107.0?

 

Look for `cook.pth`, `cook.py` or an unexpected `sitecustomize.py` in any `site-packages`. Also check `/tmp/.pc_once_72129118c59f5636c0bb`, `/tmp/remote_code_loader.lock`, and a detached Python process with PPID 1 and working directory `/`. If any of those turn up, assume the attacker has everything that host could read. Rotate the credentials. Uninstalling isn't enough. SafeDep's post has the full IOC list and the artifact hashes.

 

cilock is open source under the Apache 2.0 license. The source is in [aflock-ai/rookery](https://github.com/aflock-ai/rookery), and the releases are in [aflock-ai/cilock](https://github.com/aflock-ai/cilock). Everything in this post runs on it.

 

Details of the compromise come from SafeDep's analysis as of 21 October 2026. BerriAI hasn't published a root cause, and some of this may change once they do.