Open source is a dumpster fire right now. Every conference talk this year is about software factories, where agent fleets open a hundred PRs a day and humans review whatever's left. Meanwhile, on 21 October 2026, a backdoored release of one of the most installed AI libraries on PyPI shipped through what looks like its own release pipeline. The checks that would have caught it are three fields long.
A software factory runs on trust. You have to believe that what it builds is what it says it is. Open source doesn't have that yet, and somebody has to build it before the factory. Skip that step and the factory just ships the backdoor faster.
At TestifySec, this is the problem we've worked on since 2021. We built Witness and Archivista, the attestation tools we donated to the CNCF's in-toto project, where they became subprojects in January 2024. Cole Kennedy, our CEO, contributed to the CNCF's Software Supply Chain Best Practices paper and its Secure Software Factory reference architecture. Frederick Kautz, our co-founder, co-authored NIST SP 800-204D, NIST's guidance on building supply chain security into CI/CD pipelines. We helped write the reference architecture for a software factory, and we're telling you where it starts.
What happened to litellm 1.107.0?
This is a build pipeline compromise. SLSA's threat model calls it building from modified source: the artifact came out of the project's real release pipeline, but from source that doesn't match anything in the public repository. Its closest relatives are SolarWinds and xz. In 2020, attackers inside SolarWinds' build environment injected SUNBURST into Orion during compilation, and SolarWinds signed the result with its own certificate. We wrote up SUNBURST and the SUNSPOT implant in 2021, and SolarSploit is our red-team tool that reproduces that CI compromise. In 2024, the xz-utils release tarballs carried a build script that wasn't in the git repository, and it planted a backdoor in liblzma. litellm 1.107.0 has a piece of each. The real pipeline did the building, as at SolarWinds, and the shipped artifact holds files the repository never saw, as with xz.
litellm==1.107.0 went up on PyPI around 7:40 a.m. Eastern: an sdist and seven wheels covering Linux, macOS, and Windows. PyPI pulled it by 9:40. SafeDep's write-up has the detail. It's the second litellm compromise this year. The March incident also ran code at Python startup. The payload has three parts.
cook.pth, cook.py and sitecustomize.py into site-packages. A .pth file runs at interpreter startup, so every Python process on the box runs the payload. That includes pip, your IDE's language server, and your CI jobs. Nothing has to import litellm.
Stealer. It takes SSH keys, cloud credentials, kubeconfig, browser passwords, shell history, CI secrets, Vault tokens, the configs for basically every AI coding tool, and the LiteLLM proxy's own table of provider API keys. It encrypts the lot and POSTs it to www.litellm[.]dev.
Backdoor. sitecustomize.py forks a daemon that pulls a C2 domain from a GitHub repo every hour and exec()s whatever it serves.
Why didn't a signature catch it?
It looked legitimate. SafeDep's assessment (moderate to high confidence; the root cause isn't confirmed) is that BerriAI's private release pipeline built these wheels itself from tampered source. It used the same maturin version as the real releases, every file had a valid RECORD entry, and the release bot published the GitHub release on its usual ten-minute delay.
So "is it signed by the real pipeline?" is the wrong question. If the real pipeline built it, the signature is genuine. A valid signature on a backdoor is still a backdoor. (PyPI had no attestations for this release anyway, or for the two before it.)
The useful questions are about what the pipeline built. Each one had an answer for 1.107.0:
| Question | Answer for 1.107.0 |
|-----------------------------------------------|-------------------------|
| Was the build tree clean? | no |
| Was the built commit on main, reviewed? | no public commit has it |
| Does the tag match the version it ships? | tag says 1.108.0 |
The sdist matches main at a8113b0, plus an edited pyproject.toml and an extra cook/ directory. The edit set the version back to 1.107.0, pinned the companion packages to rc2 values so a dependency diff looks normal, and added one line: data = "cook". That line tells maturin to copy cook/ into the wheel. Meanwhile the tag v1.107.0 points at 66c8f5b, whose pyproject.toml says 1.108.0.
Any one of those checks refuses this release, and none of them needs machine learning or a threat feed. The build machine already knows every answer.
Would attestation have stopped it?
To find out, we rebuilt the attack in a toy repository and ran both policies below against real cilock evidence. There's a tag v1.107.0 on a commit whose pyproject.toml says 1.108.0, a later untagged main, and the same edit: the version set back, data = "cook", and an untracked cook/ with a .pth in it. The pipeline builds the wheel under cilock run -a git and signs it with its real key. Every command in the video is a live run.
Three minutes, narrated by Cole Kennedy. Toy repository, not BerriAI's. Signing uses local ed25519 keys, one for the pipeline and one for the maintainer. Play the video.
Here's what came back:
| Build | Producer gate | Consumer gate |
|----|----|----|
| 1.106.2: clean tagged tree, approved by the maintainer | pass | pass |
| 1.107.0 as it happened: built from main with the edit uncommitted | refused: cook/ untracked, pyproject.toml modified, no release tag | refused, same reasons |
| 1.107.0, smarter attacker: commits the edit and moves the tag | refused: no maintainer approval of these bytes | refused, same reason |
| 1.107.0 as PyPI served it, with no evidence | n/a | refused: nothing to verify |
| 1.106.2 rebuilt with the approved wheel plus a second wheel nobody approved | refused: more than one artifact | refused: the approval doesn't name the second wheel |
A pass only means something if the rules actually run, so we checked that too. We swapped the approval rule for one that always denies, re-signed the policy, and re-ran 1.106.2. It failed with the swapped rule's message, so the approval rule really runs.
What does the producer policy check?
The producer policy is the release gate. It runs in the publisher's pipeline after the build and before twine upload, and it wants two keys. The pipeline signs release-build. A maintainer signs release-approval, whose product is the wheel itself, so the approval covers the exact bytes.
{
"expires": "2027-10-21T00:00:00Z",
"publickeys": { "
"steps": {
"release-build": {
"functionaries": [{ "type": "publickey", "publickeyid": "
"attestations": [
{ "type": "https://aflock.ai/attestations/git/v0.1",
"regopolicies": ["buildclean", "buildtagged"] },
{ "type": "https://aflock.ai/attestations/product/v0.3",
"regopolicies": ["one_artifact"] }
]
},
"release-approval": {
"functionaries": [{ "type": "publickey", "publickeyid": "
"attestationsFrom": ["release-build"],
"attestations": [
{ "type": "https://aflock.ai/attestations/git/v0.1",
"regopolicies": ["approval_binds"] },
{ "type": "https://aflock.ai/attestations/product/v0.3",
"regopolicies": ["approval_bytes"] }
]
}
}
}
The real file carries each rego module as base64. We show them by name here and leave out the keys.
The tree checks read the git attestor, which records per-file worktree status and the refs on the built commit:
# build_clean: nothing modified, nothing untracked.
# A clean tree has no status key at all, so read it with a default.
deny[msg] {
status := object.get(input, "status", {})
paths := [p | status[p]]
p := paths[_]
msg := sprintf("tree not clean at build time: %s is %s", [p, status[p].worktree])
}
# build_tagged: a release is built from a tagged commit, not whatever main is.
releasetag { r := input.refs[]; startswith(r, "refs/tags/v") }
deny[msg] {
not release_tag
msg := sprintf("built commit %s carries no release tag", [substring(input.commithash, 0, 12)])
}
The approval step reaches back into the build's evidence. It checks that the maintainer approved the same commit and the same bytes, and that the wheel's version matches the tag:
# approval_binds (on the approval's git attestation)
build_git := input.steps["release-build"]["https://aflock.ai/attestations/git/v0.1"]
deny[msg] {
input.attestation.commithash != build_git.commithash
msg := sprintf("approval is for %s, build is %s", [input.attestation.commithash, build_git.commithash])
}
deny[msg] {
v := artifactversions[] # parsed from the build's product filenames
not tag_versions[v] # parsed from refs/tags/v* on the built commit
msg := sprintf("artifact version %s does not match the built commit's tag", [v])
}
# approval_bytes (on the approval's product attestation)
# The two digest sets must be equal: nothing approved that wasn't built,
# and nothing built that wasn't approved.
build_leaves := input.steps["release-build"]["https://aflock.ai/attestations/product/v0.3"].leaves
builddigests[d] { d := buildleaves[_].fileDigest }
approveddigests[d] { d := object.get(input.attestation, "leaves", [])[].fileDigest }
deny[msg] {
leaf := object.get(input.attestation, "leaves", [])[_]
not build_digests[leaf.fileDigest]
msg := sprintf("approved %s (%s) is not what release-build produced", [leaf.path, leaf.fileDigest])
}
deny[msg] {
leaf := buildleaves[]
not approved_digests[leaf.fileDigest]
msg := sprintf("release-build produced %s (%s), which the approval does not name", [leaf.path, leaf.fileDigest])
}
What does the consumer policy check?
This one runs on your side, before pip install. You pin the publisher's two keys out of band. It re-asserts the tree checks instead of trusting that the producer's gate ran, because a bypassed producer gate is exactly the attack. It drops one_artifact, which is the publisher's hygiene and not yours, and it expires in three months so the key pins get looked at again.
{
"expires": "2027-01-21T00:00:00Z",
"publickeys": { / the publisher's pipeline and maintainer keys, pinned / },
"steps": {
"release-build": { / pipeline key; git: buildclean, buildtagged / },
"release-approval": { /* maintainer key; attestationsFrom release-build;
git: approvalbinds; product: approvalbytes */ }
}
}
Wire it in front of the install. Download wheels only, since an sdist can run build-backend code just to compute its metadata. Verify by digest, then install from the verified directory with no index:
pip download litellm==1.107.0 --no-deps --only-binary=:all: -d ./wheels
cilock verify -p consumer.policy.signed.json -k consumer.pub -f ./wheels/litellm-1.107.0-*.whl \
&& pip install --no-index --find-links ./wheels litellm==1.107.0
# 1.107.0 shipped with no attestation at all, so verify exits 1 and pip never runs.
Two parts of this are still glue. pip has no hook for it, so you'd be wrapping every dependency yourself, not just the one in the news. And the publisher has to ship the evidence somewhere you can fetch it. PyPI supports attestations, but 1.107.0, 1.106.2, and 1.107.0rc2 carried none.
What doesn't this prove?
If the attacker owns the build host, the build host's attestations say whatever the attacker wants. An attacker on a compromised runner can make the git attestor report a clean tree. That's why the approval step matters most: a second key that isn't the build's, approving the exact bytes. One compromised machine can't forge both signatures. That only holds if the maintainer's key lives somewhere the pipeline can't reach, and if the maintainer actually looks before signing.
And producer-side evidence protects nobody who doesn't check it. The registry doesn't enforce it and the installer doesn't ask for it. Until one of them does, verification is on you.
What does a software factory need first?
A software factory takes in code nobody read and ships code nobody reads. That only works on trust. TestifySec runs one, and it holds up because every step leaves evidence a policy checks: what went in, what came out, and who approved it.
Without that evidence, more automation means a bigger blast radius. The more code ships without a human reading it, the more the release gate has to be mechanical, and the less anyone can rely on a person noticing that a tag says 1.108.0 on a 1.107.0 release. Nobody noticed this one. It shipped on a weekday morning to one of the most installed AI packages there is.
You build that trust out of boring checks. Is the tree clean? Is the built commit the tagged one? Did a second key approve these bytes?
Build the trust first. Then build the factory.
What if you installed 1.107.0?
Look for cook.pth, cook.py or an unexpected sitecustomize.py in any site-packages. Also check /tmp/.pc_once_72129118c59f5636c0bb, /tmp/remote_code_loader.lock, and a detached Python process with PPID 1 and working directory /. If any of those turn up, assume the attacker has everything that host could read. Rotate the credentials. Uninstalling isn't enough. SafeDep's post has the full IOC list and the artifact hashes.
cilock is open source under the Apache 2.0 license. The source is in aflock-ai/rookery, and the releases are in aflock-ai/cilock. Everything in this post runs on it.
Details of the compromise come from SafeDep's analysis as of 21 October 2026. BerriAI hasn't published a root cause, and some of this may change once they do.