pii-leak-benchmark tests any OpenAI-compatible SSE gateway in about a minute. Its only third-party Python dependency is httpx. You do not need to install one gateway to test another.
LLM-Shield-Proxy is listed by name with the other tested gateways. Its result follows the same evidence and replication rules as every other result.
Open, reproducible streaming privacy and audit evidence for enterprise LLM infrastructure Follow the links below to inspect the method, reports, and source.
All six current results were produced by this project on one workstation. No outside contributor has repeated them, so every product result is marked unreplicated.
The first prompt used invalid examples of all three data types:
| Old test value | Why Presidio rejected it |
|---|---|
[email protected] | .invalid is not a public domain suffix |
123-45-6789 | Presidio blocks this well-known invalid SSN sequence |
4532-1234-5678-9012 | The number fails the Luhn card-number checksum |
LLM-Shield-Proxy matched the patterns without checking whether the values were valid. This gave it an unfair advantage over validating detectors. A LiteLLM and Presidio run revealed the problem. The affected result was not published, the values were replaced, and all six configurations were tested again.
The benchmark also found two streaming bugs in LLM-Shield-Proxy. Both are fixed and covered by regression tests.
Known limitation: the test uses three fixed data formats. A small program written specifically for those formats can pass without being a general PII detector. The values change on every run, but the formats do not. Testing more formats caused two false failures in six trials.
Standard library plus httpx. You should not have to install one gateway to measure another, and nothing in the harness imports the proxy.
pip install pii-leak-benchmark
# negative control: boundary check MUST report a leak
pii-leak-benchmark --target-base-url capture://self
# your gateway, pointed at the capture upstream
pii-leak-benchmark --target-base-url http://127.0.0.1:4000/v1| Target | Outcome | Runs / submitters | Status |
|---|---|---|---|
| Raw capture endpoint negative control, not a product | fail | 1 / 1 | control |
| LLM-Shield-Proxy this project | pass | 1 / 1 | unreplicated |
| LiteLLM 1.99.0 default configuration, no guardrail attached | redaction-not-enabled | 1 / 1 | unreplicated |
| LiteLLM 1.99.0 + Presidio no leak; the values are not restored to the client | no-leak-profile-not-met | 1 / 1 | unreplicated |
| Portkey Gateway OSS 1.15.2 default configuration, no guardrails | redaction-not-enabled | 1 / 1 | unreplicated |
| Portkey Gateway OSS 1.15.2 + redaction no leak; one-way replacement, tester-authored patterns | no-leak-profile-not-met | 1 / 1 | unreplicated |
Every product result was run once by this project's maintainer. A result becomes replicated only after three different people each submit a run of the same gateway and configuration. fail means the gateway sent an unmasked test value to the benchmark's capture server. Products that do not advertise PII redaction are marked not applicable, not failed.
Each row is one gateway at one pinned version in one stated configuration, measured with the same check you can run yourself. “Reached the provider” means raw test values arrived at a capture server standing in for OpenAI.
Every row above was measured by this project, including our own, and none has been replicated by anyone unaffiliated yet. A row measures the configuration named in it, not the product’s best possible configuration. The full table carries the rest of the columns, who ran each one, and how to disagree with it.
This preview runs in your browser. It does not contact a model or send data anywhere. It demonstrates the regex and synthetic-value paths with fictional examples. The server can also use an optional local ONNX model and real AES-256-GCM encryption. Choose Agent tool call to see how string values in JSON are handled.
Fictional values with a similar format.
OpenAI-compatible clients can keep the same request format. Set base_url to the proxy address and configure provider and client keys on the server.
# Before - talking straight to the provider $ curl https://api.openai.com/v1/chat/completions \ -H "Authorization: Bearer $OPENAI_API_KEY" \ -d '{"model":"gpt-4o","messages":[{"role":"user","content":"Update record for John Doe, SSN 456-12-7890"}]}' # After - point the same SDK at LLM-Shield-Proxy. Nothing else changes. $ curl https://shield.internal.acme.corp/v1/chat/completions \ -H "Authorization: Bearer $OPENAI_API_KEY" \ -d '{"model":"gpt-4o","messages":[{"role":"user","content":"Update record for John Doe, SSN 456-12-7890"}]}' → the LLM only ever sees: "Update record for Michael Ito, SSN 839-14-2207" → your agent still receives: "Update record for John Doe, SSN 456-12-7890"
These are technical controls mapped to each framework's requirements - see how in the docs. Deploying this proxy is one control among many a full compliance program requires; it is not a certification.
Run the tests, inspect the JSON reports, and verify signed audit records. The results state what was tested and what remains outside the test.
Each record links to the previous record. Sequence numbers reveal missing or reordered entries. Ed25519 signatures can be checked offline. OSCAL 1.2 export and optional confirmed JSONL writes are also available.
pii-leak-benchmark --target-base-url http://127.0.0.1:4000/v1LLM requests can contain plain text or structured JSON. LLM-Shield-Proxy has separate paths for both. Configure and test the path used by your provider or tool.
For chat text, choose whether to replace a sensitive value with a synthetic value, a tag, an empty value, or encrypted text. The selected policy can vary by request.
See the 4 masking modes →For supported JSON-RPC requests, the proxy parses the JSON and checks its string values. It can replace selected values and restore them in the response. Test the exact schemas and response behavior used by your integration.
See the stateless agent firewall →After configuring the proxy, change your client base_url to the proxy address. OpenAI-compatible clients can keep using the same request format.
Local regex rules, secret scanning, and optional ONNX name detection inspect the request before it is sent to the model provider.
The proxy replaces selected values with synthetic data, tags, empty values, or encrypted text. It then sends the changed request to the model provider.
As SSE events arrive, the proxy joins replacement tokens that were split between events and restores values the client may receive. Measure the added delay in your own deployment.
LLM-Shield-Proxy provides an experimental /v1/mcp endpoint for a documented subset of Model Context Protocol JSON-RPC calls. It checks each supported tool call against the client's policy, replaces selected values in arguments and results, and can record the decision. It is not a complete MCP implementation.
LLM-Shield-Proxy is one option for self-hosted, streaming traffic. These are the main differences to test when comparing it with batch or hosted DLP tools.
| Batch or hosted DLP | LLM-Shield-Proxy | |
|---|---|---|
| Streaming | A batch scanner may wait for the full response before it can inspect it. | Checks and restores values as SSE events arrive. Measure the total delay in your deployment. |
| Where scanning happens | A hosted DLP service receives the text that it scans. | Runs in your environment and can test the request sent to the configured model provider. |
| Detection cost | NLP detection uses memory and compute that vary by model and runtime. | Makes the ONNX detector optional. Measure its memory and request time with your selected model. |
| Agent / tool-call traffic | Replacing text directly in raw JSON can damage its syntax or change a field unexpectedly. | Parses supported JSON-RPC messages and changes selected string values. Test your actual schemas. |
| Data retention | Storage and logging behavior varies by service and configuration. | Offers encrypted in-band values or time-limited Redis mappings. The documentation states where plaintext can remain. |
LLM-Shield-Proxy is not a model router or orchestration framework. It can run alongside LangChain, LiteLLM, Portkey, and similar tools.
Examples of systems you can test with the proxy
Run the assessment locally, or apply for a private 30-day design-partner pilot.
Prefer to inspect the code first? View the repository.