Streamed Large Language Model (LLM) Response Inspection
Prisma AI Runtime Security Network Intercept secures applications streaming LLM
responses directly to end users by inspecting traffic inline using your configured security
profiles and detectors, maintaining low Time to First Token (TTFT) performance.
| Where Can I Use This? | What Do I Need? |
|---|
- Prisma AIRS- AI Runtime Security
|
|
AI applications that stream responses from large language models (LLMs) present a
security challenge: threats such as prompt injection or malicious URLs can be split
across multiple response chunks, escaping inspection systems that evaluate only complete
responses. AI Runtime Security Network Intercept inspects each chunk inline within a
context window of adjacent chunks, detecting split-boundary threats without delaying
safe content delivery.
The key capabilities include:
- User Experience / TTFT—Safe chunks pass through immediately as
received, significantly reducing TTFT and providing an experience identical to direct
LLM streaming.
- Threat Enforcement & Blocking—If chunk N
contains a threat, preceding safe chunks (1 through N-1) are
delivered, while chunk N and all subsequent chunks are blocked
immediately.
- Context Boundary Inspection—Inspects each chunk along with a context
window of adjacent chunks to catch threats split across chunk boundaries (For
example: A malicious URL split between chunk N and
N+1).
When an LLM streams output as a sequence of chunks, the firewall inspects each chunk
together with a surrounding window of adjacent content before forwarding it to the client
application. This context window ensures that threats spanning a chunk
boundary—such as a prompt injection or malicious URL split across consecutive
chunks—are detected as a single pattern.
If a threat is detected at chunk N (or split across chunks
N and N+1):
- The firewall terminates the stream immediately. No further chunks are forwarded to
the client.
- Safe chunks preceding the threat (1 to N-1)
are delivered to the client.
- If a custom error response is configured in the AI security profile, that error
content is returned as the response body when the stream is terminated.
The Max Inline Latency and Inline Timeout
Action settings configured in your AI security profile apply to streamed
responses. If the AI Core Service does not return an inspection verdict for a chunk
within the configured Max Inline Latency, the firewall
automatically executes the configured Inline Timeout Action. The
enforcement applies to both streamed and non-streamed LLM responses to ensure consistent
policy behavior to maintain consistent policy enforcement across both streaming and
non-streaming LLM traffic.
Your existing AI security profile and security policy rules automatically apply to
streamed responses using the same detectors, inline latency timeouts, and enforcement
actions you have already configured. For more information on setting up profiles, see
AI Security
Profiles.
