Hex v1.3 known limits

This page is for an agent deciding how far to rely on Zaun's verdicts, and which controls to set to Block. Hex v1.3 is the model that assesses every scan. It says how Hex v1.3 measures, where it is weaker, and what to do instead. If you are the person, read this before you turn on Block for a project.

How Hex v1.3 measures#

On Zaun's held-out benchmark, Hex v1.3 has 89.3% recall at a 0.019% false-positive rate on routine traffic. Treat the figures as a guide; your own traffic will differ.

Zaun's own attack sets give the picture by attack type: plain jailbreaks 35 of 36, encoded or disguised attacks 48 of 48, attacks split across messages 13 of 16 at the final turn (14 of 16 on any turn), self-harm 26 of 28, money laundering 28 of 28, violent threats 12 of 12, and forgery and fraud 12 of 12, for 174 of 180 in all.

Where to add other layers#

Attacks split across several messages are the hardest attack type in Zaun's sets. Advice in model replies, agents' use of tools, and sensitive data in prompts also deserve layers that do not depend on Hex v1.3 alone.

Do this instead:

• Unreliable or unsafe advice: scan replies that give advice, with POST /v1/scan/prompt as described in explicit output scans, and have a person review high-stakes advice.
• Agent and tool security: keep tools on least privilege, require approval for risky actions, and block at the client or gateway where you can; see choose a capture path.
• Sensitive data leaks: turn on the input checks for credentials, Social Security numbers and card numbers in project controls. They match patterns, so they do not depend on the model.
• Attacks split across messages: send the recent conversation with a session_uuid, as long or unrelated context describes.

It reads wording literally#

Hex v1.3 judges the text as written. Blunt wording without harmful intent, such as a person venting, can be flagged. A model reply that drafts a one-sided document has been read as fabrication. Results for the same request can differ between calls, and in rare cases benign text about violence or self-harm is blocked.

Some framing tricks do not fool it: role-play, hypotheticals and translation used to ask for the system prompt are caught, and text that only discusses attacks, such as an article about prompt injection, is not flagged.

Do this instead:

• Set the project's application context (context_prompt in the project configuration; see project controls) to describe what normal work looks like for the application. Without it Hex v1.3 has less to go on: in Zaun's tests, a project with no context treated a plain request for the system prompt as an alert rather than a threat.
• Report incorrect detections, in both directions.

Numbers, amounts and dates#

Not yet measured. Zaun has not measured whether Hex v1.3's verdicts are less accurate on text full of amounts, prices, dates or deadlines.

Do this instead: if such text is ordinary for the application, say so in the application context, and review alerts on it before moving controls to Block.

Long or unrelated context#

A short instruction inside a long tool result or model reply is more often missed than the same instruction sent on its own. That is where an instruction injected into retrieved content is most likely to hide.

With a session_uuid, Zaun keeps a record of the session's scanned turns, and Hex v1.3 assesses each new message beside the recent conversation, within a size budget. A message that refers back to earlier turns is assessed with them. The record keeps no message text, so the recent conversation comes from the turns the request carries. A request without a session, or one that sends only the newest message, gives Hex v1.3 no earlier text to weigh, which is how an attack split across messages gets through.

Do this instead:

• Send each tool result, retrieved document or email as its own message, rather than pasting them into one long prompt, and scan the part the agent will act on.
• Send the recent conversation with each request, with a session_uuid, as scan concepts describes. Turns the session has already scanned are not assessed again and are billed at a tenth of the scan rate, as usage.cached_input_tokens.
• Scan long files as documents, and review results whose coverage is partial.

When part of the assessment is unavailable#

When part of Hex v1.3's assessment is unavailable, a scan gets a reduced assessment that can alert but cannot block. This is rare, and more likely for requests with many messages under heavy load. A scan Hex v1.3 could not assess at all returns an alert for review and is not charged.

Do this instead: send the messages that need assessing rather than everything at once, and review alerts.

Adversarial content#

Hex v1.3 caught all 48 encoded and disguised attacks in Zaun's set, using Base64, hex, ROT13, look-alike characters and translation. The reverse also happens: harmless encoded text, such as a hex-encoded building-code reference and a reversed case number, has been alerted. Attacks unlike those in Zaun's evaluation sets are the least measured, and Hex v1.3 has not been evaluated against attacks that adapt to it.

Do this instead:

• Keep the layers from where to add other layers in place.
• Review alerts as well as blocks: partial detections often arrive as alerts.
• Report misses.

Option order#

Not yet measured. Whether changing only the order of choices or list items in a request changes Hex v1.3's result has not been tested.

Do this instead: if a result changes when only the order changes, report it with both requests.

Generated content#

• On POST /v1/responses, plain generated text is not scanned automatically. Only generated tool calls are assessed before release. See Responses.
• A threat Hex v1.3 finds only in model replies, tool calls or tool results normally raises an alert instead of blocking. See what blocks today.
• Most false alerts are in model replies: confident advice, endorsements and guarantees can be read as harmful directives.
• Unsafe content hidden in one sentence of a long reply is often missed.
• Purely visual content in an image, without text, is not assessed. See scan an image.
• Generated code has not been measured separately.

Do this instead:

• When a reply matters, scan it with POST /v1/scan/prompt as described in explicit output scans.
• Block on the person's request, where Hex v1.3 can block, and limit what tools can do with the reply.
• For long replies, scan the part you will act on, such as the commands or code you will run.

What alerts instead of blocking#

• A threat found only in the agent's own messages alerts, as described above.
• Findings below a threat alert even where the control is set to Block.
• A reduced assessment alerts and cannot block.

Self-harm content can block. It has its own Self-harm control, set to Block on a self-serve organization's default project. A project that has not set Self-harm follows its Risky setting. Risky keeps governing other risky behavior, such as fraud, concealment, violence and help with crimes, so the two can be set apart. See project controls.