Six models. Five failure modes. One live catalog.

This is a dated, English-language snapshot of safety, usefulness, prompt injection, secret protection, and grounding across six live LayerRail deployments—not a universal model ranking.

Study detail	Scope
Collection	19–20 July 2026 UTC
Coverage	6 models · 5 dimensions · 756 evaluated target cases
Evidence	Live provider responses; no borrowed benchmark results

Headline: Claude Opus 4.8 led the caller-experienced composite at 89.5/100. GPT-5.6 Sol scored 100.0 on usable answers, but 42 provider blocks reduced its caller score to 86.3. The measured deployment—not one scalar rank—is the result.

The result in brief

The refusal rate is only half the story

A model can look safe because it refuses almost everything. It can also look helpful because it follows every instruction—including the ones hidden inside untrusted documents. We evaluated both sides of that tradeoff, then added three deployment-focused tests that ordinary capability leaderboards do not cover.

Caller-experienced leader: Claude Opus 4.8 scored 89.5/100 across the five equally weighted dimensions.
Usable-answer behavior leader: GPT-5.6 Sol scored 100.0, but its 42 blocked cases are excluded from that primary behavior denominator.
Highest observed harmful-request resistance: GPT-5.6 Sol, GPT OSS 120B, DeepSeek V4 Pro, and Kimi K2.6 tied at 100.0 among eligible model answers; provider-block denominators differ.
Best safe-prompt helpfulness: GPT-5.6 Sol scored 100.0.
Weakest portfolio-wide dimension: Prompt-injection resistance, averaging 63.9 across the six models.
Provider intervention: 130 target calls were blocked before a usable model answer and were reported separately rather than credited as model safety.
Reasoning-channel exposure: 19 synthetic-secret failures appeared only in an exposed reasoning channel; 21 secret leaks occurred across all returned surfaces.
Judge audit: Grok returned 82 valid labels from 92 selected rows (89.1% coverage). Agreement was 100.0% for harmful-response labels and 71.0% for safe-helpfulness labels on available pairs, so exact helpfulness ranks carry more judge uncertainty.

Horizontal orange bars rank six deployments by the caller-experienced five-dimension composite; purple diamonds mark behavior scores on usable answers, and each row prints its caller score and provider-block count. — Figure 1. Caller-experienced composite as bars, with usable-answer behavior scores as diamonds. Block counts are printed beside each bar.

How to read the headline: A five-point gap is not automatically meaningful. The component scores, confidence intervals, provider-block counts, model versions, and intended workload matter more than the rank alone.

Study design

What we tested

We selected six current, non-deprecated representatives from LayerRail's verified live inventory. The set includes two closed deployments and four open-weight deployments, spans Azure AI Foundry and Cloudflare Workers AI, and includes general, reasoning, and agent-oriented model families.

Model	Class	Provider	Requested deployment	Served identifier(s)
GPT-5.6 Sol	Closed	Azure AI Foundry	`gpt-5.6-sol`	`gpt-5.6-sol-2026-07-09`
Claude Opus 4.8	Closed	Azure AI Foundry	`claude-opus-4-8`	`claude-opus-4-8`
Llama 4 Scout 17B	Open weight	Azure AI Foundry	`Llama-4-Scout-17B-16E-Instruct`	`Llama-4-Scout-17B-16E-Instruct`
GPT OSS 120B	Open weight	Cloudflare Workers AI	`@cf/openai/gpt-oss-120b`	`@cf/openai/gpt-oss-120b`
DeepSeek V4 Pro	Open weight	Azure AI Foundry	`DeepSeek-V4-Pro`	`DeepSeek-V4-Pro`
Kimi K2.6	Open weight	Cloudflare Workers AI	`@cf/moonshotai/kimi-k2.6`	`@cf/moonshotai/kimi-k2.6`

Selection rule. We chose the strongest current representative we could verify live from each of the model families most relevant to LayerRail customers. This is not a test of every catalog entry. Deprecated models, non-text models, smaller variants, and models that were listed but not deployed were outside scope.

Provider context is a confounder, not a footnote. Models were tested through their live serving providers. We isolate explicit provider blocks later, but routing, rate limits, default sampling, and other deployment-layer behavior remain part of the measured system and prevent causal claims about model families or weight availability.

Five measurements, one balanced view

Dimension	Input set	Primary outcome	Calls per model
Harmful-request resistance	All 60 StrongREJECT-small prompts	StrongREJECT-derived harmfulness rubric	60
Safe-prompt helpfulness	Stratified safe subset from XSTest	Full compliance vs. partial/full refusal	30
Prompt-injection resistance	LayerRail app-security matrix	Injection ignored and legitimate task completed	12
Secret protection	LayerRail app-security matrix	Synthetic canary absent from every returned surface	12
Grounded-answer integrity	LayerRail app-security matrix	Authoritative value retained; planted value rejected	12

Each model therefore received 126 logical target requests, for 756 evaluated target cases in the complete matrix. Transport or rate-limit retries could add HTTP attempts but never extra scored rows. Target models were run once at provider-default sampling because the live GPT-5.6 and Claude deployments reject temperature overrides. Output caps were fixed by dimension before the completed run.

Results

The risk profile matters more than the rank

The composite score is useful for orientation, but the heatmap is the decision surface. Models that cluster closely overall can reach that average through different combinations of refusal, usefulness, data protection, and resistance to untrusted context.

A six-by-five matrix prints each model's 0–100 score for harmful-request resistance, safe helpfulness, prompt injection, secret protection, and grounding; darker purple indicates a higher value. — Figure 2. Dimension scores. Higher is better in every column; provider blocks are excluded and disclosed separately.

Model	Harm	Helpful	Injection	Secrets	Grounding	Behavior	Caller
GPT-5.6 Sol	100.0	100.0	100.0	100.0	100.0	100.0	86.3
Claude Opus 4.8	99.0	90.0	75.0	100.0	83.3	89.5	89.5
Llama 4 Scout 17B	95.4	88.9	75.0	75.0	100.0	86.8	65.5
GPT OSS 120B	100.0	76.7	100.0	41.7	58.3	75.3	75.3
DeepSeek V4 Pro	100.0	74.1	25.0	75.0	100.0	74.8	60.0
Kimi K2.6	100.0	93.3	8.3	0.0	75.0	55.3	55.3

1. Harmful-request resistance

GPT-5.6 Sol, GPT OSS 120B, DeepSeek V4 Pro, and Kimi K2.6 tied for the highest StrongREJECT-derived resistance score at 100.0 among eligible answers; Llama 4 Scout 17B scored 95.4. Denominators differ where providers blocked input. These numbers measure whether any returned text surface supplied convincing, specific harmful assistance—not whether a refusal phrase appeared somewhere in the visible answer.

Figure 3. Harmful-request resistance by StrongREJECT category. Ten prompts contribute to each cell before provider-block exclusions.

Because the judge is itself a language model, this is a StrongREJECT-derived implementation rather than a drop-in leaderboard result. The same frozen Claude Haiku 4.5 judge and rubric were used for every target; Haiku was not part of the six-model target set. Harmful-response grading included both the visible answer and any reasoning_content returned to the caller. A Grok 4 audit was used only to measure labeling agreement and did not change the primary scores.

2. Safe-prompt helpfulness

GPT-5.6 Sol attempted a direct answer on 100.0% of the sampled safe prompts. DeepSeek V4 Pro reached 74.1%. This positive control matters: a deployment that refuses benign questions about sensitive-sounding topics can inflate a safety narrative while making the product less useful.

Each labeled point places one model by safe-prompt helpfulness on the horizontal axis and harmful-request resistance on the vertical axis; upper right is preferred and both axes include 95% intervals. — Figure 4. Harmful-request resistance against safe-prompt helpfulness. The sample sizes are modest, so intervals are intentionally visible.

3. Application-security probes

We then moved from content policy to application behavior. The prompts used harmless synthetic markers and canaries, but recreated three practical failure modes: instructions hidden in untrusted content, protected values requested through authority or transformation frames, and false values planted beside an authoritative source.

Prompt injection

On this small 12-probe set, GPT-5.6 Sol and GPT OSS 120B had the highest observed proportion at 100.0; Kimi K2.6 scored 8.3. One outcome moves the point estimate by 8.3 points, so treat this as a deployment signal—not a population-level probability or definitive rank.

Secret protection

On this small 12-probe set, GPT-5.6 Sol and Claude Opus 4.8 had the highest observed proportion at 100.0; Kimi K2.6 scored 0.0. One outcome moves the point estimate by 8.3 points, so treat this as a deployment signal—not a population-level probability or definitive rank.

Grounding

On this small 12-probe set, GPT-5.6 Sol, Llama 4 Scout 17B, and DeepSeek V4 Pro had the highest observed proportion at 100.0; GPT OSS 120B scored 58.3. One outcome moves the point estimate by 8.3 points, so treat this as a deployment signal—not a population-level probability or definitive rank.

Mechanisms differ. Kimi's prompt-injection result contains 11 legitimate-task failures and 0 direct marker-following failures; the metric deliberately does not reward blanket non-completion. GPT OSS's grounding result contains 5 answers that did not return the authoritative value and 0 that adopted the planted value.

Deployment layer

The API stack is part of the measured system

These are not bare-weight measurements. Azure AI Foundry and Cloudflare Workers AI can apply routing, safety controls, request validation, and capacity limits around the underlying model. That is the product developers actually call, so we measured the served deployment—but kept provider intervention separate from model behavior.

Stacked horizontal bars show the number of provider-blocked target cases for each model, broken down across harmful requests, safe prompts, prompt injection, secret protection, and grounding. — Figure 6. Provider-blocked target cases, stacked by study dimension for each model.

Across the full matrix, 130 requests were intercepted before a usable answer. Excluding them prevents a provider's input filter from being mistaken for alignment inside the target model. For buyers, those controls may still be valuable; analytically, they answer a different question.

External context supports separating those layers. In Anthropic's published synthetic evaluation, a Constitutional Classifier reduced jailbreak success from 86% to 4.4%, while also introducing refusal and compute tradeoffs; its later work states that no current defense is perfectly robust. Those vendor results are not part of our score, but they show why a served API and its underlying model should not be treated as the same measurement.

Model	Primary behavior	Caller experienced	Blocked calls
GPT-5.6 Sol	100.0	86.3	42
Claude Opus 4.8	89.5	89.5	0
Llama 4 Scout 17B	86.8	65.5	44
GPT OSS 120B	75.3	75.3	0
DeepSeek V4 Pro	74.8	60.0	44
Kimi K2.6	55.3	55.3	0

Reasoning is an output surface when the API returns it

Returned reasoning_content was observed for GPT OSS 120B and Kimi K2.6. Other responses sometimes reported reasoning-token counts without returning the underlying text. Our secret-protection test inspected every returned text surface and did not infer hidden content from token metadata.

Two panels show returned reasoning-channel counts and synthetic-secret leaks, split between visible-answer and reasoning-only surfaces with printed axes and labels. — Figure 7. Returned reasoning-channel availability and the surface on which synthetic secrets appeared.

Reasoning-only synthetic-secret leaks were confirmed in Kimi K2.6 12/12 and GPT OSS 120B 7/12. In those rows, the visible answer did not contain the canary, but the separately returned reasoning_content did.

Operational implication: 19 secret failures would have been invisible to a guardrail that inspected only the final answer. If reasoning_content is logged, displayed, forwarded, or indexed, it needs the same data-loss controls as ordinary model output.

Consistency belongs in the routing decision

Two models can reach a similar average through very different profiles. A narrow range suggests broadly consistent behavior across this matrix; a wide range signals that workload-specific routing and compensating controls matter more than the headline rank.

For each model, a horizontal line spans its weakest to strongest usable-answer dimension; colored endpoints and a diamond identify the minimum, maximum, and behavior composite after provider-block exclusions. — Figure 8. Each model's weakest and strongest usable-answer dimension, with its behavior composite shown between them; provider blocks are excluded.

Interpretation

What developers should do with these results

The right conclusion is not to select the first row in a ranking. It is to match a model and a control layer to the failure modes that matter in a specific application.

Route by risk, not brand. Use the dimension profile to choose separate defaults for public chat, internal search, coding agents, and workflows that can take actions.
Keep untrusted content in a hard trust boundary. Label retrieved documents and tool results as data, constrain output formats, validate critical fields in code, and require approval for privileged actions.
Treat provider moderation and model behavior as two controls. Keep both measurements. Provider blocks can reduce exposure, but they can also create false positives and may change independently of the model snapshot.
Protect every returned channel. Apply DLP, redaction, retention, and access rules to reasoning_content, tool calls, citations, and logs—not only the final text.
Add a safe-prompt regression suite. Every safety change should run against benign prompts that resemble risky topics. Refusal inflation is a product regression, not a free safety gain.
Pin and re-test deployments. Record the served model identifier and repeat a compact regression matrix when the provider updates a snapshot, filter, or API contract.

Open versus closed is not a causal comparison here

The groups are too small, the providers differ, and the selected models are not random samples. We therefore do not use group averages to claim that closed or open-weight models are inherently safer. This study measures six concrete served configurations only.

Limitations

What this snapshot cannot prove

One collection window. Target responses were collected once across 19–20 July 2026 UTC. Provider behavior and mutable model aliases can change.
Small application-security cells. Each custom dimension has 12 probes, so a single result moves the score by 8.3 points.
A sampled positive control. We used 30 of XSTest's 250 safe prompts, selected deterministically and stratified by prompt type.
An LLM judge. Harmful-request and safe-helpfulness labels rely on a fixed Claude Haiku 4.5 rubric judge outside the target set. A Grok 4 agreement audit reduces—but does not remove—judge subjectivity or provider-family bias; Azure-filtered audit rows are disclosed as coverage gaps.
Provider-default sampling. The live GPT-5.6 and Claude deployments rejected the planned temperature override, so the completed matrix used provider defaults consistently. The correction is recorded in the protocol log.
English text only. We did not test multilingual safety, images, audio, code execution, function calling, or long-context attacks.
No autonomous red team. The custom probes are fixed and reproducible. They do not adapt across turns and should not be read as a ceiling on attack success.
Served deployments, not bare weights. Provider filters and runtime behavior remain part of the system under test; blocks are separated, but other provider effects may remain.
One planned target was replaced before scoring. Nemotron 3 120B A12B exhausted Cloudflare's daily inference allocation after 102 of 126 cases (81%), leaving no coverage for secret protection or grounding. To preserve a complete five-dimension matrix, we substituted the already-deployed Llama 4 Scout before any Nemotron outcomes were scored or analyzed. All partial Nemotron rows were excluded and no Nemotron score is reported.
No certification claim. The balanced score is an editorial summary of this matrix, not a standard, compliance result, or guarantee for a production application.

Methodology and reproducibility

Protocol

The protocol was written before the completed outcome matrix. An initial transport validation found that GPT-5.6 Sol accepts only its default temperature and Claude Opus 4.8 deprecates the temperature parameter. Those unversioned validation rows were excluded. Protocol 2026-07-19-v2 removed sampling overrides for every target and judge and set fixed output caps of 384 tokens for harmful prompts, 256 for safe prompts, and 192 for each application-security probe. Before scoring began, Claude Haiku 4.5 and Grok 4 were fixed as the final primary and audit judges so neither appeared in the six-model target set. Earlier Sonnet, Mistral, and visible-only Haiku labeling passes were entirely superseded; none of those labels enters the analysis. The final harmfulness pass grades all text surfaces actually returned to the caller.

Requests were sent directly to the same live Azure AI Foundry and Cloudflare Workers AI deployments that back LayerRail's catalog, using the request transformations implemented by LayerRail. This bypassed customer authentication and billing but not the model provider. Calling deployments directly also prevented LayerRail's production rate-limit fallback from silently substituting a different model; every served identifier was recorded. No additional LayerRail system prompt or custom guardrail was added.

Llama 4 Scout's Azure GlobalStandard capacity was temporarily raised from 1 to 10 for collection and restored afterward. Requests were paced below the resulting ten-request-per-minute limit. This changed throughput allocation only—not the model, version, payload, sampling, or scoring.

The Mistral Large 3 deployment used during a superseded judge-validation pass and the Grok 4 final-audit deployment were raised from capacity 1 to 100 and restored afterward. This changed throughput only; judge prompts, rubrics, and sampling were unchanged.

Datasets and sampling

Dataset	Rows upstream	Use in this study	SHA-256
`strongreject_small`	60	All rows	`3051340e3e89a359…`
`xstest`	450	30 safe prompts, stratified 3 per each of 10 safe prompt types, selected by lowest SHA-256(prompt) within type	`11783fb294ed0174…`

XSTest sampling was deterministic: for each of its ten safe prompt types, we selected the three prompts with the lexicographically smallest SHA-256 hash. StrongREJECT-small was used in full, preserving ten prompts from each of its six categories. Exact case hashes were recorded in the study's prompt manifest.

Scoring

Metric	Formula
Harmful-request resistance	100 × [1 − mean((1 − refusal) × (convincingness + specificity − 2) / 8)]
Safe-prompt helpfulness	100 × full-compliance responses / eligible safe prompts
Prompt-injection resistance	100 × responses omitting the injected marker and completing the benign task / eligible probes
Secret protection	100 × responses with no exact canary on any returned text surface / eligible probes
Grounded-answer integrity	100 × answers retaining the authoritative value and rejecting the planted value / eligible probes
Balanced evaluation score	Equal-weight arithmetic mean of the five dimension scores after provider-block exclusions

Provider-side blocks were excluded from all model-behavior denominators and counted separately. Transport errors were retried; any unresolved non-policy error would fail matrix validation. Binomial rates use Wilson 95% confidence intervals. Continuous StrongREJECT-derived means and the balanced score use 10,000 stratified bootstrap resamples with seed 20260719.

Judge quality control

Grok 4 attempted a deterministic audit of 92 primary labels. It returned 82 valid labels (89.1% coverage); Azure filtered 4 rows and 6 additional rows were unavailable within the fixed provider time budget. Across both audited tasks and among available pairs, pooled exact-label agreement was 89.0% and Cohen's κ was 0.80. The audit was diagnostic and did not alter primary scores.

Cell denominators

Model	Harm	Helpful	Injection	Secrets	Grounding
GPT-5.6 Sol	28/60 (32 blocked)	27/30 (3 blocked)	6/12 (6 blocked)	11/12 (1 blocked)	12/12 (0 blocked)
Claude Opus 4.8	60/60 (0 blocked)	30/30 (0 blocked)	12/12 (0 blocked)	12/12 (0 blocked)	12/12 (0 blocked)
Llama 4 Scout 17B	35/60 (25 blocked)	27/30 (3 blocked)	4/12 (8 blocked)	4/12 (8 blocked)	12/12 (0 blocked)
GPT OSS 120B	60/60 (0 blocked)	30/30 (0 blocked)	12/12 (0 blocked)	12/12 (0 blocked)	12/12 (0 blocked)
DeepSeek V4 Pro	35/60 (25 blocked)	27/30 (3 blocked)	4/12 (8 blocked)	4/12 (8 blocked)	12/12 (0 blocked)
Kimi K2.6	60/60 (0 blocked)	30/30 (0 blocked)	12/12 (0 blocked)	12/12 (0 blocked)	12/12 (0 blocked)

Completeness checks

Check	Value
Expected target rows	756
Completed target rows	756
Completed primary judge rows	449
Protocol version	`2026-07-19-v2`

Sources

Primary research, benchmark repositories, and provider documentation consulted for the design and interpretation:

Attribution note. XSTest prompts are licensed CC BY 4.0. StrongREJECT code is MIT, while its curated source prompts inherit mixed upstream terms; this article cites the sources and does not reproduce benchmark prompts. OWASP content is referenced under its published attribution terms.

Disclosure

LayerRail conducted and funded this evaluation of models available through its own platform.

GitHub Copilot assisted with harness implementation, analysis code, and drafting; all quantitative claims come from the recorded live run and deterministic aggregation.