The Assurance Gap

eventJohn Waller presents this paper at Cybercamp.ai in Brno, Czechia, on November 6, 2026.

Executive Summary

Open-weight AI models are rapidly changing the economics of artificial intelligence. OpenRouter reported that Chinese-developed models accounted for more than 60% of its token traffic in June 2026, a dramatic reversal from the previous year.^[1] Lower costs, the ability to deploy models within an organization's own environment, and greater control over customization are driving adoption. But the speed of that shift raises a difficult question: how can an organization trust a model whose origin, security, or behavior may be poorly documented or not independently verified?

Governance policies are common. UltraViolet Cyber's 2026 AISec Study found that 8 of 10 organizations had vendor and model trust assessment policies. None had fully established a model registry with verified provenance, and only 3 had fully established third-party AI testing.^[2] Existing frameworks identify what organizations should consider; security teams still need a repeatable way to collect evidence, test models, and reach decisions they can defend.

This paper proposes a six-domain Model Assurance methodology and describes a working automated platform combining behavioral testing, AI-assisted research, evidence-based scoring, and human review. The assessment produces an overall score and domain-level results that can be evaluated against the requirements of a proposed use case. Those results belong within a broader enterprise Model Registry, which can also track model identity, versions, ownership, application approvals, and changes over time. The implementation is a practical starting point. Organization-specific testing is still required.

AI Adoption Has Changed the Security Equation

For years, enterprise AI adoption largely meant choosing among a handful of established providers. Organizations evaluated the vendor, reviewed contractual protections, examined data-handling practices, and relied on the provider to manage the underlying models. That arrangement was familiar to security teams because it resembled the vendor-risk and SaaS assessment processes they had used for decades. Capable, inexpensive open-weight models have changed the decision. An organization can now obtain a model from a developer it has no commercial relationship with, modify it, run it through a third-party inference provider, or deploy it entirely within its own infrastructure, often with little documentation about how the model was trained, tested, or secured. That flexibility is one of their greatest advantages, and it means the familiar vendor relationship may not exist. A model might originate with a well-known research laboratory, pass through several community modifications, and ultimately be served by an infrastructure provider that had nothing to do with its development. The name attached to the model may remain recognizable even though the precise artifact, quantization, configuration, or serving environment has changed. An assessment of the original publisher tells us something about the model's history, but it does not necessarily establish the security characteristics of the version actually being used. The traditional vendor assessment no longer gives the answers security teams need, and many organizations have yet to develop a reliable alternative.

There is also a fundamental difference between evaluating conventional software and evaluating an AI model. Traditional application security has spent decades developing techniques to inspect source code, identify known vulnerabilities, analyze dependencies, and test defined application behavior. Those techniques remain relevant to AI infrastructure and applications, but they do not fully describe the behavior of a model whose responses depend on probabilistic inference and the context supplied to it. A model may resist a prompt injection attempt several times before complying with a variation of the same request. It may respect an instruction in one conversation while abandoning it after several turns. It may refuse an unauthorized action when asked directly, yet attempt that same action when the request arrives through a document, tool response, or other material it has been instructed to process.

Open-weight models are not inherently less trustworthy than commercial alternatives. Frontier models are also vulnerable to prompt injection, information disclosure, inappropriate tool use, and unexpected behavior, and commercial providers do not always disclose enough information to support independent assessment.^[3] Open models make the diversity of model sources, modifications, and serving arrangements impossible to ignore. Their rapid adoption has exposed a broader problem that applies across the AI industry: organizations have developed increasingly sophisticated ways to acquire and deploy AI models without developing equally mature ways to establish and maintain confidence in them.

From AI Governance to Model Assurance

Frameworks such as the NIST AI Risk Management Framework, its Generative AI Profile, and ISO/IEC 42001 give organizations a structure for governing AI risk.^[4][5][6] None of them includes an operational capability for evaluating the security characteristics of individual models. A governance policy may require a model to be assessed before deployment, but somebody still has to determine what constitutes sufficient evidence, which tests should be performed, how the results should be interpreted, and when an earlier assessment is no longer adequate. A framework can identify prompt injection as a security concern, but it does not tell an organization whether the particular model under consideration has demonstrated acceptable resistance to the forms of injection relevant to its applications. That requires an assessment process capable of converting general security concerns into specific evidence and then into a decision.

The proposed model assurance methodology organizes that work into six domains. Provenance and supply chain examines the model's origin, lineage, licensing, and artifact integrity.^[7] Data handling considers available information about the treatment of inputs, outputs, and sensitive information, including the characteristics of the service through which a model is accessed. Technical security posture examines observable weaknesses such as susceptibility to prompt injection, inappropriate disclosure, and failure to maintain security constraints. Access and integration risk considers the model's ability to interact with tools and external systems, while operational maturity addresses maintenance, support, disclosures, and the practices surrounding the model's continued availability. Evaluation and known behavior brings together published research, independent findings, and repeatable testing of the model's performance under defined conditions.

The resulting assessment produces both an overall score and individual domain scores. The overall score gives a useful indication of the confidence that can be placed in the model across the areas examined, making it possible to compare candidates and maintain a view of an organization's model portfolio. The domain scores preserve the details that a single number cannot communicate, particularly weaknesses that may be important for some applications but relatively unimportant for others. Evidence quality accompanies those scores, distinguishing unsupported assertions from credible documentation and findings supported by actual testing. An organization may decide to weight particular domains more heavily or establish minimum thresholds, and those choices should reflect its own risk requirements and the quality of the evidence, since no single numerical formula fits every environment.

Building a Model Registry

An enterprise Model Registry is the system of record for the models an organization uses. It should capture model identity and version, origin and provenance, ownership, the applications using each model, and relevant approvals or restrictions. Model assurance adds the evidence-based trust assessment: tests performed, research reviewed, overall and domain scores, known limitations, and the history of reassessments. A registry should retain records of models that have been rejected or cannot yet be adequately assessed, too. Those outcomes help when another team considers the same model, and they should not disappear because approval was withheld.

The registry changes the relationship between application development and security review. Developers can begin by examining models whose identity, assurance history, and approval status have already been recorded, before selecting one. A team building an internal knowledge assistant might identify several candidates with sufficient data-handling and security-posture scores, while another team developing an application with autonomous tool access might find that none of the existing candidates satisfies its requirements. In the first case, the prior assurance work substantially reduces the amount of new investigation required; in the second, it identifies the gap before the organization has committed substantial development resources to an unsuitable model.

An organization's registry should also reflect the fact that assurance belongs to a particular model artifact or service configuration. A community-produced quantization, an independently modified fine-tune, and the original publisher's release may have different provenance, behavior, or operating characteristics.^[8] Similarly, a model accessed through a hosted endpoint may be affected by changes in the serving provider or its configuration. Two distributions bearing the same model name should not automatically inherit identical assurance conclusions, and a change that materially affects what was evaluated should prompt a review.

This becomes particularly important when new model versions are released. Suppose an organization has evaluated version 3 of a model and several applications are using it. When version 4 becomes available, the organization can use the existing assessment as a starting point, repeat relevant behavioral tests, examine newly published evidence, and determine whether the new version continues to meet its requirements. Improvements in general benchmark performance do not establish that security behavior has improved, and a newer version should not automatically inherit an earlier approval. The registry should preserve the history of both assessments, allowing reviewers to understand what changed and which findings remain relevant.

A registry keeps a reliable record of model assurance decisions, preserves the evidence behind them, and makes that evidence available when circumstances change. It turns individual assessments into a reusable security capability, and reassessment turns that capability into an ongoing process.

Automating the Model Assurance Process

To explore how that could be accomplished, we developed a working Model Assurance Platform that combines automated behavioral testing, AI-assisted evidence gathering, structured assessment, and a model registry. The platform demonstrates that the essential steps of a model assurance assessment fit into one repeatable workflow. It is designed to gather evidence before reaching conclusions, keep test observations separate from assessment judgments, and preserve the information needed to understand how each result was produced. Human reviewers remain responsible for deciding what the findings mean and whether the evidence supports the proposed assurance scores.

The behavioral testing component uses OpenRouter to interact with selected models through a common interface, allowing an assessor to evaluate models without having to deploy each one locally.^[9] The initial implementation includes five testing suites covering security constraints, data handling, tool-use behavior, consistency, and model self-reporting. These tests examine such questions as whether a model will reveal protected instructions, follow malicious directions embedded in material it has been asked to process, disclose information that should remain restricted, or attempt to use a tool outside its intended authority. Other tests examine whether a model maintains its instructions across multiple conversational turns, produces consistent responses to related requests, or invents details about its own origin and capabilities. The suites include permitted-behavior checks as well, since a model that refuses every request has not necessarily demonstrated useful or appropriately constrained behavior.

Each test is executed according to a defined procedure, with results recorded for subsequent analysis. Deterministic checks evaluate conditions that can be established directly, such as whether a protected value appeared in the response, whether the model produced the required format, or whether it attempted an unauthorized tool call. A separate language model can assist in evaluating responses that require semantic interpretation, and the system records errors and inconclusive results as separate outcomes from passes and failures. Repeating tests helps expose inconsistent behavior, which can be as important to an assessor as a consistent failure. The platform also records information about the model endpoint and serving conditions, recognizing that behavioral evidence applies only to what was tested.

Testing alone cannot answer every assurance question, which is why the platform includes a separate research process. This component retrieves available information from model publishers and selected independent sources, then uses AI to organize relevant findings against the six assurance domains. It considers documentation, published evaluations, and other available research, including whether the source of a particular claim is genuinely independent of the model publisher. This distinction is important because a manufacturer's technical report remains the publisher's own claim even when it appears in a public research repository. The research component prepares suggested scores and rationales for the reviewer, and the reviewer decides which suggestions become final assessment results.

A central design decision is that automation does not make the final trust determination. It makes the evidence-gathering and testing process more consistent, reduces the time required to perform repeatable tasks, and gives human judgment a common basis. This matters particularly when an automated research component is itself an AI model capable of making mistakes, misinterpreting sources, or presenting uncertain information with unwarranted confidence. Allowing one model to investigate another can be useful, but the process must preserve source information, constrain unsupported claims, and keep review decisions accountable to people. The automation should make assurance decisions better informed.

Model Assurance in Practice

Consider how the process applies to a model that an organization is evaluating for possible adoption. DeepSeek V4.1 Flash is a useful example because it is one of the most popular open-weight models at the time of this writing. The model can be accessed through OpenRouter, allowing behavioral testing without requiring the assessment team to deploy and maintain a local copy. The platform can associate the selected model with its available documentation, run its predefined test suites, collect supporting research, and assemble the findings for review. This walkthrough describes what that process produces and how an assessor would use it. The platform's demonstration scores are not independently validated measurements of model security.

The behavioral testing stage then exposes the model to a series of defined scenarios. A prompt injection test might provide an instruction that conflicts with the model's assigned constraints, while an indirect injection test places an adversarial instruction inside material the model is expected to treat as data. A data-handling scenario might provide more information than the model needs to complete a legitimate task and examine whether its response unnecessarily reproduces sensitive details. Tool-use scenarios present controlled, nonexecuting tools and examine whether the model attempts actions that it has not been authorized to perform. Each result, pass or fail, goes into a reproducible record of the model's behavior that another reviewer can examine.

In the current reference implementation, published demonstration results use placeholder scoring calibration and should not be interpreted as final trust ratings. That is an important methodological boundary. The system demonstrates how test results and research findings can be gathered, organized, reviewed, and retained, but an organization making actual deployment decisions must first validate its scoring criteria, establish appropriate thresholds, and conduct testing sufficient for its intended risks. Publishing a number is straightforward; establishing what that number means, and how much confidence should be placed in it, is the more important challenge.

The value of this example becomes clearer when the model is considered for more than one application. A development team proposing a limited internal assistant could consult the model's recorded assurance results, identify the domains relevant to that application, and determine what additional testing remains necessary. A second team considering the same model for an autonomous workflow could reuse the existing evidence while applying more demanding requirements to tool use, security behavior, and operational maturity. If a subsequent model version is released, the assessment team can evaluate the new version against the existing baseline and avoid reconstructing the entire assessment from the beginning. The registry therefore supports both the immediate selection decision and the longer-term responsibility to maintain assurance as the technology changes.

From Reference Implementation to Enterprise Adoption

The next step is developing a meaningful testing baseline. The reference implementation's behavioral suites give examples of how repeatable tests can examine model constraints, data handling, tool use, and consistency. They are a starting set and do not cover every possible threat or operational requirement. A financial institution might introduce tests involving unauthorized transaction instructions, disclosure of account information, or manipulation of financial decision processes. A healthcare organization might focus on inappropriate disclosure of patient information, unsafe responses in clinical workflows, and the model's handling of uncertain or incomplete information. A defense contractor might require additional attention to controlled information, provenance, supply-chain integrity, and attempts to induce unauthorized disclosure. These tests should come from threat modeling and analysis of intended applications.

Organizations must also establish how assurance results will be interpreted. A scoring system is useful only when its criteria are understood, applied consistently, and supported by evidence. The six domains give a common structure, but the relative importance of each domain, the thresholds used for particular applications, and the findings that require escalation should be calibrated against organizational requirements. The objective is for reviewers working from the same evidence and criteria to reach conclusions that are consistent, explainable, and open to challenge. Independent review, periodic calibration, and comparison with actual test and operational findings are therefore important parts of the program.

A model's overall assurance score gives a starting point for selection, and application approval requires additional consideration. An organization proposing a new AI application should identify the data involved, the users and systems with which the model will interact, the actions it may perform, and the consequences of an undesirable response. Those characteristics establish which assurance domains require particular attention and what additional application-specific tests must be performed. A model may meet the organization's baseline assurance requirements yet remain unsuitable for an application involving sensitive information or autonomous actions. Conversely, a model with moderate general assurance may be acceptable when its intended function is narrow and appropriate safeguards limit the consequences of failure.

This application-level decision must remain connected to established security engineering practices. A model with strong assurance results can still be incorporated into an insecure application, particularly when it is given excessive permissions, exposed to untrusted information without appropriate controls, or connected to sensitive systems without meaningful authorization boundaries. Traditional application security testing, access controls, logging, monitoring, and incident response remain necessary.^[10] Model assurance contributes information about the model's characteristics and behavior, while application security determines whether the complete system has been designed and implemented appropriately. Treating the two as complementary disciplines avoids creating a false sense of security around a model simply because it performed well in isolation.

The program also needs clearly assigned responsibility for keeping assurance records current. New model versions, changed serving providers, newly found weaknesses, and aging evidence should trigger review according to the significance of the change. In some cases, the existing assessment may remain relevant after a limited examination; in others, the change may require substantial retesting or a new approval decision. The organization should be able to identify which applications rely on an affected model and determine whether existing safeguards remain sufficient. Over time, this process can become part of the organization's normal AI development and change-management activities.

Toward Continuous Model Assurance

The practical response is to make model assurance a continuous part of the AI lifecycle. Models should be assessed before they are adopted, their evidence and results retained in a registry, and their suitability evaluated against the requirements of each proposed application. New versions should be examined before replacing existing ones, and significant new findings should prompt reconsideration of models already in use. Automation makes these activities more practical by reducing the effort required to collect information, run repeatable tests, and compare results, while human reviewers remain responsible for interpreting the evidence and making consequential decisions.

Organizations have spent years learning how to assess the security of the software they build and acquire. As AI models become fundamental components of that software, they deserve a comparable level of scrutiny. The frameworks already exist, the need is evident, and much of the assessment process can now be automated. What remains is to make model assurance a normal, repeatable part of how organizations adopt and operate AI.

Talk to Us About Model Assurance

UltraViolet Cyber’s Risk Advisory practice assesses the AI models organizations plan to adopt, using behavioral testing, evidence research, and human-reviewed scoring across six domains. If you are evaluating open-weight models and want a repeatable way to establish trust in them, contact John Waller and the Risk Advisory team.

CONTACT US

References

1. OpenRouter. The Open Weight Models that Matter. June 2026. https://openrouter.ai/blog/insights/the-open-weight-models-that-matter-june-2026/ ↩
2. UltraViolet Cyber. AISec Study, Inaugural Data-Pool Report. August 2026. https://www.uvcyber.com/resources/guides/aisec-study-closing-the-gap-in-enterprise-ai-security ↩
3. OWASP GenAI Security Project. Top 10 for Large Language Model Applications and Top 10 for Agentic Applications. https://genai.owasp.org/ ↩
4. National Institute of Standards and Technology. Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST AI 100-1, 2023. https://www.nist.gov/itl/ai-risk-management-framework ↩
5. National Institute of Standards and Technology. Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. NIST AI 600-1, July 2024. https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf ↩
6. International Organization for Standardization. ISO/IEC 42001:2023, Artificial Intelligence Management System. https://www.iso.org/standard/81230.html ↩
7. Cloud Security Alliance. AI Controls Matrix. https://cloudsecurityalliance.org/artifacts/ai-controls-matrix-v1-1 ↩
8. Open Source Security Foundation. Supply-chain Levels for Software Artifacts (SLSA). https://slsa.dev/ ↩
9. MITRE. Adversarial Threat Landscape for Artificial-Intelligence Systems (ATLAS). https://atlas.mitre.org/ ↩
10. National Institute of Standards and Technology. Secure Software Development Practices for Generative AI and Dual-Use Foundation Models. NIST SP 800-218A, July 2024. https://csrc.nist.gov/pubs/sp/800/218/A ↩