Skip to content

AI Security

AI security assurance

What this is

AI security assurance tests deployed artificial intelligence systems against adversarial conditions and monitors their behaviour over time. It covers prompt injection, jailbreaking, training and context data extraction, guardrail bypass, model denial of service, and the lifecycle monitoring that detects a system behaving differently after deployment.

The problem

Your AI system passed its evaluation. Evaluations are not adversaries.

Model evaluation measures whether a system does the right thing when used as intended. Security assurance asks what it does when someone is actively trying to make it do the wrong thing, and the two produce very different answers. A system with excellent benchmark scores can still be talked into disclosing its instructions, retrieving another customer's data through the retrieval layer, or executing an action on behalf of text it read in a document.

The failure modes are also unlike traditional application security. Prompt injection is not input validation - the instruction and the data occupy the same channel by design. An agent with tool access turns a language problem into an execution problem. A retrieval layer inherits the access control of whatever it indexed, which is frequently more than the user should see.

And AI systems change without being redeployed. The provider updates the model, the retrieval corpus grows, usage patterns shift. Behaviour that was safe at launch is not automatically safe six months later, which makes monitoring part of the control rather than an optional extra.

Who this is for

  • Organisations with generative AI in a customer-facing product.
  • Teams running retrieval-augmented generation over data with differentiated access.
  • Anyone deploying agents with tool access, function calling or the ability to act on external content.
  • Companies whose enterprise customers have begun asking for AI-specific penetration test results.
  • Firms preparing for ISO 42001 or EU AI Act obligations that require testing before and after deployment.

Delivery

How the engagement actually runs

Sprints inside your engineering cycle, with findings arriving continuously rather than a report arriving at the end. Durations are elapsed time for a typical scope, and get re-scoped against your estate before anything is committed.

  1. 011 week

    Threat model

    Understand the architecture and establish what an attacker would actually be trying to achieve, which shapes everything that follows.

  2. 022-3 weeks

    Adversarial testing

    Structured red teaming against the deployed system, with critical findings reported the day they are confirmed rather than held for the report.

  3. 031-2 weeks

    Guardrails and monitoring

    Strengthen the controls that failed, and specify the monitoring that catches the ones that will fail later.

  4. 041 week

    Retest and hand over

    Verify the fixes actually fix it, and hand over a test set your team can rerun against future releases.

Deliverables

What you actually receive

Artefacts your team owns and can maintain after the engagement, not a report that ages the moment it lands.

Deliverables for AI security assurance, with what each one contains and when it lands
DeliverableWhat it containsWhen
AI threat modelThe specific attack surface of your system: trust boundaries, where instruction meets data, what the model can reach, and what an attacker gains from each failure.Week 1
Adversarial test reportFindings from structured red teaming - prompt injection, jailbreak, extraction, guardrail bypass and abuse of tool access - each with a working reproduction.Week 3-4
Retrieval and access reviewWhether the retrieval layer respects the access control of its sources, which is the most commonly broken assumption in RAG deployments.Week 3
Guardrail assessmentHow the existing guardrails perform under pressure, where they fail, and what the residual risk is once they are strengthened.Week 4
Monitoring specificationWhat to log, what to alert on, and how to detect behaviour drifting after deployment - written for your platform rather than in the abstract.Week 5
Prioritised remediation backlogFindings as engineering work with owners and severity, not a PDF with a risk matrix.Week 5

Frameworks this touches

FAQ

Questions people actually ask

What is prompt injection and why can it not just be filtered?

Prompt injection is an attack where instructions reach the model through content it processes - a document, a web page, an email, a database field - and the model follows them. It cannot be reliably filtered because instructions and data share one channel by design: the model has no mechanism to distinguish a developer's instruction from text that looks like one. Mitigation is architectural rather than filtering - constraining what the model can reach, requiring confirmation for consequential actions, and treating model output as untrusted input to anything downstream.

How is AI red teaming different from a normal penetration test?

A penetration test targets the application, infrastructure and access control around a system. AI red teaming targets the model's behaviour: whether it can be induced to ignore its instructions, disclose its context or training data, produce prohibited output, or misuse tools it has been granted. Both are needed and they find different things - an AI system with perfect infrastructure security can still leak another customer's data through its retrieval layer.

Can guardrails be relied on?

Guardrails reduce risk and do not eliminate it. Classifier-based and instruction-based guardrails are themselves models and can be circumvented by the same techniques used against the primary model. They are worth having as one layer, and the failure is treating them as the control rather than as one of several. The controls that hold under pressure are architectural: least privilege on what the model can reach, human confirmation before consequential actions, and no implicit trust in model output.

How often should AI systems be retested?

Retest on change, and periodically regardless. Change includes a model version update from the provider, a new tool or data source connected, a change in the system prompt, or a material shift in how the system is used. Periodic retesting matters because the adversarial techniques evolve continuously - a system tested six months ago was tested against a materially older set of attacks.

More in AI Security

Start

Three ways in, depending on how close the deadline is.

Or just email a human: hello@cloudgap.ai

We reply within24 hours