AI and LLM Security Requirements: A Practical Checklist

AI and LLM security requirements come down to eight areas: prompt injection defence, RAG data leakage controls, agent permission boundaries, output handling, model and supply chain vetting, logging and monitoring, and alignment with the OWASP LLM Top 10. Below is a practical checklist you can run against any product that touches a large language model, whether it is a customer-facing chatbot, an internal copilot, or an autonomous agent with tool access.

Most companies bolt an LLM onto a product roadmap faster than their security review process can keep up. That gap is where the risk lives. This checklist is written for engineering and security leads in Canadian tech hubs like Toronto, Waterloo, Ottawa, and Vancouver who are shipping AI features and need to know what "secure" actually means before a customer, auditor, or investor asks.

Why AI and LLM Security Requirements Are Different From Traditional AppSec

Traditional application security assumes a fixed set of inputs and code paths. LLMs break that assumption. The "input" is natural language, the "logic" is a probabilistic model you did not fully train or audit, and the "output" can trigger real actions if the model is wired to tools, databases, or APIs. A SQL injection has a known shape. A prompt injection can be phrased a thousand ways, hidden inside a PDF, an email signature, or a web page the model is asked to summarize. That is why generic penetration testing checklists miss LLM-specific failure modes, and why a dedicated AI security assessment looks at the model layer, not just the network and application layers around it.

Prompt Injection Defence Checklist

• Test direct injection. Can a user override the system prompt with instructions typed straight into the chat box? Try it before an attacker does.
• Test indirect injection. If the model reads external content (web pages, PDFs, emails, tickets), can instructions hidden in that content hijack its behaviour?
• Separate instructions from data. Use structured prompts (delimiters, XML tags, separate message roles) so the model can distinguish "trusted instruction" from "untrusted content it is processing."
• Apply output filtering, not just input filtering. Injection defences that only screen user input miss attacks embedded in retrieved documents.
• Re-test after every prompt or model change. A defence that worked against GPT-4 may not hold against a new model version or a modified system prompt.

Retrieval-Augmented Generation (RAG) Leakage Checklist

• Enforce document-level access control at retrieval time, not just at the application layer. If your vector store returns chunks the requesting user should not see, the model will happily summarize them.
• Test cross-tenant leakage in multi-tenant SaaS products. A single shared vector index without tenant filtering is one of the most common RAG failures we see.
• Check for embedding inversion risk. Sensitive source data embedded without care can sometimes be partially reconstructed from its vector representation.
• Sanitize retrieved content before it reaches the prompt. Treat retrieved chunks as untrusted input, the same way you would treat user-submitted text.
• Log what was retrieved, not just what was answered. If a leak happens, you need to know which documents the model actually saw.

Agent and Tool-Use Abuse Checklist

Agentic systems, where an LLM can call APIs, run code, send emails, or query databases, multiply the blast radius of any prompt injection. Treat every tool the agent can call as an attack surface.

• Apply least privilege to every tool binding. An agent that only needs to read a calendar should not have a token that can also delete events or read other users' data.
• Require human approval for irreversible or high-impact actions (payments, deletions, external emails, permission changes).
• Test for goal hijacking. Can an attacker, through injected content, redirect the agent's task toward exfiltrating data or performing unauthorized actions?
• Rate-limit and sandbox code execution tools. If the agent can run code, assume that sandbox will eventually be tested by someone hostile.
• Chain-test, not just unit-test. A single tool call might look safe in isolation but dangerous when chained with two or three others.

Aligning With the OWASP LLM Top 10

The OWASP Top 10 for LLM Applications is the closest thing to an industry-standard checklist right now, and it should anchor your internal review process. At minimum, confirm you have tested for and documented mitigations against:

• Prompt injection (LLM01)
• Insecure output handling, where model output is passed unsanitized into a shell, database query, or rendered HTML (LLM02)
• Training data poisoning, relevant if you fine-tune on user-submitted or third-party data (LLM03)
• Model denial of service through resource-exhausting inputs (LLM04)
• Supply chain vulnerabilities in third-party models, plugins, and datasets (LLM05)
• Sensitive information disclosure through the model's own outputs (LLM06)
• Insecure plugin or tool design (LLM07)
• Excessive agency, giving the model more autonomy or permission than the task requires (LLM08)
• Overreliance on model output without human verification in high-stakes decisions (LLM09)
• Model theft or unauthorized extraction of proprietary weights and prompts (LLM10)

Mapping your findings to this framework matters beyond the technical exercise. Enterprise procurement teams and auditors increasingly ask for it by name, and it gives your board or compliance team a common language for AI risk.

Output Handling and Data Exposure Checklist

• Never render raw model output as executable HTML or JavaScript without sanitization. Insecure output handling is one of the most exploitable and most overlooked issues in production LLM apps.
• Redact or block PII and secrets from appearing in model responses, especially in customer support or internal knowledge-base copilots.
• Test what happens when a user asks the model to repeat its system prompt. System prompts often contain business logic, pricing rules, or internal instructions you do not want exposed.
• Watch for regurgitation of training or fine-tuning data, particularly if you fine-tuned on internal or customer data.

Model and Supply Chain Vetting Checklist

• Inventory every model, plugin, and third-party API your AI features depend on, the same way you would inventory open-source libraries.
• Review vendor data handling terms for whether your prompts and data are used for further training, and where they are processed and stored (relevant under PIPEDA and, for Quebec-based operations, Law 25).
• Pin model versions where behaviour matters for compliance or safety, and re-test before accepting silent upgrades.
• Assess self-hosted or open-weight models for the same rigour you'd apply to any self-hosted software, including patching and access control on the serving infrastructure.

Logging, Monitoring, and Incident Response Checklist

• Log prompts, retrieved context, and model outputs with enough fidelity to reconstruct an incident, while respecting data retention and privacy obligations.
• Set up anomaly detection for unusual prompt patterns, repeated injection attempts, or abnormal tool-call volume from a single session.
• Define an AI-specific incident response runbook. A leaked system prompt, a jailbreak that produced harmful output, and an agent that took an unauthorized action each need different containment steps than a typical breach.
• Review logs regularly, not just after something breaks. Injection attempts often show up in logs long before they succeed.

How Often Should You Test AI and LLM Systems

Treat AI security testing as continuous, not a one-time gate before launch. Models get updated by vendors, prompts get tweaked by product teams, and new tools get bolted onto agents on a regular cadence, any one of which can reopen a vulnerability that was previously closed. A reasonable baseline is a full assessment before launch, a focused re-test after any material model, prompt, or tool change, and a lightweight review on a quarterly cycle for systems in active development. If your organization is also pursuing AI governance standards, this testing cadence pairs directly with the controls required under ISO 42001 readiness work.

Get an AI Security Assessment

This checklist gives you a starting point, but a real assessment tests your specific prompts, your specific data flows, and your specific agent permissions against live attack techniques, not a generic list. TrazTech is a boutique Canadian security and compliance consultancy led by a published security researcher with five CVEs to his name, serving technology companies across Toronto, Waterloo, Ottawa, Vancouver, Calgary, and Montreal. If you are shipping an LLM-powered product and need to know where it actually breaks, contact us to scope an AI security assessment.

How to Actually Run a Prompt Injection Test

The checklist says test for injection. Doing it well is harder than it sounds, because the system under test is probabilistic and a single successful payload is not a bug report anyone can act on. What you need is a method that produces a rate rather than an anecdote.

Start by defining the violation you are testing for, in terms a reviewer can judge without argument. "The model behaved badly" is not testable. "The model disclosed content from the system prompt", "the model called the refund tool with a customer ID the session did not authenticate", "the model emitted text from a document outside the requesting tenant" all are. Write the violation conditions first, because they determine what you have to log to detect a success.

Then build a payload corpus rather than improvising in a chat window. Group it by technique: direct instruction override, role play and persona framing, encoding and obfuscation, language switching, instruction smuggling inside markup or code blocks, and payloads delivered indirectly through content the model retrieves. Indirect delivery deserves the most attention because it is the class that scales for an attacker. If your product summarizes documents, ingests email, or browses a URL, then anyone who can get content in front of the model is an untrusted input source, including people who never touch your product.

Run each payload multiple times, because temperature and model routing make a single pass meaningless. A payload that succeeds two times in twenty is not fixed, it is intermittent, and intermittent injection against a consequential tool is worse than a reliable one because it survives casual retesting. Record the success rate per technique and track it across releases. That number is the artefact worth having; a list of clever prompts is not.

Test any guardrail separately from the application. A combined pass rate hides which layer is doing the work and which one you can safely change.

Non-Determinism Breaks Your Existing Test Discipline

Engineering teams assume a security fix is verified when the failing test passes. With models, a fix is verified when the failure rate across a corpus falls below a threshold you decided in advance and stays there through a model version change you did not control. That is a different discipline and it needs different plumbing.

Keep the injection corpus in the repository next to the code, run it in continuous integration on any change to the system prompt, the retrieval configuration, the tool definitions or the model version pin, and fail the build on regression. Treat prompt changes as production changes subject to review, because a product manager editing a system prompt in a console is making a security-relevant change with none of the controls that would apply to a code deploy. Auditors have started asking about this specifically, and "the prompt lives in a config file under version control with peer review" is a much better answer than "the growth team owns it".

Vendors deprecate pinned versions on their own schedule, so the migration happens whether you plan it or not. The teams that get hurt are the ones where an upgrade changed how strictly the model followed a system instruction and nobody had a corpus to notice.

Threat Modelling the Feature Before You Test It

Most AI security findings are architectural, and you can identify them on a whiteboard before anyone writes a payload. Draw the context window as a container and ask, for every string that can end up inside it, who controls that string. User messages, retrieved chunks, tool results, prior conversation turns, system instructions, and any content fetched from the web are all inputs, and they are not equally trusted even though the model receives them as one undifferentiated blob.

Then draw the outputs and ask what each one can cause. Text rendered to a screen is one risk class. Text passed to a shell, a database query, a template engine or a downstream service is another entirely. Text that becomes a tool call is the highest, and the question that matters there is whose authority the call executes under. An agent that holds a service account with broad permissions and acts on behalf of whichever user is talking to it has collapsed your authorization model into a natural language interface, and no amount of prompt hardening fixes that. The durable fix is to execute tool calls with the requesting user's own credentials and let the downstream system enforce its own access control.

Rank the resulting findings by consequence. A model that can be talked into rude language is an embarrassment; a model that can be talked into reading another tenant's records is an incident with notification obligations attached.

Tool Servers and Connectors Are Your New Supply Chain

Connector ecosystems have made it trivial to give a model access to a ticketing system, a code repository or a customer database with a few lines of configuration, which means the integration decision has moved from a platform team to whoever is building the feature. Treat each connector as a third-party component with a review attached: who publishes it, what scopes the token it requests actually grants, whether tool descriptions are fetched from a remote server at runtime, and what happens to your data once it passes through.

The runtime-fetched description point is the one worth dwelling on. If a tool's description arrives from a server you do not control, it is untrusted content landing directly in the model's instruction context, and a changed description can redirect behaviour with no change to your code and nothing visible in version control. Pin and monitor those definitions the way you pin a dependency version.

The same discipline applies to retrieval sources. A wiki page any employee can edit, a shared inbox, a public issue tracker: each is a path for an outsider to place instructions where your model will read them. Inventory the writeable surfaces, not just the readable ones.

What Buyers and Auditors Are Asking For Now

Enterprise security reviews have added AI sections, and the questions are fairly consistent. Which models and providers do you use, and where is inference performed. Is our data used for training, and is that contractual or merely stated in documentation. What is retained, for how long, and can it be deleted on request. Can a human review model outputs before consequential actions. What testing have you performed against prompt injection, and can you show results. Do you have an AI-specific incident response plan.

Have written answers ready before the questionnaire arrives, with evidence behind each. Under an attestation program the mapping is straightforward: change management covers prompt and model changes, access control covers tool credentials and vector store permissions, vendor management covers model providers, and logging covers prompts and outputs subject to your retention position. For Canadian operations, the data handling answers also need to line up with what you have already committed to under privacy obligations, since the model provider is a processor like any other. If you are pursuing formal AI governance on top of that, our compliance practice covers where those control sets overlap and where they genuinely differ.

When You Should Not Buy an AI Security Assessment

If your AI feature is a thin wrapper that takes user text, sends it to a provider, and renders the response as plain text with no retrieval, no tools and no private data, you do not need an assessment. You need output sanitization so the response cannot execute in your page, rate limiting so the feature cannot be used to burn your API budget, and a note in your privacy documentation about the provider. That is an afternoon of engineering work, not an engagement.

If you have not yet shipped and the design is still moving weekly, testing now measures something that will not exist next month. Spend the time on the trust boundary questions above instead, which are cheap to answer early and expensive to change late, and test once the shape has settled.

If your immediate problem is a customer questionnaire rather than a real exposure, answer it honestly and offer to share test results when you have them. Experienced reviewers can tell the difference between a report from a real test and a report from a checklist walkthrough.

Where an assessment does earn its cost is when the model can act: tools with write access, agents chaining calls, retrieval over data with tenancy or confidentiality boundaries, or anything where a wrong action is not reversible with an apology. That is the point where the failure modes stop being theoretical, and it is also the point where an internal team testing its own design tends to miss things, because the assumptions that created the gap are the same assumptions guiding the test. If you are unsure which side of that line you are on, describe the architecture and we will tell you plainly whether there is anything here worth paying for. If the answer turns out to be ongoing rather than one-off, a retainer is the shape that fits a system changing every sprint.