AI and LLM security requirements come down to eight areas: prompt injection defence, RAG data leakage controls, agent permission boundaries, output handling, model and supply chain vetting, logging and monitoring, and alignment with the OWASP LLM Top 10. Below is a practical checklist you can run against any product that touches a large language model, whether it is a customer-facing chatbot, an internal copilot, or an autonomous agent with tool access.
Most companies bolt an LLM onto a product roadmap faster than their security review process can keep up. That gap is where the risk lives. This checklist is written for engineering and security leads in Canadian tech hubs like Toronto, Waterloo, Ottawa, and Vancouver who are shipping AI features and need to know what "secure" actually means before a customer, auditor, or investor asks.
Why AI and LLM Security Requirements Are Different From Traditional AppSec
Traditional application security assumes a fixed set of inputs and code paths. LLMs break that assumption. The "input" is natural language, the "logic" is a probabilistic model you did not fully train or audit, and the "output" can trigger real actions if the model is wired to tools, databases, or APIs. A SQL injection has a known shape. A prompt injection can be phrased a thousand ways, hidden inside a PDF, an email signature, or a web page the model is asked to summarize. That is why generic penetration testing checklists miss LLM-specific failure modes, and why a dedicated AI security assessment looks at the model layer, not just the network and application layers around it.
Prompt Injection Defence Checklist
- Test direct injection. Can a user override the system prompt with instructions typed straight into the chat box? Try it before an attacker does.
- Test indirect injection. If the model reads external content (web pages, PDFs, emails, tickets), can instructions hidden in that content hijack its behaviour?
- Separate instructions from data. Use structured prompts (delimiters, XML tags, separate message roles) so the model can distinguish "trusted instruction" from "untrusted content it is processing."
- Apply output filtering, not just input filtering. Injection defences that only screen user input miss attacks embedded in retrieved documents.
- Re-test after every prompt or model change. A defence that worked against GPT-4 may not hold against a new model version or a modified system prompt.
Retrieval-Augmented Generation (RAG) Leakage Checklist
- Enforce document-level access control at retrieval time, not just at the application layer. If your vector store returns chunks the requesting user should not see, the model will happily summarize them.
- Test cross-tenant leakage in multi-tenant SaaS products. A single shared vector index without tenant filtering is one of the most common RAG failures we see.
- Check for embedding inversion risk. Sensitive source data embedded without care can sometimes be partially reconstructed from its vector representation.
- Sanitize retrieved content before it reaches the prompt. Treat retrieved chunks as untrusted input, the same way you would treat user-submitted text.
- Log what was retrieved, not just what was answered. If a leak happens, you need to know which documents the model actually saw.
Agent and Tool-Use Abuse Checklist
Agentic systems, where an LLM can call APIs, run code, send emails, or query databases, multiply the blast radius of any prompt injection. Treat every tool the agent can call as an attack surface.
- Apply least privilege to every tool binding. An agent that only needs to read a calendar should not have a token that can also delete events or read other users' data.
- Require human approval for irreversible or high-impact actions (payments, deletions, external emails, permission changes).
- Test for goal hijacking. Can an attacker, through injected content, redirect the agent's task toward exfiltrating data or performing unauthorized actions?
- Rate-limit and sandbox code execution tools. If the agent can run code, assume that sandbox will eventually be tested by someone hostile.
- Chain-test, not just unit-test. A single tool call might look safe in isolation but dangerous when chained with two or three others.
Aligning With the OWASP LLM Top 10
The OWASP Top 10 for LLM Applications is the closest thing to an industry-standard checklist right now, and it should anchor your internal review process. At minimum, confirm you have tested for and documented mitigations against:
- Prompt injection (LLM01)
- Insecure output handling, where model output is passed unsanitized into a shell, database query, or rendered HTML (LLM02)
- Training data poisoning, relevant if you fine-tune on user-submitted or third-party data (LLM03)
- Model denial of service through resource-exhausting inputs (LLM04)
- Supply chain vulnerabilities in third-party models, plugins, and datasets (LLM05)
- Sensitive information disclosure through the model's own outputs (LLM06)
- Insecure plugin or tool design (LLM07)
- Excessive agency, giving the model more autonomy or permission than the task requires (LLM08)
- Overreliance on model output without human verification in high-stakes decisions (LLM09)
- Model theft or unauthorized extraction of proprietary weights and prompts (LLM10)
Mapping your findings to this framework matters beyond the technical exercise. Enterprise procurement teams and auditors increasingly ask for it by name, and it gives your board or compliance team a common language for AI risk.
Output Handling and Data Exposure Checklist
- Never render raw model output as executable HTML or JavaScript without sanitization. Insecure output handling is one of the most exploitable and most overlooked issues in production LLM apps.
- Redact or block PII and secrets from appearing in model responses, especially in customer support or internal knowledge-base copilots.
- Test what happens when a user asks the model to repeat its system prompt. System prompts often contain business logic, pricing rules, or internal instructions you do not want exposed.
- Watch for regurgitation of training or fine-tuning data, particularly if you fine-tuned on internal or customer data.
Model and Supply Chain Vetting Checklist
- Inventory every model, plugin, and third-party API your AI features depend on, the same way you would inventory open-source libraries.
- Review vendor data handling terms for whether your prompts and data are used for further training, and where they are processed and stored (relevant under PIPEDA and, for Quebec-based operations, Law 25).
- Pin model versions where behaviour matters for compliance or safety, and re-test before accepting silent upgrades.
- Assess self-hosted or open-weight models for the same rigour you'd apply to any self-hosted software, including patching and access control on the serving infrastructure.
Logging, Monitoring, and Incident Response Checklist
- Log prompts, retrieved context, and model outputs with enough fidelity to reconstruct an incident, while respecting data retention and privacy obligations.
- Set up anomaly detection for unusual prompt patterns, repeated injection attempts, or abnormal tool-call volume from a single session.
- Define an AI-specific incident response runbook. A leaked system prompt, a jailbreak that produced harmful output, and an agent that took an unauthorized action each need different containment steps than a typical breach.
- Review logs regularly, not just after something breaks. Injection attempts often show up in logs long before they succeed.
How Often Should You Test AI and LLM Systems
Treat AI security testing as continuous, not a one-time gate before launch. Models get updated by vendors, prompts get tweaked by product teams, and new tools get bolted onto agents on a regular cadence, any one of which can reopen a vulnerability that was previously closed. A reasonable baseline is a full assessment before launch, a focused re-test after any material model, prompt, or tool change, and a lightweight review on a quarterly cycle for systems in active development. If your organization is also pursuing AI governance standards, this testing cadence pairs directly with the controls required under ISO 42001 readiness work.
Get an AI Security Assessment
This checklist gives you a starting point, but a real assessment tests your specific prompts, your specific data flows, and your specific agent permissions against live attack techniques, not a generic list. Traztech is a boutique Canadian security and compliance consultancy led by a published security researcher with six CVEs to his name, serving technology companies across Toronto, Waterloo, Ottawa, Vancouver, Calgary, and Montreal. If you are shipping an LLM-powered product and need to know where it actually breaks, contact us to scope an AI security assessment.