AI and LLM Security for Healthtech

Healthtech companies are shipping AI features faster than their security programs can keep up. Clinical documentation assistants, patient-facing chatbots, RAG systems pulling from electronic health records, and autonomous agents that triage messages or schedule care are all now in production at companies that, two years ago, had none of this in scope. The problem is that most of these systems were built and tested for accuracy, not for security. Nobody red-teamed them against a hostile prompt.

That gap matters more in healthtech than almost anywhere else. A jailbroken customer support bot at a retailer is embarrassing. A jailbroken clinical assistant that leaks protected health information (PHI), fabricates a dosage recommendation, or lets an attacker pivot from a chat interface into a patient record system is a regulatory incident, a breach notification, and a trust problem your sales team will be fighting for a year.

Why healthtech carries higher stakes

Three things make LLM security a different conversation for healthtech than for a generic SaaS company.

First, the data. Large language models deployed in clinical or administrative workflows are almost always connected to PHI, whether directly through a database or indirectly through retrieval-augmented generation (RAG) pipelines that pull patient records, lab results, or claims data into the model's context window. If that retrieval layer isn't scoped correctly, a low-privilege user can craft a query that returns another patient's data. This is not a hypothetical, it's one of the most common findings in RAG-backed systems that haven't been tested.

Second, the regulatory exposure. PHIPA in Ontario, PIPEDA federally, and HIPAA for any company touching US patients all treat an AI-driven data disclosure the same way they treat any other breach: notification obligations, potential fines, and a very uncomfortable conversation with enterprise customers who signed a business associate agreement or a data processing addendum on the assumption your AI systems were tested. If you're pursuing SOC 2 or preparing for enterprise procurement, unreviewed AI systems are increasingly a specific line item auditors and security questionnaires ask about.

Third, the agentic risk. A growing number of healthtech products give an LLM tool-calling access, letting it query a scheduling system, send a message, update a record, or trigger a workflow. Every tool you hand an agent is a new attack surface. If an attacker can influence the model's input (through a message, an uploaded document, or even a field in a patient intake form), and the model has a tool that writes to a system of record, you have a path from a chat box to unauthorized data modification.

What actually gets tested

An AI security assessment for a healthtech product isn't a generic penetration test with an LLM bolted on. It follows the OWASP LLM Top 10 as a baseline and adapts it to how your product actually uses AI. In practice, that means testing for:

• Prompt injection, both direct (a user typing adversarial instructions) and indirect (malicious instructions hidden in a document, email, or web page the model later ingests).
• RAG leakage, verifying that retrieval scoping actually enforces per-user or per-tenant boundaries, and that the model can't be coaxed into surfacing data outside the requester's authorization.
• Agent tool abuse, probing whether an attacker can manipulate the model into calling tools it shouldn't, with parameters it shouldn't, on behalf of a user it shouldn't be able to impersonate.
• Output handling and insecure integration, checking whether model output is trusted downstream in ways that create injection or execution risk in connected systems.
• Data and model supply chain issues, including how third-party models, fine-tuning data, and plugins are vetted before they touch patient data.

The goal isn't a checklist exercise. It's a working exploit chain, where possible, that shows exactly how a real attacker would get from an unauthenticated or low-privilege entry point to PHI exposure or unauthorized action, along with a clear remediation path your engineering team can act on without a six-month rewrite.

How traztech scopes this for healthtech companies

We start with an architecture review before any testing begins. We need to understand what the model can see, what it can touch, and who it's supposed to be able to see and touch on behalf of. For a healthtech company, that usually means mapping the RAG retrieval layer against your actual data access model, not just the happy path.

From there, our AI security assessments combine manual adversarial testing led by Jacob Masse, a published CVE researcher, with structured coverage of the OWASP LLM Top 10 categories relevant to your build. We test the way an attacker actually would: crafted prompts, poisoned documents, adversarial file uploads, and tool-call manipulation, not just automated scanners running canned payloads against your API. The output is a report your engineering and compliance teams can both use: technical findings with reproduction steps for developers, and a plain-language summary of PHI and regulatory exposure for whoever owns your compliance posture. If you're also working toward SOC 2 or building out a broader compliance program, we scope the AI assessment to feed directly into that evidence trail rather than sitting as a separate exercise.

The cost of skipping this

Most healthtech leaders we talk to know their AI features carry risk. What they underestimate is how quickly that risk becomes a blocker. Enterprise health system procurement teams are starting to ask specific questions about LLM testing in security questionnaires. Cyber insurance underwriters are starting to ask too. And a single incident involving PHI moving through an untested model is exactly the kind of story that ends up in a breach disclosure, not a case study.

Testing your AI systems before an attacker, an auditor, or a regulator finds the gap is a lot cheaper than the alternative, and for a healthtech company it's fast becoming table stakes rather than a nice-to-have.

If your product uses an LLM anywhere near patient data, contact traztech to talk through scoping an AI security assessment for your specific architecture.

The Retrieval Layer Is Where Authorization Actually Breaks

Nearly every serious RAG finding we produce comes down to the same architectural decision: where the access check happens. Teams typically embed everything into one vector index, then filter results by tenant or user after retrieval, or worse, instruct the model in its system prompt to only discuss records belonging to the current user. Both are broken. Post-retrieval filtering means the sensitive chunk was already loaded into context, and anything in context can be surfaced through paraphrase, summarization, or a translated request even when the filter drops the citation. A system-prompt instruction is not an access control at all.

The version that holds up runs the authorization check at query time, against the same policy the rest of your application uses, before any embedding is retrieved. That usually means per-tenant index separation or metadata filters applied inside the vector query, plus the retrieval service running as the requesting user rather than a service account with global read.

There is a second failure underneath it. Documents get chunked and embedded once, at ingestion, carrying whatever permissions existed then. When a clinician moves teams, a patient revokes consent, or a record is corrected, the index does not know. We routinely find embeddings for records deleted from the source database months earlier. Deletion and retention obligations under PHIPA and PIPEDA apply to the vector store as much as the database, and almost nobody has a purge path built for it.

Audit Logging That an Investigation Can Actually Use

Ontario's PHIPA gives individuals the right to know who accessed their record, and health information custodians are expected to be able to produce that. If a clinician asks your assistant to summarize a patient chart, that is an access to the record, and it belongs in your audit trail with the same fidelity as someone opening the chart directly.

Most AI features log the prompt, the response, and a timestamp. That is not enough for an investigation. What you need recorded is which records were retrieved, on whose authority, which model version and prompt template were in use, which tools were called with which parameters, and what the model returned. Without the retrieval list, you cannot answer the only question that matters after an incident: whose information was exposed, and to whom.

The counterweight is that those logs now hold patient information in plain text. They inherit the same retention limits, access restrictions, and breach obligations as the clinical data, so you cannot ship them to a general-purpose observability tool half the engineering team can query.

The Model Provider Is a Subprocessor, and Your Contract Has to Say So

If PHI touches a hosted model, that provider is processing health information on your behalf. Three things need to be true and documented. The provider has to appear on the sub-processor list you show customers, which means notifying them when you switch or add one. Your commercial terms need to cover training and retention explicitly, including zero-retention or short-retention configuration where the provider offers it, and you need to be able to show which API tier or setting enforces it rather than pointing at a marketing page. And where US patients are involved, the provider needs to sign a business associate agreement, which not every model endpoint or region supports.

Check the deployment too. A team that negotiated zero-retention on the enterprise endpoint and then routes overflow traffic to a cheaper consumer-tier key during load spikes has undone the contract. That is a configuration question, and it belongs in change management, not legal review.

Human in the Loop Is Only a Control if the Human Can Actually Refuse

The most common mitigation offered in security questionnaires is that a clinician reviews every AI output. It is a legitimate control when the review is real. It stops being one when the interface presents a pre-filled note with an accept button, the reviewer handles dozens per hour, and the system records no evidence of what was changed.

If you are going to claim this control, instrument it. Capture the model draft and the final signed version separately, so edit rates are measurable. An assistant with a ninety-nine percent unedited acceptance rate is not being reviewed, and an auditor or a plaintiff's counsel will read the metric exactly that way.

When You Should Not Buy This Assessment

If your AI feature is a marketing site chatbot with no connection to patient data and no tool access, you do not need an LLM-specific assessment. Cover it in your normal application testing.

If you have never had a conventional penetration test of the application the AI feature sits inside, start there. We have found more real PHI exposure through ordinary broken object-level authorization in an API than through prompt injection, and testing the model while the surrounding application is untested puts the money in the wrong place.

If you are still prototyping and the feature is not in front of patients or clinicians, wait. An assessment of an architecture that will be rewritten in six weeks produces findings that expire with the code. Spend that time instead on the retrieval authorization design, because that decision is expensive to reverse later.

Where testing earns its cost is when a real user population is touching a model that can read or write clinical data, or when an enterprise health system has put AI questions in front of your deal. If that is where you are, describe the architecture to us and we will scope to the part that carries the risk. If your AI surface changes every sprint, a point-in-time report ages badly and a retainer that retests on release is the more honest arrangement.