LLMOps (2/2): safety, governance and platforms

Key categories: prompt injection; sensitive information disclosure; supply-chain risks (models, datasets, plugins and MCP servers); data and model poisoning; improper output handling (rendering model output as HTML or running it as SQL or shell); excessive agency (too many tools or permissions); system prompt leakage; vector and embedding weaknesses (cross-tenant leakage in a shared vector store); misinformation; and unbounded consumption (cost and denial-of-service through huge or looping requests).

Walk your flagships through the list and write down one mitigation per relevant item. That document is a strong portfolio artefact.

Note the overlap with classic web security: output handling is XSS and injection again, and excessive agency is least privilege again.

Direct injection: the user types instructions to override yours ('ignore previous instructions…'). Indirect injection: malicious instructions hide in content the model processes: a web page, an email, a PDF, a retrieved chunk, a tool result, a tool description. The user may be the victim, not the attacker.

The danger scales with capability: an injected instruction in a RAG answer can mislead; in an agent with email and file tools it can exfiltrate data (for example via a link or image URL containing secrets) or take actions.

There's no complete fix, because models can't reliably separate instructions from data. Defence in depth: delimit and label untrusted content; least privilege for tools; human approval for side effects; filter outputs (block unknown URLs and markdown images); avoid combining private data, untrusted content and an exfiltration channel in one agent; and test with an injection eval set.

Input guardrails: topic or intent classification (is this in scope?), jailbreak and injection detectors, PII detection and redaction before text is sent to a third party, size limits. Output guardrails: schema validation, moderation for harmful content, PII leakage checks, grounding checks, URL allow-lists.

Tools: Guardrails AI, NeMo Guardrails, moderation APIs, Presidio for PII, or small fine-tuned classifiers. Deterministic checks (regex, schema, allow-lists) are cheap and reliable; use them first.

Measure false positives too. An over-eager guardrail that blocks legitimate users is a product bug, and rare-attack detectors face exactly the base-rate problem from week 5.

Audit trails: log model, prompt version, inputs, outputs, tool calls and approvals with retention policies, while respecting privacy (redact or minimise what you store). Model and system cards: document intended use, limitations, evaluation results and known risks.

Data residency and privacy: where prompts and outputs are processed and stored, provider data-retention and training policies, GDPR rights (access, deletion, including of memories).

EU AI Act: risk-based obligations. Prohibited practices, strict requirements for high-risk uses (e.g. hiring, credit, education), transparency duties (disclose AI interactions and synthetic content), and obligations for general-purpose model providers. The NIST AI RMF offers a practical risk-management structure.

Cloud platforms offer many foundation models behind one API, with IAM, VPC/private networking, regional data residency, logging, guardrail services, managed knowledge bases (RAG), agent services and fine-tuning.

Why enterprises use them: procurement and billing through an existing cloud contract, compliance, and data staying in their account and region. Trade-offs: feature lag versus direct provider APIs, platform lock-in, and region-dependent model availability.

Be able to map your stack onto each: where the vector store, tracing, guardrails and model endpoints live.

Spring AI brings portable chat-model and embedding abstractions, VectorStore implementations (including pgvector), structured output to Java records, tool calling and RAG 'advisors' to Spring Boot. LangChain4j is the alternative Java library.

Rebuild the core of Flagship 1 as a Spring Boot service with pgvector, run your existing eval set against it, and compare the numbers side by side.

Optional, one week, and it makes your résumé immediately legible to enterprise hiring managers if you're coming from a Java background.

Red-teaming probes for failures: prompt injection (direct and indirect), jailbreaks, data exfiltration, harmful outputs, PII leaks, excessive agency and denial-of-wallet (inputs that trigger runaway cost).

Combine manual creativity with automated scanners: tools like garak and promptfoo's red-team mode generate thousands of attack variants and grade responses. Many-shot and multi-turn attacks matter, not just single prompts.

Each successful attack becomes a test case in the eval suite, so defences are verified on every release, the same loop as security regression testing.