Building Secure AI Applications Training Course

Module 1 — Understanding Vulnerabilities in AI Applications

Lab: None — Architecture review and technical discussion

An overview of the attack surface from the perspective of application developers.

Topics:

• Architectures for Large Language Models (LLM), Retrieval-Augmented Generation (RAG), and autonomous agents from a developer’s perspective
• The lifecycle of requests and responses within AI-enabled features
• Prompt structure: interactions between system, developer-defined, user-provided, and tool-specific messages
• Points where untrusted data enters or re-enters the model context
• Distinction between trust boundaries established by developers versus those inherited from external dependencies
• The semantic nature of AI vulnerabilities compared to traditional syntactic attacks
• Mapping OWASP LLM Top 10 risks to specific code implementations for government applications

Key insight: Every interface where untrusted text reaches the model, or where model output interacts with application logic, constitutes a trust boundary under developer control.

Module 2 — Prompt Injection Techniques for Developers

Lab: Lab 01 — 01-Prompt-Injection

Analogous to the historical challenges of SQL injection, this module addresses inseparable prompt risks.

Topics:

• Differentiation between direct and indirect prompt injection vectors
• Injection techniques embedded within documents, web content, or tool outputs
• Jailbreak attempts and role-based confusion tactics
• The critical importance of separating instructions from data inputs
• Defensive prompt engineering strategies: delimiters, structured formats, and principle of least authority
• The limitations of prevention-only approaches: designing for containment and mitigation

Hands-on activities:

• Conducting attack simulations against a prototype chatbot
• Demonstrating bypasses of simplistic input filters
• Refactoring prompt structures to limit the potential impact of successful injections

Module 3 — Managing Untrusted Model Outputs

Lab: Lab 02 — 02-Output-Handling

Addressing a vulnerability class frequently underestimated by development teams.

Topics:

• Treating model-generated text as untrusted input for downstream application processes
• Mitigating insecure output handling (LLM02), including cross-site scripting (XSS), server-side request forgery (SSRF), and command injection risks
• Prohibiting the use of raw model outputs in execution contexts such as eval, exec, or rendering engines
• Utilizing structured outputs and schema validation mechanisms
• Implementing output encoding and strict allowlist policies
• Safe rendering practices for web and user interface contexts

Hands-on activities:

• Identifying and remediating insecure output handling vulnerabilities
• Enforcing JSON schema validation on model responses to ensure data integrity

Module 4 — Securing Retrieval-Augmented Generation (RAG)

Lab: Lab 03 — 03-RAG-Security

Identifying and securing a significant new attack surface created by data ingestion processes.

Topics:

• Vulnerabilities within vector databases and retrieval mechanisms
• Sanitization protocols for data ingestion pipelines
• Establishing document provenance and trust scores for source materials
• Retrieval scoping and metadata isolation to prevent unauthorized data access
• Detecting hidden instructions within retrieved content (indirect injection)
• Preventing data exfiltration via retrieval mechanisms

Hands-on activities:

• Demonstrating vulnerability in a RAG pipeline through malicious document insertion
• Implementing ingestion sanitization and retrieval scoping controls to defend the pipeline

Module 5 — Agent and Tool Safety

Lab: Lab 04 — 04-Agent-Safety

Addressing risks where software vulnerabilities translate into unintended operational actions.

Topics:

• Mitigating excessive agency (LLM06) and preventing tool misuse
• Applying least privilege principles to autonomous agents
• Implementing tool allowlists and rigorous argument validation
• Establishing approval gates and human-in-the-loop verification processes
• Sandboxing tool execution environments
• Issuing scoped, short-lived credentials for agent operations
• Restricting autonomous loops and chained actions to prevent runaway behaviors

Hands-on activities:

• Restricting permissions for an over-privileged agent
• Implementing allowlists and approval gates for high-risk tool interactions

Module 6 — Secrets Management, Identity, and Cost Control

Lab: Lab 05 — 05-Secrets-and-Cost

Addressing operational errors that pose immediate risk to system integrity and fiscal responsibility.

Topics:

• Secure management of API keys and secrets (excluding prompts, code repositories, and logs)
• Implementing per-user authentication and authorization for AI services
• Propagating user identity securely to tools and retrieval systems
• Mitigating denial-of-wallet attacks via unbounded token or cost consumption
• Enforcing rate limits, token budgets, and operational timeouts
• Configuring logging protocols that protect secrets and Personally Identifiable Information (PII)

Hands-on activities:

• Removing secrets from prompts and code execution paths
• Implementing per-user rate limits and establishing token/cost budgets

Module 7 — Implementation of Guardrail Frameworks

Lab: Lab 06 — 06-Guardrails

Evaluating the decision to develop in-house solutions versus utilizing third-party libraries for input and output safety.

Topics:

• Capabilities and limitations of existing guardrail frameworks
• Input protections: classification of injections, PII detection, and topic filtering
• Output protections: validation, content filtering, and grounding checks
• Determining appropriate use cases for guardrails versus custom deterministic checks
• Integrating guardrails with security controls established in prior modules
• Managing performance impacts, false positives, and potential failure modes

Hands-on activities:

• Deploying an input/output guardrail layer for an AI feature
• Measuring detection efficacy and identifying gaps in coverage

Module 8 — Red-Teaming Internal Applications

Lab: Lab 07 — 07-Red-Teaming

Adopting an attacker-centric perspective prior to deployment.

Topics:

• Developing comprehensive abuse and test suites for AI functionalities
• Automating tests for prompt injection and jailbreak techniques
• Regression testing of guardrails and security policies
• Integrating AI security checks into Continuous Integration (CI) pipelines
• Managing model and dependency supply chain integrity, including provenance tracking and version pinning
• Establishing a pre-deployment security checklist for AI features relevant to government systems

Hands-on activities:

• Developing automated red-team tests for an AI feature
• Integrating these tests into a CI workflow for continuous monitoring

Module 9 — Assessing AI Security: The SAIS-100 Framework

Lab: None — Scoring exercise utilizing the Capstone application

Establishing a repeatable methodology for evaluating security posture.

Topics:

• The AI Security Hexagon: six critical questions replacing binary "secure/insecure" assessments
• Six scored categories: Data, Prompt, Agent, Supply Chain, Detection, and Governance
• Application of the 100-point rubric and specific weightings
• Verdict bands and the rule for single-category overrides
• The Elephant Scale Secure AI Score (SAIS-100) as a standardized, re-runnable assessment framework for government use
• Using pre- and post-hardening scores to measure remediation effectiveness

Hands-on activities:

• Assigning a score to the Capstone app using the 100-point scale
• Identifying the single most impactful change to improve the security score

Key insight: The three highest-weighted categories align with trust boundaries owned by developers, ensuring the assessment reflects the core competencies taught in this course.

Capstone Project

Participants harden a deliberately vulnerable AI application end-to-end.

The starter application includes the following vulnerabilities:

• Prompt injection susceptibility
• Insecure output handling mechanisms
• Unscoped RAG pipeline configuration
• Over-permissioned agent permissions
• Hardcoded secrets in the prompt path
• Absence of cost controls

Participants apply course concepts to:

• Restructure prompts to ensure containment
• Validate and encode model outputs
• Sanitize inputs and scope retrieval operations
• Apply least privilege principles and approval gates to agents
• Remove secrets and implement cost/rate limits
• Add guardrails and automated red-team testing capabilities

Deliverable: A hardened application accompanied by a self-assessment against the OWASP LLM Top 10 guidelines.

Module and Lab Mapping

Labs proceed in sequential order, corresponding to module progression. The curriculum comprises 9 modules and 7 labs. Module 1 serves as an architecture walkthrough, and Module 9 functions as a scoring exercise; therefore, neither includes a dedicated lab folder.

• Lab 01 - 01-Prompt-Injection: Attack simulation and containment design (Module 2)
• Lab 02 - 02-Output-Handling: Remediation of insecure output handling (Module 3)
• Lab 03 - 03-RAG-Security: Simulated compromise and defense of a RAG pipeline (Module 4)
• Lab 04 - 04-Agent-Safety: Restriction of over-permissioned agents (Module 5)
• Lab 05 - 05-Secrets-and-Cost: Secure key management and cost controls (Module 6)
• Lab 06 - 06-Guardrails: Implementation of input/output guardrails (Module 7)
• Lab 07 - 07-Red-Teaming: Automation of red-team tests within CI (Module 8)

Note: Module 1 focuses on architecture discussion without a lab component. Module 9 utilizes the Capstone app for scoring exercises and does not require a separate lab folder.