Course Outline
Module 1 — Structural Integrity and Attack Surface Analysis of AI Applications
Practical Component: None — Architectural Review and Strategic Discussion
Establishing a foundational mental model for identifying systemic vulnerabilities in AI systems for government deployment.
Core Areas of Inquiry:
- Architectural patterns of LLMs, Retrieval-Augmented Generation (RAG), and autonomous agents from an engineering perspective
- The complete request/response lifecycle inherent to AI-enabled functionalities
- Message flow analysis: system instructions, developer directives, user inputs, and tool interactions
- Identification of vectors where unverified data interfaces with model processing (including re-injection scenarios)
- Delineation of trust boundaries under direct developer control versus those inherited through external dependencies
- The semantic nature of AI vulnerabilities compared to traditional syntactic security flaws
- Alignment of the OWASP Top 10 for LLM applications with specific code implementation standards
Critical Principle: Any interface where unverified data impacts the model, or where model output influences application logic, constitutes a trust boundary requiring explicit management.
Module 2 — Prompt Injection Defense Strategies for System Builders
Practical Component: Lab 01 — 01-Prompt-Injection
Analogous to the historical impact of SQL injection, yet requiring distinct containment strategies due to the inability to fully sanitize prompt-level attacks.
Core Areas of Inquiry:
- Differentiating between direct and indirect prompt injection mechanisms
- Detection of covert instructions embedded in documents, external web resources, or tool-generated outputs
- Analysis of jailbreak techniques and role-confusion vulnerabilities
- The critical importance of segregating instructional directives from data inputs
- Defensive prompt engineering techniques, including delimiters, structural constraints, and authority minimization
- Acknowledging the limitations of preventive measures and designing systems for effective threat containment
Practical Exercises:
- Simulation of chatbot attack scenarios
- Testing the efficacy of basic filtering mechanisms against bypass attempts
- Refactoring prompt structures to minimize the potential impact scope of successful injections
Module 3 — Managing Model Output as Unverified Data
Practical Component: Lab 02 — 02-Output-Handling
A commonly underestimated category of application security vulnerabilities.Core Areas of Inquiry:
- Protocol enforcement for treating model outputs as unverified inputs to downstream application components
- Mitigation of insecure output handling (LLM02), including downstream risks such as XSS, SSRF, and command or SQL injection
- Prohibition of direct execution or rendering of raw model-generated content via eval, exec, or similar functions
- Implementation of structured data outputs and rigorous schema validation
- Application of output encoding and strict allowlisting protocols
- Secure rendering practices within web interfaces and user experience layers
Practical Exercises:
- Identification and remediation of insecure output handling vulnerabilities
- Enforcement of JSON schema compliance for all model responses
Module 4 — Security Protocols for Retrieval-Augmented Generation (RAG)
Practical Component: Lab 03 — 03-RAG-Security
Addressing one of the most significant emerging attack surfaces in AI infrastructure, which requires deliberate architectural safeguarding.
Core Areas of Inquiry:
- Threat analysis of vector databases and retrieval mechanisms
- Sanitization procedures during data ingestion
- Establishing document provenance and trust scoring methodologies
- Implementation of retrieval scoping and metadata isolation protocols
- Defense against covert instructions embedded in retrieved content (indirect injection)
- Prevention of data exfiltration through retrieval channels
Practical Exercise: Introduce malicious documents into a RAG pipeline to simulate poisoning, then implement ingestion sanitization and retrieval scoping controls to mitigate the threat.
Module 5 — Safety Controls for Autonomous Agents and Tools
Practical Component: Lab 04 — 04-Agent-Safety
Transforming potential vulnerabilities into constrained, auditable actions.
Core Areas of Inquiry:
- Mitigation of excessive agency (LLM06) and prevention of tool misuse
- Application of least-privilege principles to autonomous agents
- Implementation of tool allowlists and rigorous argument validation
- Integration of approval gates and human-in-the-loop oversight
- Sandboxing strategies for tool execution environments
- Use of scoped, short-lived credentials for agent operations
- Limitation of autonomous decision loops and action chaining
Practical Exercises:
- Restricting the permissions of an over-privileged agent
- Implementing allowlists and mandatory approval gates for high-risk tools
Module 6 — Management of Secrets, Identity, and Operational Costs
Practical Component: Lab 05 — 05-Secrets-and-Cost
Addressing operational security errors that pose immediate and significant risks.
Core Areas of Inquiry:
- Secure management of API keys and secrets, ensuring they are excluded from prompts, code repositories, and logs
- Implementation of per-user authentication and authorization for AI services
- Propagation of user identity across tool interactions and retrieval processes
- Prevention of resource exhaustion attacks (denial-of-wallet) via unbounded token or cost consumption
- Enforcement of rate limits, token budgets, and timeout controls
- Logging practices that prevent the exposure of secrets or personally identifiable information (PII)
Practical Exercises:
- Removal of sensitive credentials from prompt and code execution paths
- Implementation of per-user rate limits and financial/token usage budgets
Module 7 — Implementation of Guardrail Frameworks
Practical Component: Lab 06 — 06-Guardrails
Strategic evaluation of commercial versus custom solutions for input and output safety.
Core Areas of Inquiry:
- Functional capabilities and limitations of guardrail frameworks
- Input safeguards: classifiers for injection attempts, PII detection, and topic restrictions
- Output safeguards: validation, filtering, and grounding verification
- Determining the appropriate use of guardrails versus custom deterministic checks
- Integration of guardrails with control measures established in preceding modules
- Analysis of performance impacts, false positive rates, and failure modes
Practical Exercises:
- Integration of an input/output guardrail layer into an AI functionality
- Measurement and analysis of detection efficacy and potential gaps
Module 8 — Red-Teaming Internal AI Applications
Practical Component: Lab 07 — 07-Red-Teaming
Pre-emptive security validation by simulating adversarial conditions prior to deployment.
Core Areas of Inquiry:
- Development of abuse and test suites for AI functionalities
- Automation of prompt injection and jailbreak testing protocols
- Regression testing for guardrails and policy enforcement
- Incorporation of AI security checks into continuous integration pipelines
- Management of model and dependency supply chains, including provenance and version pinning
- Creation of a pre-deployment security checklist for AI features
Practical Exercises:
- Authoring automated red-team test cases for specific AI features
- Integration of these tests into CI/CD workflows
Module 9 — AI Security Assessment: The SAIS-100 Framework
Practical Component: None — Scoring Exercise (Utilizing the Capstone Application)
Translating implemented security controls into a standardized, repeatable metric.
Core Areas of Inquiry:
- The AI Security Hexagon: replacing binary security questions with six specific evaluative criteria
- Assessment of six key categories: Data, Prompt, Agent, Supply Chain, Detection, and Governance
- Application of the 100-point rubric and associated weighting factors
- Interpretation of verdict bands and application of single-category override rules
- Utilization of the Elephant Scale Secure AI Score (SAIS-100) as a standardized, re-evaluable framework
- Use of pre- and post-hardening scores as performance indicators
Practical Exercises:
- Conducting a full scoring assessment of the Capstone application against the 100-point scale
- Identification of the single most impactful modification to improve the overall security score
Critical Principle: The highest-weighted assessment categories correspond directly to the trust boundaries under developer control, ensuring the metric accurately reflects the competencies developed in this course.
Capstone Project
Participants will harden a deliberately vulnerable AI application across its entire lifecycle.
The provided starter application includes the following vulnerabilities:
- Injectable prompt structures
- Insecure output handling mechanisms
- Unscoped RAG pipeline architecture
- Over-permissioned autonomous agents
- Exposure of secrets within the prompt path
- Absence of cost or rate limiting controls
Participants will apply course-derived strategies to:
- Reconfigure prompts to enforce containment
- Implement validation and encoding for model outputs
- Apply sanitization and scoping to retrieval processes
- Enforce least privilege and approval gates for agents
- Remove secrets from critical paths and implement cost/rate limits
- Integrate guardrails and automated red-team testing
Final Deliverable: A hardened application accompanied by a concise OWASP LLM Top 10 self-assessment report.
Module and Laboratory Mapping
Laboratories are executed in numerical order, corresponding to the module sequence. The course comprises 9 modules and 7 laboratories. Module 1 serves as an architectural review and discussion, while Module 9 functions as a scoring exercise; consequently, neither module includes a dedicated laboratory folder.
- Lab 01 - 01-Prompt-Injection: Simulate chatbot attacks and design containment strategies (Module 2)
- Lab 02 - 02-Output-Handling: Remediate insecure output handling vulnerabilities (Module 3)
- Lab 03 - 03-RAG-Security: Simulate and defend against RAG pipeline poisoning (Module 4)
- Lab 04 - 04-Agent-Safety: Restrict permissions for over-privileged agents (Module 5)
- Lab 05 - 05-Secrets-and-Cost: Secure credential management and implement cost guardrails (Module 6)
- Lab 06 - 06-Guardrails: Integrate input/output guardrail layers (Module 7)
- Lab 07 - 07-Red-Teaming: Implement automated red-team testing in CI (Module 8)
Module 1 (Structural Integrity and Attack Surface Analysis of AI Applications) does not include a laboratory component, functioning instead as an architectural review and discussion. Module 9 (AI Security Assessment) does not include a laboratory folder, functioning instead as a scoring exercise against the Capstone application.
Requirements
- Proficiency Level: Intermediate.
- Participants should be proficient in: building and consuming REST APIs, working with a scripting language (laboratories utilize Python), basic application authentication, Git, and command-line interface (CLI) operations.
- No background in machine learning is required; this is an application security course intended for professionals building with LLMs, not training them.
Target Audience
- Software and backend engineers developing LLM features
- Full-stack and API developers
- AI/ML application engineers
- Platform engineers deploying copilots and agents
- Technical leads and senior engineers responsible for AI features
Testimonials (3)
inventory and identifying the different risk exposures within AI
Gary Cook - Cybersecurity and Information Technology Risk Division
Course - Introduction to AI Trust, Risk, and Security Management (AI TRiSM)
I really enjoyed learning about AI attacks and the tools out there to begin practicing and actively using for security testing. I took a lot of knowledge away which I didn't have at the beginning and the course met what I hoped it would be. My favorite part shown from the training was Comet Browser and was amazed at what it could do. Definitely something will be looking into more. Overall it was a great course and enjoyed learning all OWASP GenAI Top 10.
Patrick Collins - Optum
Course - OWASP GenAI Security
The profesional knolage and the way how he presented it before us
