Course Outline
Module 1 — How AI Applications Fail
Lab: None — Architecture Review & Technical Discussion
A developer-oriented model of the attack surface.
Topics:
- Architectures for LLMs, Retrieval-Augmented Generation (RAG), and agents from a development perspective
- The request-and-response lifecycle within AI-enabled features
- Prompt construction: System, Developer, User, and Tool message layers
- Points of entry for untrusted data into and through the model
- Differentiation between trust boundaries owned versus inherited by developers
- The semantic nature of AI vulnerabilities compared to traditional syntactic attacks
- Mapping the OWASP Top 10 for LLM Applications to source code implementation
Key insight: Every interface where untrusted text enters the model, or where model output interacts with application code, represents a trust boundary under developer control.
Module 2 — Prompt Injection Mitigation for Developers
Lab: Lab 01 — 01-Prompt-Injection
The equivalent of "SQL injection" for AI systems—a persistent challenge requiring architectural awareness.
Topics:
- Distinguishing direct from indirect prompt injection vectors
- Hidden instructions embedded in documents, web content, and tool outputs
- Jailbreak techniques and role-confusion strategies
- The critical importance of separating instructions from data
- Defensive prompt engineering: Utilizing delimiters, structured formats, and minimal authority principles
- Acknowledging that prevention is insufficient; designing for containment is essential
Hands-on:
- Execute prompt injection attacks against a sample chatbot
- Bypass simplistic filtering mechanisms
- Restructure prompts to minimize potential impact
Module 3 — Treating Model Output as Untrusted Data
Lab: Lab 02 — 02-Output-Handling
A vulnerability class frequently underestimated by development teams.
Topics:
- Model output treated as untrusted input to downstream application components
- Insecure output handling (LLM02): Potential for XSS, SSRF, and command/SQL injection
- Prohibition against executing, evaluating, or rendering raw model output directly
- Implementation of structured outputs and schema validation
- Output encoding techniques and allowlist enforcement
- Secure rendering practices for web and UI contexts
Hands-on:
- Identify and remediate an insecure output handling vulnerability
- Enforce JSON schema validation on model responses
Module 4 — RAG Pipeline Security
Lab: Lab 03 — 03-RAG-Security
A significant new attack surface that requires deliberate security design.
Topics:
- Threats associated with vector databases and retrieval mechanisms
- Sanitization of ingested documents
- Document provenance tracking and trust scoring
- Retrieval scoping and metadata isolation techniques
- Mitigation of hidden instructions within retrieved content (indirect injection)
- Prevention of data exfiltration via retrieval queries
Hands-on:
- Demonstrate poisoning of a RAG pipeline using malicious documents
- Implement ingestion sanitization and retrieval scoping to mitigate risks
Module 5 — Agent & Tool Security
Lab: Lab 04 — 04-Agent-Safety
The transition from data processing to actionable execution.
Topics:
- Excessive agency (LLM06) and potential for tool abuse
- Implementation of least privilege principles for agents
- Tool allowlists and rigorous argument validation
- Mechanisms for approval gates and human-in-the-loop oversight
- Sandboxing strategies for tool execution
- Use of scoped, short-lived credentials for agent operations
- Restrictions on autonomous loops and complex chaining
Hands-on:
- Remediate an overly permissive agent configuration
- Implement an allowlist and approval gate for high-risk tools
Module 6 — Secrets Management, Identity & Cost Control
Lab: Lab 05 — 05-Secrets-and-Cost
Critical operational errors with immediate security and financial impact.
Topics:
- Proper management of API keys and secrets (never included in prompts, code, or logs)
- User-specific authentication and authorization for AI features
- Propagation of user identity to tools and retrieval systems
- Mitigation of denial-of-wallet attacks via unbounded token consumption
- Implementation of rate limits, token budgets, and timeouts
- Secure logging practices that prevent leakage of secrets or PII
Hands-on:
- Remove secrets from prompt contexts and code paths
- Implement per-user rate limits and establish token/cost budgets
Module 7 — Guardrail Libraries & Frameworks
Lab: Lab 06 — 06-Guardrails
Evaluating build versus buy strategies for input and output safety.
Topics:
- Capabilities and limitations of guardrail frameworks
- Input guardrails: Detection of injection attempts, PII, and topic violations
- Output guardrails: Validation, filtering, and grounding verification
- Determining appropriate use cases for guardrails versus deterministic checks
- Layering guardrails with controls established in prior modules
- Considerations for performance, false positives, and failure modes
Hands-on:
- Implement an input/output guardrail layer for an AI feature
- Measure detection rates and identify gaps in coverage
Module 8 — Red-Teaming Your Own Application
Lab: Lab 07 — 07-Red-Teaming
Maintaining a posture of defensive readiness.
Topics:
- Development of abuse and test suites for AI features
- Automation of prompt injection and jailbreak detection tests
- Regression testing of guardrails and security policies
- Integration of AI security checks into CI/CD pipelines
- Management of model and dependency supply chain (provenance and pinning)
- Pre-deployment security checklist for AI features
Hands-on:
- Develop automated red-team tests for an AI feature
- Integrate these tests into a CI pipeline
Module 9 — Scoring AI Security: The SAIS-100 Framework
Lab: None — Scoring Exercise (Utilizes Capstone Application)
Establishing a repeatable metric for security posture.
Topics:
- The AI Security Hexagon: Six essential questions replacing generic security assertions
- Scoring categories: Data, Prompt, Agent, Supply Chain, Detection, and Governance
- 100-point rubric and associated weightings
- Verdict bands and single-category override rules
- The Elephant Scale Secure AI Score (SAIS-100) as a branded, reusable framework
- Using scoring pre-and-post hardening as a performance metric
Hands-on:
- Evaluate the Capstone application on the 100-point scale
- Identify the single enhancement that yields the highest security score improvement
Key insight: The three highest-weighted categories align with developer-owned trust boundaries, ensuring the score reflects the competencies taught in this course.
Capstone Project
Participants harden a deliberately vulnerable AI application end-to-end.
The starter application contains:
- Prompt injection vulnerabilities
- Insecure output handling mechanisms
- An unscoped RAG pipeline
- An over-permissioned agent
- Exposed secrets within the prompt path
- Absence of cost limits
Participants apply course material to:
- Restructure prompts for containment
- Validate and encode model output
- Sanitize and scope retrieval processes
- Apply least privilege and approval gates to the agent
- Remove secrets and implement cost/rate limits
- Add guardrails and automated red-team tests
Deliverable: A hardened application and a brief OWASP LLM Top 10 self-assessment.
Module to Lab Mapping
Labs are executed in numerical order, aligning with module progression. The course comprises nine modules and seven labs. Modules 1 and 9 do not have associated lab folders; Module 1 focuses on architecture review, while Module 9 focuses on scoring exercises.
- Lab 01 - 01-Prompt-Injection: Attack chatbot & design for containment (Module 2)
- Lab 02 - 02-Output-Handling: Remediate insecure output handling (Module 3)
- Lab 03 - 03-RAG-Security: Mitigate RAG pipeline poisoning (Module 4)
- Lab 04 - 04-Agent-Safety: Restrict over-permissioned agents (Module 5)
- Lab 05 - 05-Secrets-and-Cost: Secure credentials and manage costs (Module 6)
- Lab 06 - 06-Guardrails: Implement guardrail layers (Module 7)
- Lab 07 - 07-Red-Teaming: Automate red-team tests in CI (Module 8)
Module 1 (How AI Applications Fail) is conducted as an architecture walkthrough without a dedicated lab. Module 9 (Scoring AI Security) is conducted as a scoring exercise against the Capstone application and does not require a separate lab folder.
Requirements
- Prerequisite: Intermediate proficiency.
- Participants must possess foundational knowledge in constructing and utilizing REST APIs, scripting (specifically Python within laboratory exercises), standard application authentication protocols, version control via Git, and command-line interface operations.
- This curriculum does not require prior machine learning expertise; it is designed as an application security course for developers implementing Large Language Models (LLMs), rather than those specializing in model training.
Audience
- Software and backend engineers developing LLM-integrated capabilities
- Full-stack and API developers
- Engineers specializing in AI/ML application development
- Platform engineers deploying copilot and agent solutions
- Tech leads and senior engineers responsible for AI feature ownership
Testimonials (2)
I really enjoyed learning about AI attacks and the tools out there to begin practicing and actively using for security testing. I took a lot of knowledge away which I didn't have at the beginning and the course met what I hoped it would be. My favorite part shown from the training was Comet Browser and was amazed at what it could do. Definitely something will be looking into more. Overall it was a great course and enjoyed learning all OWASP GenAI Top 10.
Patrick Collins - Optum
Course - OWASP GenAI Security
The profesional knolage and the way how he presented it before us
