Course Outline
Day 1: Core Principles and Primary Threat Vectors
Module 1: Overview of the OWASP GenAI Security Initiative (1 hour)
Learning Objectives:
- Trace the progression from general web application security standards to specialized GenAI risk challenges
- Examine the OWASP GenAI Security Project ecosystem and its available resources
- Distinguish the fundamental differences between legacy application security and modern AI system security
Topics Covered:
- Mission and scope of the OWASP GenAI Security Project
- Introduction to the Threat Defense COMPASS framework
- Review of the AI security landscape and associated regulatory mandates
- Comparison of AI attack surfaces with traditional web application vulnerabilities
Practical Exercise: Configuration of the OWASP Threat Defense COMPASS tool and execution of an initial threat assessment
Module 2: OWASP Top 10 for LLMs – Section 1 (2.5 hours)
Learning Objectives:
- Analyze the first five critical vulnerabilities associated with Large Language Models
- Comprehend specific attack vectors and exploitation methodologies
- Apply effective mitigation strategies to address these risks
Topics Covered:
LLM01: Prompt Injection
- Direct and indirect prompt injection mechanisms
- Risks associated with hidden instruction attacks and cross-prompt contamination
- Case studies: Evasion of safety controls and manipulation of chatbot behavior
- Defensive measures: Input sanitization, prompt filtering, and differential privacy techniques
LLM02: Sensitive Information Disclosure
- Risks of training data extraction and system prompt leakage
- Analysis of model behavior to identify potential exposure of sensitive data
- Privacy implications and regulatory compliance requirements
- Mitigation strategies: Output filtering, access controls, and data anonymization
LLM03: Supply Chain Vulnerabilities
- Security considerations for third-party model dependencies and plugins
- Risks posed by compromised training datasets and model poisoning
- Conducting vendor risk assessments for AI components
- Best practices for secure model deployment and verification
Practical Exercise: Demonstration of prompt injection attacks on vulnerable LLM applications and the application of defensive countermeasures
Module 3: OWASP Top 10 for LLMs – Section 2 (2 hours)
Topics Covered:
LLM04: Data and Model Poisoning
- Techniques for manipulating training data
- Modifying model behavior through the injection of poisoned inputs
- Backdoor attacks and methods for verifying data integrity
- Prevention strategies: Data validation pipelines and provenance tracking
LLM05: Improper Output Handling
- Insecure processing of content generated by LLMs
- Code injection risks arising from AI-generated outputs
- Cross-site scripting vulnerabilities within AI responses
- Frameworks for output validation and sanitization
Practical Exercise: Simulation of data poisoning attacks and implementation of robust output validation mechanisms
Module 4: Advanced LLM Threats (1.5 hours)
Topics Covered:
LLM06: Excessive Agency
- Risks associated with autonomous decision-making and boundary violations
- Management of agent authority and permission scopes
- Unintended system interactions and potential for privilege escalation
- Implementation of guardrails and human oversight controls
LLM07: System Prompt Leakage
- Vulnerabilities related to the exposure of system instructions
- Disclosure of credentials and internal logic through prompt manipulation
- Techniques used by attackers to extract system prompts
- Securing system instructions and external configurations
Practical Exercise: Design of secure agent architectures incorporating appropriate access controls and monitoring
Day 2: Advanced Threats and Implementation Strategies
Module 5: Emerging AI Threats (2 hours)
Learning Objectives:
- Understand current and emerging AI security threats
- Implement advanced detection and prevention techniques
- Design resilient AI systems capable of withstanding sophisticated attacks
Topics Covered:
LLM08: Vector and Embedding Weaknesses
- Vulnerabilities in RAG systems and vector database security
- Embedding poisoning and attacks based on similarity manipulation
- Adversarial examples in semantic search contexts
- Securing vector stores and implementing anomaly detection
LLM09: Misinformation and Model Reliability
- Methods for detecting and mitigating hallucinations
- Considerations regarding bias amplification and algorithmic fairness
- Mechanisms for fact-checking and source verification
- Integration of content validation and human oversight
LLM10: Unbounded Consumption
- Resource exhaustion and denial-of-service attack vectors
- Strategies for rate limiting and resource management
- Cost optimization and budget control measures
- Performance monitoring and alerting systems
Practical Exercise: Construction of a secure RAG pipeline with vector database protection and hallucination detection capabilities
Module 6: Agentic AI Security (2 hours)
Learning Objectives:
- Understand the distinct security challenges posed by autonomous AI agents
- Apply the OWASP Agentic AI taxonomy to practical systems
- Implement security controls for multi-agent environments
Topics Covered:
- Overview of Agentic AI and autonomous systems
- OWASP Agentic AI Threat Taxonomy: Agent Design, Memory, Planning, Tool Use, and Deployment
- Security risks in multi-agent system coordination
- Threats involving tool misuse, memory poisoning, and goal hijacking
- Securing agent communication and decision-making processes
Practical Exercise: Threat modeling exercise utilizing the OWASP Agentic AI taxonomy for a multi-agent customer service system
Module 7: OWASP Threat Defense COMPASS Implementation (2 hours)
Learning Objectives:
- Master the practical application of the Threat Defense COMPASS methodology
- Integrate AI threat assessment into organizational security programs
- Develop comprehensive AI risk management strategies
Topics Covered:
- Detailed analysis of the Threat Defense COMPASS methodology
- Integration with the OODA Loop: Observe, Orient, Decide, Act
- Mapping AI threats to MITRE ATT&CK and ATLAS frameworks
- Creation of AI Threat Resilience Strategy Dashboards
- Integration with existing security tools and operational processes
Practical Exercise: Execution of a complete threat assessment using COMPASS for a Microsoft Copilot deployment scenario
Module 8: Practical Implementation and Best Practices (2.5 hours)
Learning Objectives:
- Design secure AI architectures from the initial stages of development
- Implement monitoring and incident response capabilities for AI systems
- Establish governance frameworks for AI security
Topics Covered:
Secure AI Development Lifecycle:
- Security-by-design principles for AI applications
- Code review practices specific to LLM integrations
- Testing methodologies and vulnerability scanning procedures
- Deployment security and production hardening techniques
Monitoring and Detection:
- AI-specific logging and monitoring requirements
- Anomaly detection techniques for AI systems
- Incident response procedures for AI-related security events
- Forensic analysis and investigation methods
Governance and Compliance:
- AI risk management frameworks and internal policies
- Regulatory compliance considerations (e.g., GDPR, AI Act)
- Third-party risk assessment for AI vendors
- Security awareness training for AI development teams
Practical Exercise: Design of a comprehensive security architecture for an enterprise AI chatbot, including monitoring, governance, and incident response procedures
Module 9: Tools and Technologies (1 hour)
Learning Objectives:
- Evaluate and implement AI security tools
- Understand the current landscape of AI security solutions
- Build practical detection and prevention capabilities
Topics Covered:
- The AI security tool ecosystem and vendor landscape
- Open-source security tools: Garak, PyRIT, and Giskard
- Commercial solutions for AI security and monitoring
- Integration patterns and deployment strategies
- Criteria for tool selection and evaluation frameworks
Practical Exercise: Hands-on demonstration of AI security testing tools and formulation of an implementation plan
Module 10: Future Trends and Conclusion (1 hour)
Learning Objectives:
- Understand emerging threats and future security challenges
- Develop strategies for continuous learning and improvement
- Create actionable plans for organizational AI security programs
Topics Covered:
- Emerging threats: Deepfakes, advanced prompt injection, and model inversion
- Future developments and roadmap of the OWASP GenAI project
- Building AI security communities and facilitating knowledge sharing
- Continuous improvement and integration of threat intelligence
Action Planning Exercise: Development of a 90-day action plan for implementing OWASP GenAI security practices within participants' organizations
Requirements
- General understanding of web application security principles
- Basic familiarity with AI/ML concepts
- Experience with security frameworks or risk assessment methodologies preferred
Audience
- Cybersecurity professionals
- AI developers
- System architects
- Compliance officers
- Security practitioners
Testimonials (1)
I really enjoyed learning about AI attacks and the tools out there to begin practicing and actively using for security testing. I took a lot of knowledge away which I didn't have at the beginning and the course met what I hoped it would be. My favorite part shown from the training was Comet Browser and was amazed at what it could do. Definitely something will be looking into more. Overall it was a great course and enjoyed learning all OWASP GenAI Top 10.
