AI SecureOps: Attacking & Defending AI Applications & Agents
May 14- May 15 2026 (Thursday, Friday). Price $2600 • $2600
Can prompt injections lead to complete infrastructure takeovers? Could AI agents be exploited to compromise backend services? Can jailbreaks create false crisis alerts in security systems? In multi-agent systems, what if an attacker takes over an agents goals, turning other agents into coordinated threats? This immersive, CTF-styled training in AI and LLM security dives into these pressing questions. Engage in realistic attack and defense scenarios focused on real-world threats, from prompt injection and remote code execution to backend compromise. Tackle hands-on challenges with actual AI applications & agentic systems to understand vulnerabilities and develop robust defenses. Youll learn how to create a comprehensive security pipeline, mastering AI red and blue team strategies, building resilient defenses for AI apps & agents, and handling incident response for AI-based threats. Additionally, implement a Responsible AI (RAI) program to enforce ethical AI standards across enterprise services, fortifying your organizations AI security foundation.
Course Outline
Acquire hands-on experience in GenAI and LLM security through CTF-styled training, tailored to real-world attacks and defense scenarios. Dive into protecting both public and private GenAI & LLM solutions, crafting specialized models for distinct security challenges. Excel in red and blue team strategies, create robust LLM defenses, and enforce ethical AI standards across enterprise services. This training covers both "Securing GenAI" as well as "Using GenAI for security" for a well rounded understanding of the complexities involved in AI-driven security landscapes.
By the end of this training, you will be able to:
- Exploit vulnerabilities in AI applications to achieve code and command execution, uncovering scenarios such as instruction injection, agent control bypass, remote code execution for infrastructure takeover as well as chaining multiple agents for goal hijacking.
- Conduct AI red-teaming using adversary simulation, OWASP LLM Top 10, and MITRE ATLAS frameworks, while applying AI security and ethical principles in real-world scenarios.
- Execute and defend against adversarial attacks, including prompt injection, data poisoning, jailbreaks and agentic attacks.
- Perform advanced AI red and blue teaming through multi-agent auto-prompting attacks, implementing a 3-way autonomous system consisting of attack, defend and judge models.
- Develop LLM security scanners to detect and protect against injections, jailbreaks, manipulations, and risky behaviors, as well as defending LLMs with LLMs.
- Build and deploy enterprise-grade LLM defenses, including custom guardrails for input/output protection, security benchmarking, and penetration testing of LLM agents.
- Establish a comprehensive LLM SecOps process to secure the supply chain from adversarial attacks and create a robust threat model for enterprise applications.
- Implement an incident response and risk management plan for enterprises developing or using GenAI services.
Detailed Outline
Introduction
- Introduction to LLM and AI.
- Terminologies and architecture.
- Transformers, Attention & their security implications(hallucinations, jailbreaks etc).
- Agents, multi-agents and multi-modal models.
Elements of AI Security (1 lab)
- Understanding AI vulnerabilities with case studies on AI security breaches.
- OWASP LLM Top 10 and MITRE mapping of attacks on AI supply chain.
- Threat modeling of AI Applications.
Adversarial LLM Attacks and Defenses (6 labs)
- Direct and indirect prompt injection attacks and their subtypes.
- Advanced prompt injections through obfuscation and cross-model injections.
- Breaking system prompts and their trust criteria.
- Indirect prompt injections through external input sources.
Responsible AI & Jailbreaking (6 labs)
- Jailbreaking public LLMs covering adversarial AI, offensive security, and CBRN use-cases.
- Responsible AI frameworks and benchmarks.
- Model alignment, system prompt optimization, and defense.
Building Enterprise-grade LLM Defenses (2 labs)
- Deploying LLM security scanner, adding custom rules, prompt block-lists, and guardrails.
- Writing custom detection logic, trustworthiness checks, and filters.
- Building security log monitoring and alerting for models using open-source tools.
- LLM security benchmarking and continuous reporting.
Red & Blue Teaming of Enterprise AI applications(4 labs)
- Business control flow testing for risky responses & misaligned behavior of applications.
- Using Colab notebooks for automation of API calls and reporting
- Vector database and model-weight tracing for root-cause investigation.
- Rainbow teaming through a 3-way LLM implementation: target, attacker, and judge with self-improving attack prompts.
Attacking & Defending Agentic Systems (5 labs)
- Attacking LLM agents for task manipulation, risky behavior and PII disclosure in RAG.
- Injection attacks on AI agents for code and command execution.
- Compromising backend infrastructure by abusing over-permissioning and tool usage in agentic systems.
- Multi-agent attacks causing privilege too calls, goal manipulation & chained escalations.
Building AI SecOps Process
- Summarizing the learnings into a SecOps workflow.
- Monitoring trustworthiness, safety and security of enterprise AI applications.
- Implementing NIST AI Risk Management Framework (RMF) for security monitoring.
Upcoming & Past trainings
- 2025: Insomnihack, Switzerland: Insomnihack
- 2025: BruCon, Belgium: BruCon
- 2025: Hack Miami, USA: Hack Miami
- 2025: RSA San Francisco, USA: RSA San Francisco
- 2025: DEFCON, Las Vegas, USA: - DEFCON
- 2025: Nsec, Montreal, Canada: Nsec
- 2025: LASCON, Austin, USA: LASCON
- 2025: OWASP Appsec, Auckland, New Zealand: OWASP AppSec
- 2024: Blackhat, MEA: Blackhat MEA
- 2024: RSA San Francisco Workshop: RSA San Francisco
- 2024: Hack Miami, Florida: Hack Miami
- 2024: OWASP, New Zealand: OWASP New Zealand
- 2024: Lascon 2024, Texas: Lascon 2024
- 2024: Deepsec, Austria: Deepsec
- 2023: Blackhat
- 2023: DEFCON Las Vegas
- 2023: OWASP AppSec Days, New Zealand
- 2023: RSA Conference
- 2023: Insomni'hack, Geneva
- 2023: Infosecworld
- 2023: BruCon(virtual)
- 2023: BruCon 2023
- 2023: OWASP Lascon
Who should attend this course?
>
This course is focused on solving enterprise use-cases around securing GenAI applications and services. The content has been developed by speaking with the security leaders of 8 different large-scale enterprises that are actively using GenAI.
What should students bring
* Familiarity with AI and machine learning concepts is beneficial but not required.
* API keys for OpenAI, Langchain, Anthropic.
* Access to AWS and Azure accounts.
* Google Colab and AWS Sagemaker.
* Complete the pre-training setup before the first day.
Trainer Bio
Abhinav Singh is an esteemed cybersecurity leader & researcher with over a decade of experience across technology leaders, financial institutions, and as an independent trainer and consultant. Author of "Metasploit Penetration Testing Cookbook" and "Instant Wireshark Starter," his contributions span patents, open-source tools, and numerous publications. Recognized in security portals and digital platforms, Abhinav is a sought-after speaker & trainer at international conferences like Black Hat, RSA, DEFCON, BruCon and many more, where he shares his deep industry insights and innovative approaches in cybersecurity. He also leads multiple AI security groups at CSA, responsible for coming up with cutting-edge whitepapers and industry reports around safety and security of GenAI.