This workshop, first presented at Black Hat USA 2024, empowers security professionals to systematically red team AI systems. It goes beyond traditional security failures by incorporating novel adversarial machine learning and Responsible AI (RAI) failures, enabling a holistic approach to identifying potential issues before an AI system is deployed.

The curriculum, adapted from our internal upskilling programs designed for other red teams at Microsoft, is tailored to meet the demands of safeguarding AI technologies. Participants will gain practical insights into AI architecture, functionality, and real-world applications, with a focus on popular systems like GPT, Stable Diffusion, and LLaMA.

The workshop is designed to be hands-on, providing an environment where participants can actively engage with custom-built AI applications. Practical modules challenge participants to red team these applications, while subsequent sessions offer proven strategies for fortifying them. Attendees will explore various prompting strategies for LLMs, including zero-shot, few-shot, and advanced techniques such as Retrieval-Augmented Generation.

In this training, participants will learn to red team AI systems using innovative methodologies that address emerging risks, following practices pioneered by the Microsoft AI Red Team (AIRT). The course covers AI-specific vulnerabilities, Responsible AI (RAI) vulnerabilities, and effective mitigations. Participants will engage in a comprehensive learning experience through a blend of lectures, hands-on playground sessions, and coding labs.

By the end of the course, attendees will be equipped to probe any machine learning system for vulnerabilities, including prompt injection attacks, using both manual and automated methods. The coding portion will introduce participants to PyRIT, AIRT’s open-source toolkit for red teaming. Additionally, since the course is hosted by Microsoft, attendees will have the unique opportunity to examine real vulnerabilities discovered in Microsoft products and services, illustrating the “break-fix” cycle that the AI Red Team employs.

This course is led by experts from Microsoft’s AI Red Team, the first team to integrate RAI red teaming alongside traditional security red teaming. Over the past year, this team has assessed every high-risk AI system at Microsoft, including foundation models and Copilots.

2 Days

AI Red Teaming in Practice

Application Security

Hacking

Process

Security Tooling

"Hack Yourself First" is all about building up defensive skills in developers. It looks at security from the attacker's perspective and takes them through the steps necessary to exploit vulnerable software on the web so that they can experience hacking first hand. Workshop participants are set specific goals they must complete that involve probing for risks and then exploiting discrete vulnerabilities in a specially built vulnerable application. The interactive nature of the workshop means that multiple attack vectors are usually identified across the spectrum of participants and each person contributes their own unique perspective as to how specific risks are exploited.

The objective of the workshop is that each person walks away with demonstrated experience across a broad spectrum of specific risks. They not only learn about but also demonstrate practical experience across a range of different vulnerabilities targeted to the specific needs of the group.

Courses run for two days on the following schedule:

The first day build fundamental security skills that all technology professionals delivering applications on the web should posses:

- Discovering Risks via the Browser
- Using an HTTP proxy
- XSS
- SQL Injection
- CSRF
- HTTPS
- Framework Disclosure

The second day delves deeper into online risks, covering more advanced topics in greater depth:

- Password Cracking
- Account Enumeration
- FiddlerScript
- Content Security Policy
- Session Hijacking
- Subresource integrity
- Brute Force Attacks
- Automating Attacks and Review

Attendees will get taught the mechanics of each of these risks and of course the defensive patterns required to defend against them. But more than that, they get exposed to how to think about security; how to apply it in depth via multiple defences, how to choose appropriate controls based on the specific risk of the feature and how to have the discussion about what makes sense in different circumstances.

Above all though, security is just one factor in delivering working software and it has to be applied appropriately. Sometimes it comes with a trade-off against usability or cost and decisions have to be made about not what's just most secure, but what's in the overall best interests of the product being built. This workshop helps those who attend have the right discussions about when and where to invest in security.

- Software Developers
- System Administrators
- Testers

This workshop is aimed at any software developer, system admin or tester who wants to get a better understanding what is going on in cyber space if it comes to hacking and cracking of systems. This workshop enables you to take a pro-active approach and you learn how hackers will try to break your system. This workshop will be an eye opener for most attendees and it is the starting point of becoming a better developer. It all starts with awareness and improving your own habits. So start hacking yourself first, to become a better developer!

Attendees will need to use their own computer with one of following software options installed: Fiddler: 

https://portswigger.net/burp/communitydownload

If possible please also bring your smartphone.

Hack Yourself First: How to go on the Cyber-Offence

Security

This workshop is here to give you the skills you need to make your APIs secure. We're going to think like an attacker to test APIs and like a defender to figure out the best ways to protect them. With deep-dive talks, real-world demos, fun quizzes, and hands-on labs, you'll learn how to lock down your APIs.
During this hands-on training, we'll explore:

The security model of API-based web applications

Recognizing and addressing authorization failures

Understanding Broken Object Property Level Authorization (BOPLA)

Fixing Broken Object Level Authorization (BOLA)

Testing the security of APIs that use JWTs

Best practices for making JWTs secure in modern APIs

Identifying, exploiting, and fixing Server-Side Request Forgery (SSRF) issues

Understanding Cross-Origin Resource Sharing (CORS)

Configuring secure CORS policies for various use cases

Tracking user authentication securely with sessions or tokens

Relying on OAuth 2.0/2.1 for securing APIs

Q & A throughout the workshop to clear up any doubts


This workshop is about more than theory. We're all about giving you practical security tips you can use right away as an API developer. We dig into the root causes of API threats and how to handle them. We don't just skim the surface of problems and solutions - we get into the why's and how's, looking at common fixes, why some fall short, and which ones are currently the best way to go.
By the end of this workshop, you'll be up-to-speed on the best practices for API security. You'll also leave with a handy list of steps to check and boost the security of your applications.




This training is perfect for developers and architects who work a lot with APIs. If your role involves building, testing, or designing modern apps, this workshop will give you a thorough, up-to-date understanding of the best ways to keep things secure. We'll often use NodeJS, Flask, and Spring Boot in our code examples and demos, but you'll easily be able to apply what you learn to other languages and frameworks.




These testimonials from previous workshops give you a good idea of what to expect:

Trainer is great and an expert in the domain. All of the topics are very relevant. Practical examples for most of the topics. Excellent communication and addressing of questions.

Even though the topic is broad, there was no single moment where my focus went astray. Philippe talks in a way to keep you interested to listen to him.

I liked the the pleasant and relaxed way of speaking and the fresh style of presentation of this kind of dry stuff :)

Philippe is a friendly and knowledgable trainer and delivered an interesting course that was well presented. Questions were answered promptly and in a detailed way.


To participate in this training, you should have some experience with building API-based applications. Knowledge of application security can be helpful, but is not required.




To participate in the lab sessions, participants need a computer with a full-featured modern browser installed (preferably Chrome).


Bulletproof APIs: Hands-On API Security

We levelled up the threat modeling war game. Engaged in CTF-style challenges, your team will battle for control over an offshore wind turbine park.

The level of this training is Beginner/Intermediate. Participants who are new to threat modeling are advised to follow our self-paced Threat Modeling Introduction training (which is about 2 hours and is included in this training).

As highly skilled professionals with years of experience under our belts, we're intimately familiar with the gap between academic knowledge of threat modeling and real-world practice. To minimize that gap, we have developed practical use cases, based on real-world projects. Each use case includes a description of the environment, together with questions and templates to build a threat model.

Students will be challenged in groups of 3 to 4 people to perform the different stages of threat modeling:

• Diagram techniques applied on a travel booking service
• Threat model a cloud-based update service for an IoT kiosk
• Create an attack tree against a nuclear research facility
• Create a SOC Risk Based Alerting system with MITRE ATT&CK
• Mitigate threats in a payment service build with microservices and S3 buckets
• Apply data protection by design and default on a loyalty app
• Apply the OWASP Threat Modeling Playbook on agile development
• Threat modeling the CI/CD pipeline
• Battle for control over "Zwarte Wind", an offshore wind turbine park

After each hands-on exercise, the results are discussed, and students receive a documented solution. All participants get our Threat Modeling Playbook to improve you threat modeling practice, and a one-year access to our online threat modeling learning platform.

As part of this training, you will be asked to create and submit your own threat model, on which you will get feedback. One month after the training we organize an individual coaching session with all the participants.


Course Syllabus/Outline

Agile Whiteboard Hacking - aka Hands-on Threat Modeling - Course Outline

Threat modeling introduction
• Threat modeling in a secure development lifecycle
• What is threat modeling?
• Why perform threat modeling?
• Threat modeling stages
• Different threat modeling methodologies
• Document a threat model
Diagrams – what are you building?
• Understanding context
• Doomsday scenarios
• Data flow diagrams
• Trust boundaries
• Sequence and state diagrams
• Advanced diagrams
• Hands-on: Diagram techniques applied on a travel booking service
Identifying threats – what can go wrong?
• STRIDE introduction
• STRIDE threats
• Hands-on: Threat model a cloud-based update service for an IoT kiosk
• Attack trees
• Hands-on: Create an attack tree against a nuclear research facility
• Attack libraries
• MITRE ATT&CK
• Hands-on: Create a SOC Risk Based Alerting system with MITRE ATT&CK

Addressing each threat
• How to address threats
• Mitigation patterns
• Value of standard mitigations
• Setting priorities through risk calculation
• Risk management
• Threat agents
• The mitigation process
• Hands-on: Mitigate threats in a payment service build with microservices and S3 buckets
Threat modeling and compliance
• How to marry threat modeling with compliance
• GDPR and Privacy by design
• Privacy threats
• LINDUNN and Mitigating privacy threats
• Threat modeling medical devices
• Threat modeling Industrial Control Systems (IEC 62443)
• Threat Assessment and Remediation Analysis for automotive (TARA, SAE 21434)
• Mapping threat modeling on compliance frameworks
• Hands-on: Apply data protection by design and default on a loyalty app
Advanced threat modeling
• Typical steps and variations
• Validation threat models
• Effective threat model workshops
• Communicating threat models
• Agile and DevOps threat modeling
• Improving your practice with the Threat Modeling Playbook
• Scaling up threat modeling
• Hands-on: Apply the OWASP Threat Modeling Playbook on agile development
• Hands-on: Threat modeling the CI/CD pipeline
Threat modeling resources
• Open-Source tools
• Commercial tools
• General tools
• Threat modeling tools compared
Examination
• Hands-on examination
• Grading and certification
Battle for control over "Zwarte Wind", an offshore wind turbine park
Red team versus Blue team battle for control over an offshore wind turbine park

Review session (online coaching session after 1 month)
• Hand-in of your own threat model
• Individual feedback on your threat model
• Review session


Agile Whiteboard Hacking – aka Hands-on Threat Modeling

AI/ML

Architecture

Cloud Security

Privacy

SDLC

UNDERSTANDING EXPLOITATION AND FINDING VULNERABILITIES

Gain essential knowledge, and hands-on experience, in effective vulnerability detection tools and techniques, and how these vulnerabilities are wielded in exploitation of C++ and C applications. By deepening your understanding of exploitation, the motivations driving mitigations, and the identification of high-risk constructs, you will be able to design software that better meets your security needs.

This training is explicitly targeted at C++ developers, though C developers will also benefit.

The training will provide its students with:

knowledge on how to use tools to find vulnerabilities in native applications

give a hands-on experience in some exploitation techniques

 - Slack: Will be setup a week in advance to facilitate resolving of any technical issue

 cloud instance: guarantees same environment

This training is explicitly targeted at security professionals with some programming experience in C or C++.

: How to use tools like Address Sanitizer and fuzzers like AFL/libFuzzer to find and fix security vulnerabilities. Here you will use fuzzing to find the Heartbleed vulnerability in OpenSSL. You will also be tasked with fixing Heartbleed, and then reviewing the fix that was shipped at the time, to get a realistic impression of how difficult it can be to analyze and fix vulnerabilities in real life scenarios.

Exploiting Buffer Overflows with Custom Exploit Shellcode

: How to exploit buffer overflows and execute arbitrary code, and the mitigations that can help prevent it from happening. Here you will exploit a program with your own custom shellcode.

Return Oriented Programming (ROP) and Format Strings

: How to bypass stack protection mechanisms using ROP and generated ROP chains. And we’ll use format string vulnerabilities as an example of a completely different way of exploiting applications.

DAY 1 - FINDING VULNERABILITIES USING FUZZING

Introduction to exploitation, vulnerabilities and specifications

Mitigations and Tooling: Static and Dynamic Analysis

Undefined Behaviour and Compiler Optimizations

DAY 2 - EXPLOITATION AND WRITING SHELLCODE

Exploitation: Writing and Testing Custom Shellcode

Exploitation: Return Oriented Programming (ROP)

(In)Secure C++: Sec Edition

Day 1: Foundations of Application Security in AI Prompt Engineering

Introduction to Application Security and AI - Overview of the foundational principles of application security. - Introduction to AI in software development and the role of prompt engineering in influencing the security of AI-generated code. - Understanding the risks associated with AI-based code generation and why security should be a priority from the start.

OWASP Top 10 Overview in the Context of AI - A detailed review of the OWASP Top 10 security vulnerabilities and how they manifest in web applications. - Examining real-world examples where AI-generated code can introduce vulnerabilities if not guided properly. - How to use AI to mitigate common risks like cross-site scripting (XSS), injection attacks, and improper authentication handling. - Exercise: Participants analyze AI-generated code for security vulnerabilities related to the OWASP Top 10 and discuss how to craft better prompts to avoid these issues.

Crafting Secure Prompts for Code Generation - Best practices for constructing AI prompts that lead to secure and well-structured code. - Techniques for encouraging the AI to implement security best practices, such as input validation, secure authentication mechanisms, and error handling. - Prompt examples to help AI follow secure frameworks and design patterns, particularly in environments like React, Node.js, and other modern web development frameworks. - Exercise: Participants craft and test prompts designed to generate secure code and avoid common vulnerabilities like SQL injection, XSS, and data exposure.


Day 2: Advanced Techniques and Practical Applications

Secure Software Development Lifecycle (SDLC) with AI Assistance - Integrating security into the software development lifecycle when using AI for code generation. - Ensuring that AI-generated code complies with industry security standards and best practices throughout development phases. - Key checkpoints for validating and verifying security during development and testing.

Practical Prompt Engineering for Secure Coding in React - Deep dive into prompt engineering techniques for generating secure React code. - How to guide AI to implement safe component design, secure state management, and secure API interactions in React-based applications. - Exercise: Participants create and refine prompts that generate secure React components with proper security and design controls in place.

Threat Modeling and AI-Assisted Security - Introduction to threat modeling and how it can be applied to AI-generated code. - How to prompt AI to consider security risks at every stage of code development. - Identifying potential threats, attack vectors, and ways to mitigate them through careful prompt design. - Exercise: Participants perform a basic threat model on AI-generated code and refine prompts to address identified security concerns.


By the end of this two-day course, participants will have a solid understanding of how to integrate application security principles into AI prompt engineering. They will be equipped with practical skills to craft prompts that guide AI to generate secure, scalable, and robust web applications. Additionally, they will gain insights into common vulnerabilities and the strategies needed to mitigate them in AI-generated code.

Application Security in AI Prompt Engineering

Multi-platform, multi-client, and highly-mobile users bring a new set of challenges, so the approaches of the past are no longer appropriate for modern applications. This two-day workshop is your chance to dive into all things security related to these new technologies. Learn how to securely connect native and browser-based applications to your back-ends and integrate them with enterprise identity management systems as well as social identity providers and services.


.NET Core, ASP.NET Core, MVC, Web APIs, Claims, OpenID Connect, OAuth 2.0, WS-Federation, SAML, JSON Web Tokens, Single Sign-on and off, Federation, Delegation, Home Realm, Discovery, CORS

Social Logins (e.g. Google, Facebook, Twitter, etc.)


Attendees will need to bring a computer with the latest .NET Core SDK and the IDE of your choice (e.g. Visual Studio) installed.

Identity & Access Control for modern Applications and APIs using ASP.NET 9

.NET

Root, UID transitions, and capabilities (*)

Creating a child process in a new namespace: clone()

Combining user namespaces with other namespaces

What does it mean to be superuser in a namespace?

The BPF virtual machine and BPF instructions


The primary audience comprises designers and programmers building privileged applications, container applications, and sandboxing applications. Systems administrators and DevOps engineers who are managing such applications are also likely to find the workshop of benefit.


The workshop consists of a mixture of presentations coupled with practical exercises that allow participants to apply the knowledge learned in the presentations.


Participants should be familiar with fundamental system programming topics such as file I/O using system calls, signals, and the system calls that define the lifecycle of a process (

). For a refresher on these topics, you can download the course materials available at 

. In addition, participants should have a reading knowledge of the C programming language. (Note, however, that the practical sessions do not require writing C programs.)


You'll need a laptop with Linux installed - either as a native install or inside a virtual machine (VM). You should ensure that you have a fairly recent Linux distribution.

NDC { Security }

Workshops

09:00 - 17:00 (UTC+01)

Identity & Access Control for modern Applications and APIs using ASP.NET 9

Anders Abel

Bulletproof APIs: Hands-On API Security

Philippe De Ryck

Application Security in AI Prompt Engineering

Jim Manico

Linux Security and Isolation APIs Fundamentals

Michael Kerrisk

(In)Secure C++: Sec Edition

Patricia Aas

Hack Yourself First: How to go on the Cyber-Offence

Scott Helme

Agile Whiteboard Hacking – aka Hands-on Threat Modeling

Sebastien Deleersnyder

AI Red Teaming in Practice

Amanda Minnich (AIRT)

Gary Lopez